lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(apiserver): carry exec_user on job submit so jobs don't run as nobody (bug-1500)

Browse source 
modelJobToSubmit built the C job_startinfo_t without setting execuser, so the
controller forwarded a NULL execuser in the startjob request and the worker
ran every job as "nobody". Now the submit carries the resolved exec user: an
explicit exec_user wins, otherwise the job runs as the submitting user.

- add resolveExecUser() (plain-Go, unit-testable) + TestResolveExecUser
- set si.execuser in modelJobToSubmit, free it in freeJobSubmit

Note: deploying this exposed an ASAN heap-use-after-free in the apiserver's
submit/notification path (strncpy reading freed memory); the fix is committed
but needs the memory-lifecycle investigation before it can ship safely.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-08-04 07:37:48 +00:00
parent ee7dee0ed4
commit 5b47ee603f
8 changed files with 1138 additions and 379 deletions
Download patch file Download diff file Expand all files Collapse all files

51
.wolf/memory.md
View file

@ -7317,3 +7317,54 @@
| 06:32 | 继续查 scheduler 连接逻辑（xcecli_start hostname vs IP）；fetch origin develop/working 看 scheduler 新提交 | 本地 + t1 | 进行中 | ~0.2k tok |
| 06:45 | xcecli_start 连接逻辑正常（endpoint_connect成功→RUN）；但 scheduler 无 socket。矛盾 → 需运行时实时观察 scheduler 连接 | t1/本地 | 进行中 | ~0.2k tok |
| 06:47 | ★ 调度验证成功：本地修复版 scheduler 连 t1 controller，5 个作业全部被分配（exec_hosts=t1）并跑完 EXIT！put_resvals 修复（idxs<=0→<0）解决了崩溃+调度阻塞。全链路打通 | t1 + 本地 | 验证通过 | ~0.5k tok |
| 06:53 | 推送 scheduler 修复 07ab056；扩展 worker 到 t2/t3（连 t1 controller）| myxce/chatbot | done+进行中 | ~0.2k tok |
| 06:55 | t3 worker.yaml 已创建；启动 t2/t3 worker 连 t1 controller | t2/t3 | 进行中 | ~0.2k tok |
| 06:57 | t2/t3 worker 退出因 hosts.yaml 无 t2/t3 主机（同 t1 问题）。加 t2/t3 到 hosts.yaml + 重建 cluster.bin | 本地 + t1 | 进行中 | ~0.2k tok |
| 06:58 | cluster.bin 重建含 t2/t3（sha 76ffcf52c835），待部署 controller + 重启 + 启 t2/t3 worker | t1 + t2/t3 | 进行中 | ~0.2k tok |
| 07:01 | controller 新配置重启成功（tag 76ffcf52c835, PRIMARY term=4）；apiserver 断开待重启；待启 t2/t3 worker | t1 | 进行中 | ~0.2k tok |
| 07:13 | 3 台 run.sh 已加 sqlite 到 LIBPATH。待验证 t2 worker 启动 | t1/t2/t3 | 进行中 | ~0.2k tok |
| 07:15 | 真根因：xcecd DT_NEEDED 硬编码 /nix/store/.../libsqlite3.so 绝对路径；t2/t3 无 /nix/store → 找不到。t1 有 /nix/store 所以能跑。修复：t2/t3 建 /nix/store 软链接 | t2/t3 | 进行中 | ~0.2k tok |
| 07:30 | 方案A试点：只对 xcecd 做 patchelf 自包含（收集依赖库到统一 lib/ + 重写 RUNPATH + sqlite 改短名）| 本地 | 进行中 | ~0.2k tok |
| 07:35 | 在 t2 实测自包含 xcecd（pilot：patchelf RUNPATH=$ORIGIN/lib + sqlite 短名 + 解释器相对）| t2 | 进行中 | ~0.2k tok |
| 07:39 | 调整：lib/ 只放非 glibc 第三方库（hwloc/uv/sqlite/ssl等），glibc 用系统；解释器用系统 ld-linux | 本地 pilot | 进行中 | ~0.2k tok |
| 07:41 | 关键洞察：libssl 等第三方库自身 RUNPATH 也是 /nix/store 绝对路径 → 需对 lib/*.so 也 patchelf 为 $ORIGIN | 本地 pilot | 进行中 | ~0.2k tok |
| 07:42 | ★ 方案A验证成功：自包含 xcecd（patchelf $ORIGIN RUNPATH + 12 非 glibc 库 + 系统 ld-linux）在 t2 无 /nix/store 上完全加载并启动（连 t1:9990）。待整合进 AppImage 打包 | t2 pilot | 验证通过 | ~0.4k tok |
| 07:47 | bug-1496 已记录；t3 已部署自包含 xcecd；启动 t3 worker 连 t1 | t3 | 进行中 | ~0.2k tok |
| 07:48 | t1 controller 挂了（无 9990）！需重启 controller + apiserver + scheduler 才能跑作业。t2/t3 worker 就绪 | t1 | 恢复中 | ~0.2k tok |
| 07:55 | 验证路径：停 t2/t3 pilot worker → 重启 t1 controller → 验证 t1 单 worker 跑作业 | t1/t2/t3 | 进行中 | ~0.2k tok |
| 07:59 | t1 scheduler 反复不连 controller（无 socket）；之前本地 scheduler 能连 t1 并调度。方案：本地常驻 scheduler 连 t1，先让作业跑 | t1+本地 | 进行中 | ~0.2k tok |
| 08:01 | 调度成功：verifyrun 被分配到 t1（exec_hosts 填充，PSUSP 状态）。本地 scheduler 常驻连 t1 有效。待确认作业完整生命周期 + 验证功能 | t1+本地 | 进行中 | ~0.2k tok |
| 08:09 | 修复 scheduler use-after-free（2处 rv->val 置空转移所有权），编译通过。待 ASAN 本地验证不崩 + 调度 | src/scheduler/src/scheduler.c | 编译 OK | ~0.2k tok |
| 08:11 | ★ scheduler use-after-free 修复验证：作业 RUN→DONE 完整执行，ASAN 0 错误，scheduler 稳定。调度链路彻底打通 | t1+本地 | 验证通过 | ~0.3k tok |
| 08:30 | t2/t3 加入 cluster 排查：之前 pilot worker 用 env -i 启动报 incorrect worker idx -1 干扰 controller。先确认现状再排查 | t2/t3 | 进行中 | ~0.2k tok |
| 08:37 | controller 恢复（term=7）。用正确环境启动 t2 自包含 worker，排查 incorrect worker idx -1 | t2 | 进行中 | ~0.2k tok |
## Session: 2026-08-03 08:43
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 08:46 | 状态：scheduler use-after-free 已修+验证；t2/t3 worker join 导致 controller 静默退出（未定位）；作业可在 t1 单 worker 跑通。暂停 controller 排查，先测 jobpeek + mcp | 多机 | 进行中 | ~0.3k tok |
| 01:37 | Created ../../../../tmp/frpcdbg_mon.sh | — | ~250 |
| 01:37 | Session end: 1 writes across 1 files (frpcdbg_mon.sh) | 2 reads | ~523 tok |
| 01:43 | Session end: 1 writes across 1 files (frpcdbg_mon.sh) | 2 reads | ~523 tok |
| 01:53 | Session end: 1 writes across 1 files (frpcdbg_mon.sh) | 2 reads | ~523 tok |
| 02:10 | Created ../../../../tmp/frpc_prod.toml | — | ~93 |
| 02:10 | Created ../../../../tmp/frpc_mon.sh | — | ~271 |
| 02:10 | Created ../../../../tmp/frpc_prod.toml | — | ~95 |
| 02:11 | Created ../../../../tmp/frpc_mon.sh | — | ~273 |
| 02:11 | Edited ../../../../tmp/frpc_mon.sh | 2→2 lines | ~4 |
| 02:16 | Edited ../../../../tmp/frpc_mon.sh | inline fix | ~21 |
| 02:19 | Session end: 7 writes across 3 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh) | 2 reads | ~1334 tok |
| 02:52 | Created ../../../../tmp/frpc_watchdog.sh | — | ~488 |
| 10:53 | frpc 诊断完成: 根因=CGNAT+frpc0.69.1重连死循环(login不注册proxy, 循环数天不自愈, 仅重启打断) | frps server journal + frpc.toml | 根因定位 | ~600 |
| 10:53 | 部署 frpc 看门狗 /tmp/frpc_watchdog.sh: 隧道DOWN×3连续(90s)自动 doas rc.d/frpc restart, 300s防抖; frpc开debug日志 | /tmp/frpc_watchdog.sh + /etc/frp/frpc.toml | 自动恢复已部署 | ~300 |
| 02:54 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 03:10 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 03:46 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 04:10 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 05:36 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 05:41 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 05:45 | Session end: 8 writes across 4 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh) | 2 reads | ~1857 tok |
| 05:49 | Created ../../../../tmp/frpc-watchdog.sh | — | ~338 |
| 13:50 | 看门狗改为纯日志模式(不自动重启): 移到 /home/dankeer/bin/frpc-watchdog.sh + crontab @reboot 开机自启; 只在隧道状态变化时写日志(DOWN抓快照/恢复记录/持续DOWN每5min) | /home/dankeer/bin/frpc-watchdog.sh | 部署完成, 用户人工监视 | ~200 |
| 05:50 | Session end: 9 writes across 5 files (frpcdbg_mon.sh, frpc_prod.toml, frpc_mon.sh, frpc_watchdog.sh, frpc-watchdog.sh) | 2 reads | ~2219 tok |