chore(wolf): record merge verification + bug-1490 regression + p2-hang fix lessons

This commit is contained in:
lei.chen 2026-08-26 10:54:52 +00:00
parent 8858afed3a
commit 77616435c2
3 changed files with 471 additions and 412 deletions

View file

@ -7399,3 +7399,9 @@
| 02:1x | Added human-readable jobid (job_id_human/human_id) to apiserver responses (backend.FormatJobID reuse); renamed SSE perm event:subscribe->event:view; updated docs (left_scheduler_work, design doc v1.3, cerebrum) | src/frontend/apiserver, src/frontend/schedulerclient, docs | verified live (submit + get_job return 1-0-... form); all tests pass; pushed | ~90k tok |
| 06:21 | bug-1512 多 worker 过派发修复:handler.c code<0 才 re-PEND;24 作业 8/8/8 全 DONE;单测+提交 e4aaef4 | handler.c test_handlers.c | fixed+verified | ~2k |
| 09:1x | PEND reason 全链路补全(bug-1513):controller SETPREASON hook 推送 + apiserver 转换层填充 + JOB_DEFAULT_COLUMN;集群验证 resreq type==nosuchhost → host_reasons [{t1,3},{t2,3},{t3,3}];提交 ac6131d | apiserver.c convert.go convert_pure.go convert_test.go models/jobs.go | verified | ~40k |
| 13:4x | merge-scheduler-framework test fixes: (1) scheduler/worker testsuite meson lost the unity `-I${includedir}` parent include (unity.pc Cflags only add -I.../include/unity) -> `<unity/unity.h>` unresolvable; ported src/lib/meson.build's `unity_dep.get_variable('includedir')` workaround to both testsuite incdirs. (2) test_handlers dispatch-fail test: first sched_handle_notify(code<0) NULLs notify.data.dispatchfail on success, so the code=0 branch reused a NULLed notify -> -EINVAL; re-point dispatchfail after fixture_init. (3) worker controller_connector_test SIGSEGV: Rongsong's rejoin-success tests zero fake_cc so `worker_list==NULL` and my ported bug-1509 `resend_job_status(*cc_ctx->worker_list)` derefs NULL; fixed by giving the fixture a valid empty list ptr. | scheduler/testsuite/meson.build, worker/testsuite/meson.build, test_handlers.c, test_controller_connector.c | handlers 7/7 OK; full suite 55/62 ran (only env-sensitive recordstart symlink FAIL) before 25-min timeout | ~4k |
| 13:3x | DISK-FULL ESCALATION: `nix-store --gc` freed 6.1GB but nuked the devShell closure -> nix develop now re-downloads/re-builds ~hundreds of store paths (slow). Lesson: never run bare `nix-store --gc` on this box as a space fix; prefer deleting build.mergetest test binaries (regenerable) or old AppImages. If GC is unavoidable, it WILL trigger a massive devShell re-realization. | / | freed 6.1GB, re-downloading ~5-10GB | ~300 |
| 15:0x | merge-scheduler-framework controller hang (bug-1515): controller_unit_test test_p2_integration group hangs ~60% (sleep/futex, no FAIL). commit.c unchanged; suspect Rongsong bug-1563 flusher fix in persist.c fsync_worker (adds 2nd timedwait wal_sync_ms=5000 on no-progress flush) racing teardown join. Investigating via /proc thread wchan (gdb blocked by ptrace). | persist.c | 3/5 hangs, p2 group passes when it completes | ~2k |
| 15:4x | bug-1515 ROOT-CAUSED+FIXED: p2_integration hang (~60%) traced via gdb-parent (ptrace attach blocked by yama): main in commit_engine_fini pthread_join, sweep in pthread_cond_wait, 2 fsync_workers idle. Root: Rongsong bug-1563 flusher block used PRE-FLUSH `pending` snapshot; after==before fires on no-op flushes (another path advanced durable to tip mid-flush) too, so the worker spuriously slept wal_sync_ms=5000 in the teardown window, racing commit_engine_fini. Fix: re-evaluate pending AFTER the flush (pending_after), only backoff on REAL sustained fdatasync failure. 20/20 clean (was 3/5 hang). Debug techniques that worked: run test UNDER gdb as parent (ptrace allowed for own children) + `timeout -s INT` to interrupt the hang; /proc/<pid>/task/*/wchan for quick thread-state check. | persist.c | FIXED, 20/20 clean | ~8k |
| 15:5x | controller full suite 3 FAILs root-caused: Rongsong WAL_JOB_DISPATCH replay tests assert JOB_STAT_RUN after persist_recover, but our bug-1509 B3 (mark_all_run_jobs_unknown_in_memory in persist_recover, p->cluster set) ORs JOB_STAT_UNKWN|TRANSIENT onto RUN. Reconciled by updating the 3 tests (test_wal_job_dispatch_fact.c x2, test_dispatch_commit.c x1) to assert JOB_IS_RUN (base RUN recovered, Rongsong's intent incl. bug-1678 allocs) + UNKWN flag set (our B3). Both groups pass. | controller/testsuite/*.c | 6+4 tests OK | ~2k |
| 10:3x | merged 集群重启+验证:nix build '.#xce' 产出 w96pv5-xcecore(含 pending_after 修复);AppRun/runtime 被 nix-store --gc 清掉 → package.sh 用旧 AppImage 提取的 AppRun + 拆出的 runtime ELF 前缀打包,但 `find(b'hsqs')` 命中了 runtime 内部数据 → runtime 被截断(22460B),新 AppImage --appimage-extract 崩 SIGILL。绕过:unsquashfs -d 解包 squashfs 到 /opt/xce/appimage/squashfs-root + touch marker 让 start-cluster.sh 跳过 --appimage-extract(t1 用真实 nix store w96pv5)。集群起来:jobs submit(含 from_host)→RUN→DONE→kill 全通,hosts max_slot=8/NUM_RUN/HOST_TYPE 正常。**新发现 merge 回归**:Rongsong clib.go 覆盖了我们的 bug-1490 修复(ControlJob 的 ctrl.action 应映射到 capnp JobControl union 判别值 0/1,不是 Go JOB_* 常量 1/2/3/8)→ 所有 kill/stop 都 -22 EINVAL。已恢复 `ctrl.action = C.int(jobCtrlDiscriminant(action))`(helper 在 convert_pure.go:30 还活着)+ 重建 apiserver 到 /tmp/apiserver_merged2 + 重启 apiserver,kill 验证通过。待办:重建 xce(含 clib.go 修复)+打包 AppImage 做干净部署;修 AppImage runtime(从 nixpkgs 重建 AppRun+runtime)。 | persist.c, clib.go, start-cluster.sh, package.sh | 集群 OK;bug-1490 回归已修 | ~4k |