chore(wolf): record merge verification + bug-1490 regression + p2-hang fix lessons

.wolf/cerebrum.md

@@ -612,3 +612,6 @@ Root-caused a `committed_job_loss=1` chaotic-soak FAIL entirely from logs/timest
 
 ## Do-Not-Repeat (2026-08-13, bug-1512) — a controller ACK that shares the same event enum as the real failure is not the failure; a re-PEND-on-any-notify fix silently over-dispatches every job
 bug-1494's re-PEND fix (bug-1510, commit 6928f74) added a `jobDispatchFail` case to `sched_handle_notify` that requeued the job to PEND on ANY such notification. But the controller sends `NotifyData_data_jobDispatchFail` for EVERY dispatched job — the else branch of `handle_jobdispatch` calls `notify_dispatch(jobid, 0)`, so **code 0 is the SUCCESS ack**, and only `code < 0` is a real rejection. The unguarded handler therefore re-PENDed every successfully-dispatched job, the next election re-dispatched it while the original instance kept running, and a 3-worker cluster ran 38 instances for 24 submitted jobs, piling onto the first worker. **Do-not-repeat**: before writing a handler for a controller notify that carries an outcome `code`/`status` field, check the CONTROLLER side to see what values it actually sends on both the success and failure paths (here: the same event, code 0 vs negative) — a "handle the failure" fix that doesn't gate on the code field turns every success into a failure and amplifies, not fixes, the original symptom. Verify live by comparing running-instance count to submitted-job count, not just by checking jobs reach RUN.
+
+## Do-Not-Repeat (2026-08-25) — bare `nix-store --gc` on this box nukes the devShell closure and forces a massive re-download/rebuild
+When `/` filled up (100%, no space for ninja's `.ninja_log` even), `nix-store --gc` freed 6.1 GiB but ALSO deleted the store paths that `nix develop`'s devShell closure needs (they were not GC-reachable from any live derivation at that instant). The next `nix develop` then re-subscribed hundreds of store paths from the binary caches (USTC mirrors + cache.nixos.org) and rebuilt the flake's custom derivations (unity, hwloc, extralib, xcecore) — a ~10-20 GiB, many-minutes operation that stalls all further build/test work. **Do-not-repeat**: for disk-pressure on this box, prefer deleting regenerable artifacts FIRST (meson `build.mergetest` test binaries/`.o` — `find build.mergetest -path '*testsuite*' -type f \( -name '*.o' -o -perm -111 \) -delete` frees ~1.5 GiB instantly and ninja rebuilds on demand) or stale meson dirs (`src/build`, `src/build.dbg`), NOT `nix-store --gc`. If GC is truly needed, first `nix-store -q --references $(nix-store -q --references $(nix path-info --json .#devShells.$system.default | jq -r 'keys[0]') )` — realistically just accept that devShell re-realization will follow and budget the time. General rule: `nix-store --gc` breaks the "fast incremental dev loop" more than it helps, because the flake's closure is huge and the mirrors are slow.

.wolf/memory.md

@@ -7399,3 +7399,9 @@
 | 02:1x | Added human-readable jobid (job_id_human/human_id) to apiserver responses (backend.FormatJobID reuse); renamed SSE perm event:subscribe->event:view; updated docs (left_scheduler_work, design doc v1.3, cerebrum) | src/frontend/apiserver, src/frontend/schedulerclient, docs | verified live (submit + get_job return 1-0-... form); all tests pass; pushed | ~90k tok |
 | 06:21 | bug-1512 多 worker 过派发修复：handler.c code<0 才 re-PEND；24 作业 8/8/8 全 DONE；单测+提交 e4aaef4 | handler.c test_handlers.c | fixed+verified | ~2k |
 | 09:1x | PEND reason 全链路补全（bug-1513）：controller SETPREASON hook 推送 + apiserver 转换层填充 + JOB_DEFAULT_COLUMN；集群验证 resreq type==nosuchhost → host_reasons [{t1,3},{t2,3},{t3,3}]；提交 ac6131d | apiserver.c convert.go convert_pure.go convert_test.go models/jobs.go | verified | ~40k |
+| 13:4x | merge-scheduler-framework test fixes: (1) scheduler/worker testsuite meson lost the unity `-I${includedir}` parent include (unity.pc Cflags only add -I.../include/unity) -> `<unity/unity.h>` unresolvable; ported src/lib/meson.build's `unity_dep.get_variable('includedir')` workaround to both testsuite incdirs. (2) test_handlers dispatch-fail test: first sched_handle_notify(code<0) NULLs notify.data.dispatchfail on success, so the code=0 branch reused a NULLed notify -> -EINVAL; re-point dispatchfail after fixture_init. (3) worker controller_connector_test SIGSEGV: Rongsong's rejoin-success tests zero fake_cc so `worker_list==NULL` and my ported bug-1509 `resend_job_status(*cc_ctx->worker_list)` derefs NULL; fixed by giving the fixture a valid empty list ptr. | scheduler/testsuite/meson.build, worker/testsuite/meson.build, test_handlers.c, test_controller_connector.c | handlers 7/7 OK; full suite 55/62 ran (only env-sensitive recordstart symlink FAIL) before 25-min timeout | ~4k |
+| 13:3x | DISK-FULL ESCALATION: `nix-store --gc` freed 6.1GB but nuked the devShell closure -> nix develop now re-downloads/re-builds ~hundreds of store paths (slow). Lesson: never run bare `nix-store --gc` on this box as a space fix; prefer deleting build.mergetest test binaries (regenerable) or old AppImages. If GC is unavoidable, it WILL trigger a massive devShell re-realization. | / | freed 6.1GB, re-downloading ~5-10GB | ~300 |
+| 15:0x | merge-scheduler-framework controller hang (bug-1515): controller_unit_test test_p2_integration group hangs ~60% (sleep/futex, no FAIL). commit.c unchanged; suspect Rongsong bug-1563 flusher fix in persist.c fsync_worker (adds 2nd timedwait wal_sync_ms=5000 on no-progress flush) racing teardown join. Investigating via /proc thread wchan (gdb blocked by ptrace). | persist.c | 3/5 hangs, p2 group passes when it completes | ~2k |
+| 15:4x | bug-1515 ROOT-CAUSED+FIXED: p2_integration hang (~60%) traced via gdb-parent (ptrace attach blocked by yama): main in commit_engine_fini pthread_join, sweep in pthread_cond_wait, 2 fsync_workers idle. Root: Rongsong bug-1563 flusher block used PRE-FLUSH `pending` snapshot; after==before fires on no-op flushes (another path advanced durable to tip mid-flush) too, so the worker spuriously slept wal_sync_ms=5000 in the teardown window, racing commit_engine_fini. Fix: re-evaluate pending AFTER the flush (pending_after), only backoff on REAL sustained fdatasync failure. 20/20 clean (was 3/5 hang). Debug techniques that worked: run test UNDER gdb as parent (ptrace allowed for own children) + `timeout -s INT` to interrupt the hang; /proc/<pid>/task/*/wchan for quick thread-state check. | persist.c | FIXED, 20/20 clean | ~8k |
+| 15:5x | controller full suite 3 FAILs root-caused: Rongsong WAL_JOB_DISPATCH replay tests assert JOB_STAT_RUN after persist_recover, but our bug-1509 B3 (mark_all_run_jobs_unknown_in_memory in persist_recover, p->cluster set) ORs JOB_STAT_UNKWN|TRANSIENT onto RUN. Reconciled by updating the 3 tests (test_wal_job_dispatch_fact.c x2, test_dispatch_commit.c x1) to assert JOB_IS_RUN (base RUN recovered, Rongsong's intent incl. bug-1678 allocs) + UNKWN flag set (our B3). Both groups pass. | controller/testsuite/*.c | 6+4 tests OK | ~2k |
+| 10:3x | merged 集群重启+验证：nix build '.#xce' 产出 w96pv5-xcecore（含 pending_after 修复）；AppRun/runtime 被 nix-store --gc 清掉 → package.sh 用旧 AppImage 提取的 AppRun + 拆出的 runtime ELF 前缀打包，但 `find(b'hsqs')` 命中了 runtime 内部数据 → runtime 被截断（22460B），新 AppImage --appimage-extract 崩 SIGILL。绕过：unsquashfs -d 解包 squashfs 到 /opt/xce/appimage/squashfs-root + touch marker 让 start-cluster.sh 跳过 --appimage-extract（t1 用真实 nix store w96pv5）。集群起来：jobs submit(含 from_host)→RUN→DONE→kill 全通，hosts max_slot=8/NUM_RUN/HOST_TYPE 正常。**新发现 merge 回归**：Rongsong clib.go 覆盖了我们的 bug-1490 修复（ControlJob 的 ctrl.action 应映射到 capnp JobControl union 判别值 0/1，不是 Go JOB_* 常量 1/2/3/8）→ 所有 kill/stop 都 -22 EINVAL。已恢复 `ctrl.action = C.int(jobCtrlDiscriminant(action))`（helper 在 convert_pure.go:30 还活着）+ 重建 apiserver 到 /tmp/apiserver_merged2 + 重启 apiserver，kill 验证通过。待办：重建 xce(含 clib.go 修复)+打包 AppImage 做干净部署；修 AppImage runtime（从 nixpkgs 重建 AppRun+runtime）。 | persist.c, clib.go, start-cluster.sh, package.sh | 集群 OK；bug-1490 回归已修 | ~4k |