chore(wolf): sync anatomy/cerebrum/memory/hooks/token-ledger bookkeeping

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 06:40:25 +00:00
parent 71a814f302
commit eadf88a975
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 3224 additions and 577 deletions

View file

@ -25,7 +25,7 @@
- **HA election timer discipline (bug-1562)**: every early-return branch in a function driven by a one-shot timer (`ha_on_primary_ctrl_fd_rdhup`'s liveness-watchdog callback) must either re-arm the next timer itself or verify some OTHER live timer is actually still scheduled before skipping the re-arm — checking only elapsed-time-since-last-event (the 5.10b debounce branch) is not the same as checking an election is still in flight (the 5.6 guard's actual role/pre_vote_in_progress check). A silently-dropped re-arm freezes the node with zero CPU/log activity forever (looks identical to a clean idle state, not a crash) — diagnose via `/proc/<pid>/task/*/wchan` (all threads in `futex_do_wait`/`do_epoll_wait` with no fds beyond listen+epoll = dropped timer, not deadlock) before reaching for gdb (ptrace is blocked in this sandbox, `yama ptrace_scope=1` + no ancestor relationship to the target).
- **kiro-spec-tasks' independent task-graph sanity review is worth actually dispatching, not skipping as formality**: writing `scheduler-sim-repl-verbosity`'s design, the round-boundary "collapse active output range" logic was placed entirely in repl.c (where the round-boundary hook lives) — but a fresh review subagent (no drafting context) caught that the OTHER two triggers for ending that range (a breakpoint hit, a structural event arriving) are detected in repl_break.c, which had no interface to reach into repl.c's state. This is exactly the class of bug a same-context self-review reliably misses (the drafting mind already "knows" the round-boundary path handles it and doesn't re-derive from the other call sites) — worth the two review rounds it took (RETURN_TO_DESIGN, then NEEDS_FIXES catching a second smaller gap in the same fix) rather than rubber-stamping the first draft.
- **Reproducing HA bootstrap bugs**: match the user's actual inter-node start stagger exactly — a tight ~0.5s stagger (the `/xce-deploy-ha` skill's default) can mask a bug that only triggers when a node sits solo for tens of seconds (real peers answering fast enough prevents the trap window from ever closing). Always ask/check the real timestamps before concluding "could not reproduce."
- **The sim/replay driver (sim_loop.c) does not fully replicate the normal-mode driver's (driver.c) per-session obligations, and this has now caused two separate real bugs**: `xce_core_begin_session()`/`xce_core_signal_go()` (core_session.c) were deliberately split from the older monolithic `xce_core_go()` (scheduler-core spec task 5.2) specifically so a driver can interleave `plugin_begin_session()`/`sched_account_jobs()` between the resval reset and the picker signal — `driver.c:739-749` does this correctly; `sim_loop.c:680` still calls the old monolithic `xce_core_go()` with zero interleaving slot and never calls `sched_account_jobs()` anywhere (bug-1613: every worker's/shared resource's "used" accounting stays permanently zero in `--sim`/`--interactive`, since the per-session resval reset is never followed by re-accumulation). Before trusting any sim-driver behavior that has a normal-mode counterpart, grep the normal-mode driver (driver.c) for the equivalent call and confirm sim_loop.c actually has one too — don't assume parity.
- **The sim/replay driver (sim_loop.c) does not fully replicate the normal-mode driver's (driver.c) per-session obligations, and this has now caused two separate real bugs**: `xce_core_begin_session()`/`xce_core_signal_go()` (core_session.c) were deliberately split from the older monolithic `xce_core_go()` (scheduler-core spec task 5.2) specifically so a driver can interleave `plugin_begin_session()`/`sched_account_jobs()` between the resval reset and the picker signal — `driver.c:739-749` does this correctly; `sim_loop.c:680` still calls the old monolithic `xce_core_go()` with zero interleaving slot and never calls `sched_account_jobs()` anywhere (bug-1613: every worker's/shared resource's "used" accounting stays permanently zero in `--sim`/`--interactive`, since the per-session resval reset is never followed by re-accumulation). Before trusting any sim-driver behavior that has a normal-mode counterpart, grep the normal-mode driver (driver.c) for the equivalent call and confirm sim_loop.c actually has one too — don't assume parity. **UPDATE 2026-08-24 (`xce-audit-invariants scheduler` run)**: bug-1613 is now CONFIRMED FIXED (`scheduler-sim-decision-parity` implementation-complete 2026-08-22) — `sim_reconcile_unaccounted_jobs()` (sim_loop.c:566-709) reconstructs `job->accounts` from the recorded decision stream and `sched_account_jobs()` is called at sim_loop.c:948, mirroring driver.c. bug-1611 is only PARTIALLY fixed — the same mechanism only reconciles when a job's DISPATCH decision and its RUN-state confirmation land in the *same* replay round (trigger gate at sim_loop.c:592 requires `JOB_IS_RUN` before it will even search that round's decoded session), so the original real-world scenario (dispatch round 32, confirmation through round 107) still silently drops — a dedicated follow-up spec `scheduler-sim-eager-reconciliation` is already scoped (requirements-generated, not yet designed) to decouple allocation *capture* from the RUN-state gate. The audit also surfaced a third, currently-**inert** instance of this exact shape: `plugin_begin_session()`/`plugin_end_session()` (plugin.c, driver.c:750,802) are never called by sim_loop.c at all — harmless today only because sim mode's `sim_bootstrap_picker()` refuses every picker except `builtin_fcfs`, which leaves those hooks NULL (bug-1681). **FIXED same day**: both relocated to `xce_sched_core_t*` (mirroring `sched_account_jobs()`'s own relocation -- their bodies never touched anything beyond `core->plugins`, so the driver.c comment claiming `plugin_begin_session()` "needs the full `xce_scheduler_t*`" was itself stale/wrong, not just sim_loop.c's missing call) and mirrored into `sim_loop.c`'s round boundary. Blast radius was wider than the one file: relocating a function's signature that's `void*`-typed in its header can hide callers from a simple grep of the two files you're editing -- a full-tree grep for the function name turned up a THIRD live call site (`scheduler.c`'s legacy `sched_begin_session()`/`sched_end_session()`) plus four testsuite files with their own hand-written stub definitions of the same functions (matching the old signature), none of which surfaced until the first build attempt failed on `scheduler.c` (a file nobody had touched). **Lesson: before relocating/changing any function's signature, grep the FULL src+testsuite tree for its name first, not just the files your task description mentions** -- a `void*`-cast internal signature is exactly the shape that lets a stale caller compile silently right up until the declaration changes. RED/GREEN independently verified by temporarily commenting out the two new call sites, confirming both new spy assertions fail (`Expected 2 Was 0` / `Expected 1 Was 0`), then restoring (git-diff-confirmed byte-identical) and reconfirming green. **General lesson: this project's own buglog `fix` field can go stale relative to fast-moving specs — a spec can reach implementation-complete without anyone updating the buglog entry that motivated it, so always re-verify current code/spec.json state before trusting a buglog "not yet fixed" status as still accurate.** Also: any per-session driver obligation found once in this file (`sched_account_jobs`, `decoded_session->decisions` application) is worth checking for siblings — the invariant is general ("any core-derived state driver.c produces that sim_loop.c doesn't mirror"), not just the two specific calls first found.
- **Scheduler sim REPL has TWO independent, un-gated stdout print paths, not one**: `xce_repl_on_event()` (repl_break.c, `sim->interactive`-only) unconditionally echoes every decoded event regardless of `step` vs `continue` (bug-1595's fix over-applied — it was requested for `step` only), and `sim_print_decision()` (sim_loop.c, fires independent of the REPL/breakpoint machinery) reprints an identical `PEND`/`DISPATCH`/`PREEMPT` line every round a decision recurs, with no dedup and no session/round context on the line. Neither path consults `xce_repl_on_decision()`'s existing per-round tally (`last_n_dispatch`/`last_n_pend`/`last_n_preempt`, surfaced only via `info session`). Logged as bug-1609/bug-1610 (not yet fixed) — before touching REPL output volume, check both files, not just repl_break.c.
- **A soak's own "excluded"/PASS verdict is not a license to stop reading (bug-1563)**: `/xce-test-ha --chaotic`'s verdict deliberately excludes `bug924_fdatasync` (overlayfs EIO) from pass/fail — but a huge excluded count (300K+ in one run) or a different errno than the documented one (EINVAL, not EIO) is worth a manual look anyway. Found: `fsync_worker`'s retry loop only backs off when NOT pending, and a sustained fdatasync failure makes "pending" permanently true (durable_lsn never advances on failure) — a real busy-spin/livelock, invisible to the soak's own job-loss/divergence checks since the other 2-of-3 standbys still satisfy majority. Lesson: skim excluded/noise counts for magnitude and errno before trusting a documented exclusion covers them.
- **Fixing a busy-spin/livelock can unmask a latent notification gap (bug-1564)**: throttling an unbounded retry loop (bug-1563's fix) removed an *accidental* fast-repoll that had been quietly covering for `persist_flush()` never firing `on_durable_advance` (only the async `fsync_worker` did). A deterministic (3/3) new test failure appeared immediately after — always suspect "the old code accidentally covered for a real gap via sheer retry speed" when a correctness fix (removing a spin/race) makes a previously-green test start failing 100% of the time; that's a signal to look for a missing notification/wakeup, not to revert the fix.