myxce/.wolf/cerebrum.md

# Cerebrum

> OpenWolf's learning memory. General, reusable knowledge only — bug-specific history lives in .wolf/buglog.json, not here.
> Do not edit manually unless correcting an error.
> Last updated: 2026-08-16 (condensed from a ~40k-token file; see .wolf/buglog.json for full bug history)

## User Preferences
- Run soak tests with specific round counts and deploy directories for precise regression tracking.
- Prefer parallel agent runs for independent analysis work; sequential for dependent tasks.
- Keep test logs for analysis, don't skip intermediate output even if a test hangs.

## Key Learnings
- **`fopen(path, "r")` SUCCEEDS on a directory path on this platform (glibc/Linux) — it does NOT return NULL/set errno the way "could not open the file" naively suggests.** (2026-08-26, xce-test-rc-file task 1.2, bug-1716.) The open failure only surfaces on the FIRST READ: `getline()`/`fread()` returns -1/0 with `errno==EISDIR` and `ferror(f)` set (`feof(f)` stays 0, so don't gate on that). Confirmed via a standalone repro (`fopen("/tmp","r")` then `fread`). Any function whose error-handling contract says "print a warning if the file can't be opened for reading" needs to check BOTH the `fopen()`-returns-NULL case AND a post-read `ferror(f)` check — checking only `if (!f)` right after `fopen()` silently misses the directory case (rc stays 0, warning never printed, but also no crash/runner-call — a quiet contract violation, not a loud failure, so it only surfaces via a test that actually asserts the warning text was printed, not just the return value).
- **xcecli Cobra rootCmd location**: `src/frontend/xcecli/cmd/main.go` imports `cmd "xcecli/router"` (the import alias is `cmd`, but the actual package lives at `xcecli/router/router.go`, package-named `cmd`). Cobra's `rootCmd` is declared there as an **unexported** package-level var (`var rootCmd = &cobra.Command{...}`, router.go:21). ~~The only exported symbol is `Execute(cm *config.ConfigManager) error`, which takes no version param... boundary conflict~~ — **SUPERSEDED, resolved by build-time-versioning task 4.5 (commit fb09afe) and confirmed still true as of dirty-content-fingerprint task 4.5 (2026-08-21)**: `Execute` was extended to `Execute(cm *config.ConfigManager, version string) error`, which itself sets `rootCmd.Version = version` (router.go:29-34) before `rootCmd.Execute()`. main.go now composes the full version string (`GitCommit`/`BuildTime`, plus task 4.5's `composeVersionDisplay(GitCommit, DirtyFingerprint)` strip-and-append) and passes it into `cmd.Execute(configManager, version)` — no further router.go changes needed for version-string composition; only main.go is the boundary now.
- **Build entry point**: `flake.nix` lives in `src/`, not the repo root. Always `nix develop /abs/path/to/src --command bash -c '...'` with an explicit path — Bash cwd isn't stable across calls, and a bare `cd src && nix develop` can silently short-circuit.
- **Test coverage gaps**: documented full-suite commands (`xce-build.sh <component>`, `make test`) each cover only a subset — e.g. a component's default meson run misses newly-added standalone test binaries unless wired into its `RUN_TESTS` case, and the CGO/apiserver backend suite needs `make test-backend` with CGO_CFLAGS/LDFLAGS reconstructed via pkg-config (package is `c-capnproto`, not `capnp_c`). A green default build is not proof of full coverage — verify every new test binary actually runs.
- **Unity harness**: use `UnitySetTestFile(__FILE__)` (not nested `UNITY_BEGIN()/UNITY_END()`) when multiple test-group files share one binary — nested BEGIN/END resets global counters and can mask most of the real suite exit code. Authoritative green = grep raw testlog for `:FAIL:` / word-boundary `\bTIMEOUT\b`, never trust an aggregate `Fail:0` alone. Also rule out a concurrent competing build/soak against the same build dir before treating a timeout/crash as a regression.
- **apiserver zap `encoding: console` still renders structured fields as trailing JSON**: `logger.yaml`'s `console` encoder is only for the message/level/time/caller prefix — any `zap.Field`s passed to `logger.Debug/Info/...` still render as a single trailing `{"key": val, ...}` JSON object at end of line (e.g. `... resource-location stored {"resource": "r15m", "host": "resverify-worker", "value": 1.5}`), in the exact field-append order from the call site. Don't assume `encoding: console` means key=value text — grep/parse trailing zap log fields as a JSON tail block (index()/substr() in bash is fine, no JSON lib needed for a fixed, known field order) regardless of the configured encoding name.
- **capnp/typemap conventions**: every schema struct gets `$C.mapname(...)`; for a wire-only concept with no live C struct, hand-define a small matching struct in `typemap.h`. A `Data`-typed field's generated composite codec is a silent no-op in this backend AND the generated `_ptr` wrapper doesn't zero-init — an unset field is written as literal uninitialized stack garbage (real crash risk). Prefer `List(UInt8)` + a `$C.maplistcount` field, which this backend does generate real code for. A nested-struct getter can return an unresolved far pointer after a segment round-trip — call `capn_resolve()` before reading its scalar fields (Text/Data getters resolve internally and mask this).
- **C comments**: never write a bare `type*/` (pointer type immediately followed by `/`) inside a `/* */` block comment — it silently closes the comment early, producing a cascade of unrelated syntax errors elsewhere. Recurred repeatedly; grep new/edited comments for `[A-Za-z0-9_]\*/` before building. Same failure mode, different trigger (caught before building this time, writing `test_sched_worker_change.c`'s file header): quoting a piece of source code that itself contains `/* ... */` (e.g. describing the literal dead `#if 0 /*FIXME*/ ... #endif` block from `sched.c`) inside your own outer `/* */` doc comment closes the outer comment at that inner `*/` too — C block comments never nest. When a doc comment needs to reference literal `/* */`-bearing source text, paraphrase it (e.g. "a dead #if 0 (FIXME-commented) block") instead of pasting the real comment markers.
- **Thread/lock hazards on async-ified single-threaded code**: a library/struct built for serialized calling (Raft state, SSL* objects, an epoll batch-dispatch loop) that later gets a "drive it on a thread pool" feature needs its own lock/refcount around the *shared* state — per-endpoint or per-callback serialization does not imply serialization across different endpoints. A same-thread same-batch use-after-free (one event's handler freeing something a later event in the identical batch still references) can look identical to a genuine cross-thread race.
- **makeString() (lib/xstring.c)** is rtti-backed, not malloc/strdup — only free it via its own module's rtti-aware teardown, or don't free it at all; a generic `free()`/capnp `free_X()` on it is heap corruption. Use `strdup()` for fields a generated `free_X()` will free directly.
- **Threaded test fixtures**: teardown for a fixture with real background threads must run *unconditionally* before any `TEST_ASSERT_*` — a failed assertion longjmps past later code, orphaning the thread and corrupting the next test's fixture memory.
- **Live-soak validation discipline**: a unit-test-green fix can still be unreachable/dead code in production — always re-run the exact live scenario end-to-end, not just the mechanism fixed. A later, differently-shaped soak failure after a fix isn't automatically a regression from that fix — check whether that failure point was ever exercised before.
- **Sandbox quirks**: `/tmp` can be overlayfs with intermittent fdatasync EIO — durability-sensitive tests need a real fs (e.g. `~/tmp` on ext4). This sandbox's `date` may resolve to BusyBox (no `%N`, silently passes it through instead of erroring) — use bash's `$EPOCHREALTIME`/`SECONDS` builtins for timing instead.
- **HA election timer discipline (bug-1562)**: every early-return branch in a function driven by a one-shot timer (`ha_on_primary_ctrl_fd_rdhup`'s liveness-watchdog callback) must either re-arm the next timer itself or verify some OTHER live timer is actually still scheduled before skipping the re-arm — checking only elapsed-time-since-last-event (the 5.10b debounce branch) is not the same as checking an election is still in flight (the 5.6 guard's actual role/pre_vote_in_progress check). A silently-dropped re-arm freezes the node with zero CPU/log activity forever (looks identical to a clean idle state, not a crash) — diagnose via `/proc/<pid>/task/*/wchan` (all threads in `futex_do_wait`/`do_epoll_wait` with no fds beyond listen+epoll = dropped timer, not deadlock) before reaching for gdb (ptrace is blocked in this sandbox, `yama ptrace_scope=1` + no ancestor relationship to the target).
- **kiro-spec-tasks' independent task-graph sanity review is worth actually dispatching, not skipping as formality**: writing `scheduler-sim-repl-verbosity`'s design, the round-boundary "collapse active output range" logic was placed entirely in repl.c (where the round-boundary hook lives) — but a fresh review subagent (no drafting context) caught that the OTHER two triggers for ending that range (a breakpoint hit, a structural event arriving) are detected in repl_break.c, which had no interface to reach into repl.c's state. This is exactly the class of bug a same-context self-review reliably misses (the drafting mind already "knows" the round-boundary path handles it and doesn't re-derive from the other call sites) — worth the two review rounds it took (RETURN_TO_DESIGN, then NEEDS_FIXES catching a second smaller gap in the same fix) rather than rubber-stamping the first draft.
- **Reproducing HA bootstrap bugs**: match the user's actual inter-node start stagger exactly — a tight ~0.5s stagger (the `/xce-deploy-ha` skill's default) can mask a bug that only triggers when a node sits solo for tens of seconds (real peers answering fast enough prevents the trap window from ever closing). Always ask/check the real timestamps before concluding "could not reproduce."
- **The sim/replay driver (sim_loop.c) does not fully replicate the normal-mode driver's (driver.c) per-session obligations, and this has now caused two separate real bugs**: `xce_core_begin_session()`/`xce_core_signal_go()` (core_session.c) were deliberately split from the older monolithic `xce_core_go()` (scheduler-core spec task 5.2) specifically so a driver can interleave `plugin_begin_session()`/`sched_account_jobs()` between the resval reset and the picker signal — `driver.c:739-749` does this correctly; `sim_loop.c:680` still calls the old monolithic `xce_core_go()` with zero interleaving slot and never calls `sched_account_jobs()` anywhere (bug-1613: every worker's/shared resource's "used" accounting stays permanently zero in `--sim`/`--interactive`, since the per-session resval reset is never followed by re-accumulation). Before trusting any sim-driver behavior that has a normal-mode counterpart, grep the normal-mode driver (driver.c) for the equivalent call and confirm sim_loop.c actually has one too — don't assume parity. **UPDATE 2026-08-24 (`xce-audit-invariants scheduler` run)**: bug-1613 is now CONFIRMED FIXED (`scheduler-sim-decision-parity` implementation-complete 2026-08-22) — `sim_reconcile_unaccounted_jobs()` (sim_loop.c:566-709) reconstructs `job->accounts` from the recorded decision stream and `sched_account_jobs()` is called at sim_loop.c:948, mirroring driver.c. bug-1611 is only PARTIALLY fixed — the same mechanism only reconciles when a job's DISPATCH decision and its RUN-state confirmation land in the *same* replay round (trigger gate at sim_loop.c:592 requires `JOB_IS_RUN` before it will even search that round's decoded session), so the original real-world scenario (dispatch round 32, confirmation through round 107) still silently drops — a dedicated follow-up spec `scheduler-sim-eager-reconciliation` is already scoped (requirements-generated, not yet designed) to decouple allocation *capture* from the RUN-state gate. The audit also surfaced a third, currently-**inert** instance of this exact shape: `plugin_begin_session()`/`plugin_end_session()` (plugin.c, driver.c:750,802) are never called by sim_loop.c at all — harmless today only because sim mode's `sim_bootstrap_picker()` refuses every picker except `builtin_fcfs`, which leaves those hooks NULL (bug-1681). **FIXED same day**: both relocated to `xce_sched_core_t*` (mirroring `sched_account_jobs()`'s own relocation -- their bodies never touched anything beyond `core->plugins`, so the driver.c comment claiming `plugin_begin_session()` "needs the full `xce_scheduler_t*`" was itself stale/wrong, not just sim_loop.c's missing call) and mirrored into `sim_loop.c`'s round boundary. Blast radius was wider than the one file: relocating a function's signature that's `void*`-typed in its header can hide callers from a simple grep of the two files you're editing -- a full-tree grep for the function name turned up a THIRD live call site (`scheduler.c`'s legacy `sched_begin_session()`/`sched_end_session()`) plus four testsuite files with their own hand-written stub definitions of the same functions (matching the old signature), none of which surfaced until the first build attempt failed on `scheduler.c` (a file nobody had touched). **Lesson: before relocating/changing any function's signature, grep the FULL src+testsuite tree for its name first, not just the files your task description mentions** -- a `void*`-cast internal signature is exactly the shape that lets a stale caller compile silently right up until the declaration changes. RED/GREEN independently verified by temporarily commenting out the two new call sites, confirming both new spy assertions fail (`Expected 2 Was 0` / `Expected 1 Was 0`), then restoring (git-diff-confirmed byte-identical) and reconfirming green. **General lesson: this project's own buglog `fix` field can go stale relative to fast-moving specs — a spec can reach implementation-complete without anyone updating the buglog entry that motivated it, so always re-verify current code/spec.json state before trusting a buglog "not yet fixed" status as still accurate.** Also: any per-session driver obligation found once in this file (`sched_account_jobs`, `decoded_session->decisions` application) is worth checking for siblings — the invariant is general ("any core-derived state driver.c produces that sim_loop.c doesn't mirror"), not just the two specific calls first found.
- **Scheduler sim REPL has TWO independent, un-gated stdout print paths, not one**: `xce_repl_on_event()` (repl_break.c, `sim->interactive`-only) unconditionally echoes every decoded event regardless of `step` vs `continue` (bug-1595's fix over-applied — it was requested for `step` only), and `sim_print_decision()` (sim_loop.c, fires independent of the REPL/breakpoint machinery) reprints an identical `PEND`/`DISPATCH`/`PREEMPT` line every round a decision recurs, with no dedup and no session/round context on the line. Neither path consults `xce_repl_on_decision()`'s existing per-round tally (`last_n_dispatch`/`last_n_pend`/`last_n_preempt`, surfaced only via `info session`). Logged as bug-1609/bug-1610 (not yet fixed) — before touching REPL output volume, check both files, not just repl_break.c.
- **A soak's own "excluded"/PASS verdict is not a license to stop reading (bug-1563)**: `/xce-test-ha --chaotic`'s verdict deliberately excludes `bug924_fdatasync` (overlayfs EIO) from pass/fail — but a huge excluded count (300K+ in one run) or a different errno than the documented one (EINVAL, not EIO) is worth a manual look anyway. Found: `fsync_worker`'s retry loop only backs off when NOT pending, and a sustained fdatasync failure makes "pending" permanently true (durable_lsn never advances on failure) — a real busy-spin/livelock, invisible to the soak's own job-loss/divergence checks since the other 2-of-3 standbys still satisfy majority. Lesson: skim excluded/noise counts for magnitude and errno before trusting a documented exclusion covers them.
- **Fixing a busy-spin/livelock can unmask a latent notification gap (bug-1564)**: throttling an unbounded retry loop (bug-1563's fix) removed an *accidental* fast-repoll that had been quietly covering for `persist_flush()` never firing `on_durable_advance` (only the async `fsync_worker` did). A deterministic (3/3) new test failure appeared immediately after — always suspect "the old code accidentally covered for a real gap via sheer retry speed" when a correctness fix (removing a spin/race) makes a previously-green test start failing 100% of the time; that's a signal to look for a missing notification/wakeup, not to revert the fix.
- **nix devShell Python packages are `PYTHONPATH` entries, not merged into `python3`'s own site-packages** (found building a standalone-installable xceadmin wheel): `python3 -m venv --system-site-packages` only inherits the base interpreter's OWN site-packages dir, which is empty in this project's nix shell (`python3Packages.click`/`pycapnp`/etc. are each a separate `/nix/store/...` path added via `$PYTHONPATH`, confirmed via `nix develop … --command sh -c 'echo $PYTHONPATH'`). A `--system-site-packages` venv therefore silently CANNOT see them (`ModuleNotFoundError` at runtime, discovered live). Fix: skip venv/pip entirely — extract the wheel directly (`python3 -m zipfile -e`) and bake the *resolved* `$PYTHONPATH` string into a wrapper script's `export PYTHONPATH=...` at generation time (expand now, not `${PYTHONPATH:-}` at runtime) — mirrors `src/xceadmin/meson-install.sh`'s existing nix-packaging wrapper, which already solved this correctly; I should have grepped for that pattern before reaching for venv the first time.
- **`src/tools/xce-build.sh`'s `setup_build()` does `rm -rf "$BUILD_DIR"` for every single-component invocation** (bug-1565, FIXED): `/xce-build xceadmin` after `/xce-build controller` in the same dir used to silently delete `xce_controller`/`xce_test`/etc. Fixed by an opt-in `--incremental` flag (`meson setup --reconfigure` in place instead of wiping) — deliberately NOT the default, since this project's capnp-generation `custom_target` has known missing-dependency-edge races that a full wipe papers over as a loud, retry-fixable failure (skipping the wipe could turn the same gap into silent stale-generated-code instead). `/xce-build all` (build everything together) remains the simplest option when you don't specifically need incremental speed.
- **A never-joined background thread reading a stack-local test fixture is a latent ASAN stack-use-after-return, exposed only by test ORDERING, not by the thread's own logic** (2026-08-19, scheduler-core design-review follow-up). `sched_sim_driver_run()` (sim_loop.c) spawns `sim_picker_thread_fn` on a real pthread that is, by this codebase's own documented design, "never joined or cancelled... no sim-driver teardown/fini function exists yet" — it loops forever on `cbuf_get(&sim->core.wbuf, ...)`. Every existing white-box test in `test_sim_event_loop.c` already called this with a stack-local `xce_sched_sim_driver_t sim;`, which was ALWAYS a latent stack-use-after-return once the test function returned — it just hadn't been caught, because no test's stack frame had yet been reused by a later frame in a way ASAN's shadow-memory detector caught. Simply INSERTING one new test between two existing ones (no logic change to the existing tests) shifted the stack-reuse pattern enough to make it fire deterministically (reproduced twice). Fix: change the fixture from stack-local to `static` storage — safe here specifically because (a) `sched_sim_driver_start()` unconditionally `memset()`s the whole struct as its first action so no stale-data leak across runs, and (b) each affected test function is called exactly once per process (verified via grep, no re-entrancy) so a second call's memset can never stomp a still-live zombie thread's cbuf mutex/condvar state from a first call — `static` would be UNSAFE to apply to a helper called more than once while its first zombie thread might still be alive. General lesson: when a background thread spawned by code-under-test is documented as "never joined," any test giving that code a stack-local fixture is a ticking ASAN time bomb regardless of whether it has fired yet — check for exactly this "never joined/cancelled" phrasing in the target function's own comments before adding a new test that changes stack layout/ordering in the same file.
- **A plain (non-`--system-site-packages`) venv's `pip install` DOES work for xceadmin's deps, including pycapnp, as long as venv creation + `pip install` + running the installed entry point all happen inside `nix develop … --command`** (2026-08-20, build-time-versioning task 6.3, extends the PYTHONPATH entry above rather than contradicting it): unlike the `--system-site-packages` case, a plain venv's `pip install pycapnp` resolves the prebuilt PyPI wheel fresh over the network (confirmed reachable: `pypi.org`/`files.pythonhosted.org` both resolve in this sandbox) — no PYTHONPATH-visibility problem, since pip is installing its own separate copy into the venv rather than trying to see the nix-injected one. The catch: that PyPI pycapnp wheel dynamically links `libstdc++.so.6`, which the bare host has NO system-wide copy of at all (`import capnp` → `ImportError: libstdc++.so.6: cannot open shared object file` outside nix) — confirmed the SAME venv/pip-install/`import capnp` sequence works cleanly when the whole thing (venv creation, pip install, AND the later `python`/entry-point invocation) runs inside `nix develop $SRC_DIR --command bash -c '...'`, which resolves it via the devshell's own environment. Net: use a plain venv + real `pip install` (matching `.claude/skills/xce-build/SKILL.md`'s own documented "Installing the xceadmin wheel elsewhere" section) for throwaway/test installs where network is available and everything runs under `nix develop`; reach for the PYTHONPATH-entry's zipfile-extraction wrapper trick instead only for a permanent, no-nix, no-network deployment target (e.g. `/xce-deploy-ha`'s `scripts/xceadmin`).
- **A pipeline into `head -N` can SIGPIPE-kill the upstream script early** under `set -o pipefail` (bit twice this session: once with `timeout N cmd | head -N` leaving a stale background process running for hours after the harness reported the task "complete", once with `xce-build.sh ... | head -3` silently truncating a build before it finished writing its wheel). If you need to sanity-check just the first few lines of a long-running command's output, redirect to a file and `head` the file afterward instead of piping directly into `head`.
- **`sim_loop.c` (and its `*_isolation`-audited siblings) genuinely has zero line-budget headroom** — its own file header says so explicitly. When a fix needs new logic there, split a new sibling file (bug-1567's `sim_picker.c`, following the exact precedent chain `sim_loop.c`→`sim_record.c`→`repl.c`→`repl_break.c`/`repl_info.c`→`repl_params.c`) rather than growing it even slightly. This requires wiring the new file into: `scheduler_srcs` AND `sched_sim_isolation_lib` (both `src/scheduler/meson.build`), a new `XCE_SIM_*_PATH` macro (`testsuite/meson.build`) consumed by `test_sim_isolation.c`'s 3 per-file text-scan functions, AND every white-box test TU that `#include`s `sim_loop.c` directly (9 files this session — grep `#include "../src/sim_loop.c"` in `testsuite/` to find them all) needs the new file `#include`d too or the link fails on the new extern. Mechanical but real work — budget for it.
- **`controller_unit_test`'s intermittent SIGSEGV (bug-1553, ~1-in-3 full runs) is a known pre-existing flake**, not a new regression — reproduce standalone (`./src/build.dbg/controller/controller_unit_test`, no `meson test` wrapper) a few times before suspecting your own change; `meson test`'s `MALLOC_PERTURB_` randomization per invocation is what makes it flaky, not deterministic per-build.
- **bug-1568 (FIXED) had TWO independent, unrelated root causes stacked on the same symptom** ("info worker" showing 0.00 for real resource values) — worth remembering the general pattern: (1) `xce_core_push_events()`'s `WORKERS_PUSH` case called `sched_add_worker()` (a plain `key_add()`, add-only) unconditionally on every push; a worker connecting sends multiple pushes in quick succession, so any push after the first for an already-registered name silently -EEXIST-discarded its resource values — fixed via `core_upsert_worker()`, mirroring the existing `core_upsert_queue()` add-or-update precedent (`core_events.c`). (2) Separately, `repl_info.c`'s display code compared `xce_val_t.which` against `XCE_VAL_NUMERIC` (`xce/types.h` `#define`, value 1 — a *resource-definition* value-type constant, `xce_resource_t.valtype`'s own enum) instead of `Value_u_fvalue` (`controller.capnp.h`-generated, value 5 — the real tag every producer in this codebase actually sets on `xce_val_t.which`) — always false, so it displayed 0.00 regardless of whether (1) was fixed. **Lesson**: when value-level TRACE at every pipeline hop (push → apply → snapshot save → snapshot load/decode) all show the correct number but the final displayed/returned value is still wrong, the bug is one hop further than "the pipeline" — trace the DISPLAY/formatting code too, don't stop at the data layer. And don't assume two enum families with parallel-looking names (`XCE_VAL_*` vs `Value_u_*`) are interchangeable just because both appear near `xce_val_t`/`.which`/`.valtype` — check which one a given producer actually sets before trusting a comparison against it.
- **`IFS=$'\t' read` (or any single-tab IFS) silently COLLAPSES adjacent/empty fields, just like the default whitespace IFS does** -- bash classifies tab (along with space/newline) as "IFS whitespace" for splitting purposes regardless of what IFS is actually set to, so `a\tb\t\tc\td` (an empty 3rd field) reads back as 3 fields (`a`,`b`,`c\td` collapsed) not 4, silently shifting every field after the gap. Found live building resource-value-verification task 7.1's `aggregate_verdict()` (`run-resource-verify.sh`) -- its own required synthetic self-test caught it immediately (an UNREACHABLE-layer record's empty `*_observed` field shifted every later field by one, corrupting the apiserver classification). Every OTHER tab-delimited TSV already in that file happens to never carry a genuinely-empty field, which is why this never surfaced before. Fix: when a delimited record format can legitimately have an empty field, use a non-whitespace-class separator (e.g. ASCII Unit Separator `$'\x1f'`) instead of tab -- `IFS=$'\x1f' read` does NOT collapse/strip, tab-like or space-like IFS always does.
- **This Bash tool aborts a multi-line script at the first command returning nonzero**, even with `; true`/`|| true` on that exact line — observed repeatedly with `pkill` matching zero processes (a completely normal, expected outcome) silently killing every following line in the same call. Don't chain a possibly-zero-match `pkill`/similar with follow-on commands in one Bash call — verify via `pgrep` first (it's just as prone to this, so treat ITS nonzero "no match" the same way) or issue the pkill as its own standalone call and check the result before proceeding.
- **`nix build` in this sandbox dispatches to a remote builder (`ssh://rshen@172.31.109.18`) that is NOT reachable from inside the sandbox** (2026-08-21, dirty-content-fingerprint task 6.3 review). `nix show-config`/`/etc/nix/nix.conf` inside the sandbox show `builders = ` (empty) — the remote dispatch is configured on the nix-daemon side (invisible to a `show-config` run inside the sandbox's mount/network namespace), not discoverable by inspecting client-side config. A plain `nix build` therefore either hangs/times out on SSH connect or falls back to the known bcctool/goproxy.cn failure without ever completing locally. **Fix: pass `--builders ''` to force a fully local build** (this sandbox has `max-jobs = 22`, plenty of local capacity) whenever running a real `nix build`/`nix develop`-triggered build for verification in this sandbox — don't assume remote-dispatch failure/unreachability is unavoidable or treat it as an automatic excuse to fall back to eval-only/log-only verification; try the local-builder override first, since it can produce a genuinely complete, runnable binary instead of just a build log.

- **apiserver's `main.go` only calls `RegisterToBackend` when `BackendYamlConfig.Address != ""` — checked BEFORE `Peers` is ever consulted** (`cmd/main.go:58`). Setting `app.yaml`'s `backend.peers` to a fully-populated HA candidate list while leaving `backend.address`/`backend.port` blank (reasoning "Peers takes priority inside RegisterToBackend anyway, per `backend/config.go`'s own doc comment") is NOT sufficient — the apiserver silently boots in permanent "stub mode" (`logger.Info("Backend not configured")`) and never even attempts a connection, `peers` or not. Caught live while building `xce-deploy-ha`'s new apiserver step (resource-value-verification task 3.1): always populate `address`/`port` too (e.g. the first `--peers` candidate's host:port) even when also setting `peers` — belt-and-suspenders, not redundant, because the two fields gate two different things (whether to connect at all, vs. which candidate(s) to try).
- **resource-value-verification task 4.1's scheduler trace-log exact format** (`xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case, `src/scheduler/src/core_events.c`, right after `zarray_put(&(w->resvals), res->idx, rv->val)`): `TRACE("resval applied host=%s resource=%s value=%f", ...)` for numeric (`Value_u_fvalue`) resources, or `value=%s` for `Value_u_svalue` ones — live-confirmed via `xce-deploy-ha` + `xce_test` worker-context `join`/`resupdate -n r15m -v 12345.5` producing exactly `resval applied host=task41-worker resource=r15m value=12345.500000` in `scheduler.log` at `-vvv`. Task 6.3 (scheduler-side log-tail check) MUST grep for `resval applied host=<H> resource=<R> value=<V>` — labeled `key=value` fields, no other fixed text to depend on. `%f` on a normal resource value produces trailing zeros (e.g. `12345.500000`), not the input's original decimal precision — task 6.3's comparison should parse-and-compare as float, not do a literal string match against the injected value string.
- **bug-1572 (NOT FIXED, pre-existing, discovered live task 4.1 round-2): `xce_scheduler` double-frees on a worker's SECOND periodic resource report for the same resource(s), silently dying with no ASAN output** (backgrounded via `setsid nohup ... >/dev/null 2>&1`, so the crash report is lost -- only visible as scheduler.log stopping mid-stream + `handle_scheduler_leave` in the controller log ~1s later). Root cause: `core_events.c`'s `XCE_CORE_EVENT_RESVALS_DELTA` case does `zarray_put(&(w->resvals), res->idx, rv->val)` with NO `MOVE_PTR` (unlike its sibling `core_apply_config_resvals()`, which does `MOVE_PTR(val, nv->val)` first) -- the same `xce_val_t*` stays aliased in the decoded request. `driver.c`'s `release_translated_payload()` then unconditionally frees that request's resvals list even when `applied=1` (its `XCE_SCHED_OP_UPDATE_RESVALS` case has no `if (!applied)` guard, unlike the sibling `XCE_SCHED_OP_UPDATE_CONFIG` case which explicitly skips the free when applied). Net effect: `w->resvals[idx]` is left dangling immediately after every successful RESVALS_DELTA batch; the SECOND update to the same index triggers zarray's killer to free the already-freed pointer. Confirms identically on round-1's code (unrelated to the enum-branch bug fixed in that round) -- a live smoke test that only runs one resupdate quickly after deploy will NOT hit this; anything that waits past one real worker's report interval (xcecd_virtual reports ~every 15s) will. When live-testing anything touching RESVALS_DELTA, either run fast (single update, no wait) or expect this crash and don't misattribute it to whatever fix is actually under test.
- **`src/frontend/Makefile`'s `build-server` target is missing CGO_CFLAGS/CGO_LDFLAGS (bug-1597, NOT FIXED)**: unlike `test-backend` (bug-1558) and `apiserver/meson.build`, `build-server`'s recipe is a bare `go build` with no `-I`/`-L` for `client.h`/`controller.capnp.h`/`libxce.a`, so `nix develop ../ --command make build-server` fails with `fatal error: client.h: No such file or directory` regardless of source changes. Confirmed via git-stash A/B that this reproduces identically with or without an unrelated diff, so don't misattribute it to whatever change is under test. Workaround to actually verify compilation: reconstruct CGO_CFLAGS/CGO_LDFLAGS the same way `test-backend` does (pkg-config for librtti/c-capnproto/openssl/tinylog + `-I$BUILD_DIR/lib`/`-L$BUILD_DIR/lib -lxce`, `BUILD_DIR` defaulting to `src/xce.build`) and run `CGO_ENABLED=1 CGO_CFLAGS=... CGO_LDFLAGS=... go build -o /tmp/apiserver ./cmd/main.go` directly inside `nix develop ../` — see bug-1597 for the exact one-liner.
- **`test.c`'s `set_resval()` leaves `xce_resval_t.name` (and `.val->which`) uninitialized for the `Value_u_svalue` fallback path** (`-n type`/`-n model`/explicit `-t string` on an unknown name) — its switch statement has no `case Value_u_svalue:` arm, and `vals[]` is a non-zeroed stack array reused across REPL invocations, so `.name` can be garbage, not NULL. Pre-existing (predates resource-value-verification's task 2.1/2.2), explicitly flagged as out-of-scope-to-fix in both tasks' code comments (Requirement 1.4's "preserve byte-for-byte" constraint). When building task 2.2's per-name verification loop on top of this, the safe pattern was to track the submitted name in a separate parallel array (`subnames[]`) captured directly from the getopt loop's own `resname` variable (always valid — that pointer is deliberately leaked/never freed, so it stays alive for the whole REPL call) rather than trusting `vals[i].name`.

- **Trap-based cleanup that must survive an async signal (SIGTERM/SIGINT) should combine multiple `pkill -9 -f` calls into ONE alternation pattern, not several sequential invocations, plus a retry-until-verified (`pgrep`) loop** (bug-1575, resource-value-verification task 5.1's `run-resource-verify.sh`). Four separate sequential `pkill -9 -f -- "pattern"` calls inside `cleanup()` worked reliably every time when reached via normal control flow (a direct `fail()`/`exit` call), but non-deterministically left 1-2 of the four target processes running when `cleanup()` instead ran from the EXIT trap fired by an externally-delivered SIGTERM translated via a separate `trap 'exit N' TERM` handler -- the exact failure point (which of the 4 pkill calls was "last to complete") varied run to run, consistent with a timing race in the trap-handler code path rather than a pattern-matching bug (each pattern was independently verified correct). Fix: one combined `pkill -9 -f -- "pat1|pat2|pat3|pat4"` call (fewer fork/exec round trips through the trap handler) wrapped in a loop that re-checks via `pgrep -f` and retries (up to ~2s total) until genuinely clean. Verified reliable across 5+ repeated live SIGTERM-mid-deployment interrupts after the fix (zero leftover processes every time), vs. non-deterministic leftovers before it. General lesson: don't trust a cleanup routine's reliability from its NORMAL-completion path alone -- explicitly live-test the signal-interrupted path too, since bash's async-signal-trap execution context can behave differently from the same code reached via ordinary top-to-bottom flow or a plain function call/exit.
- **`pgrep -f`/`pkill -f` run interactively via this Bash tool can self-match (and `pkill` can self-KILL) the invoking Bash-tool call's own wrapper process** (bug-1576, discovered repeatedly while diagnosing bug-1575 above -- cost significant back-and-forth before recognized). This harness wraps every Bash call as roughly `bash -c "... && eval '<submitted script text>' ..."`, so the wrapper's own `/proc/<pid>/cmdline` contains the FULL LITERAL TEXT of whatever was submitted -- including any search pattern typed directly into that same script (unavoidable, since testing a pgrep/pkill pattern means typing it). A `pgrep -f -- "sometext"` that also appears as literal text elsewhere in the SAME submitted command block will match the wrapper too; `pkill -9 -f` in the same situation can SIGKILL the current Bash call's own shell mid-script, truncating all remaining output with no distinguishing error beyond a generic nonzero/early-stop. Symptoms seen: a match count of "2" when only 1 real process existed; a multi-line diagnostic script silently stopping after the pkill line with no further output. Mitigation: when checking process state from an interactive Bash call, scan `/proc/[0-9]*/cmdline` directly and eyeball results for the wrapper's own distinctive `bash -c "source .../shell-snapshots/... && eval '...'"` shape rather than trusting a bare match count; never assume a lone pgrep/pkill match is the real target without checking its `ps -o cmd` first. This does NOT affect a target script's OWN internal `pkill -f` calls (e.g. inside a cleanup() function) since that script's own argv is just its invocation line (interpreter + path + args), not its source text. This ALSO bites a from-scratch verification watcher, not just a diagnostic pkill: any background polling loop launched via the interactive Bash tool with the target pattern typed inline in its own command text self-matches too (found live, task 6.1) -- write the pattern into a script FILE and launch that file instead, or check `/proc/<pid>/exe` (the resolved binary path) rather than grepping `/proc/<pid>/cmdline` text at all.
- **xce_test's worker-context injection recipe** (resource-value-verification task 6.1, cross-checked against `xce-test-ha/run-ha-soak.sh`'s extensive prior use of the identical pattern): `printf 'set masters %s\nset myname %s\nworker\njoin\nresupdate -n <name> -v <value> [...]\nquit\n' "$PEERS" "$WORKER_HOST" | xtest`. `set myname <host>` is what tags every subsequent `resupdate`'d value's `.host` field (`test.c`: `config.from = get_setting("myname")`; `set_resval()` sets `vals[idx].host = get_setting("myname")`) -- Requirement 2.2's "tag every resupdate-injected value with the joined worker's own identity" is satisfied entirely by this one setting, no separate flag needed. `resupdate_report_one()`'s output lines are exact whole-line strings (`<name>: accepted` / `<name>: rejected: unknown resource` / `<name>: rejected: value mismatch`) -- parse with `grep -Fxq` (fixed-string, whole-line), not a substring/regex grep, since a short resource name (`it`, `io`, `ls`...) can appear as a literal substring of an unrelated line otherwise.
- **A synthetic worker's host-attrs resources (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks`) get RESET to baseline the moment it disconnects -- this is correct, intentional production behavior, not a bug, but it bites any test tool that queries the controller in a SEPARATE `xce_test` process/session after the injecting session's own `quit`** (resource-value-verification task 6.2, bug-1579, not-a-bug/documented). `add_hostattrs_resvals()` (`src/controller/worker.c:260-287`) is the fixed N_HOST_ATTRS list; `mark_worker_unavail()` -> `apply_worker_dead_mutation()` (`worker.c:704-716`) calls `reset_hostattrs_unknown()` then re-applies `add_hostattrs_resvals()` on every worker disconnect, by design (a dead worker shouldn't keep advertising stale capacity). `run-resource-verify.sh`'s `inject_resources()` (task 6.1) runs `join;resupdate;quit` as ONE `xce_test` process -- the `quit` disconnects the synthetic worker before ANY later per-layer check (a separate `xce_test` process) can observe it. A live controller-check run against the task's own DEFAULT_RESOURCE_ARGS set showed 8/9 MATCH + 1 false MISMATCH (`cpuf`, the only host-attrs-colliding name in that default set) -- re-running with the same 8 non-`cpuf` resources gave a clean 8/8 MATCH, confirming the check logic itself is correct and this is purely a worker-disconnect-timing artifact. Only the CONTROLLER-layer check (a live `resval` query) is exposed to this -- the scheduler/apiserver checks (tasks 6.3/6.4) are trace-log-tail based (read what was logged at apply-time, immune to later disconnect). When choosing/validating a resource name for any live check against this script, cross-check it isn't in `worker.c`'s `N_HOST_ATTRS` list first, or expect a query run in a later, separate `xce_test` session to legitimately read back 0/default. **UPDATE 2026-08-18 (task 6.2 remediation round 2, reviewer-mandated fix): the default-set instance is now FIXED.** `DEFAULT_RESOURCE_ARGS` swapped `cpuf=9.5` for `r1m=9.5` (a periodic/dynamic load-index resource per `src/worker/src/resource_sync.c`, confirmed pre-existing/"already-supported" via `resval_typeresolve.c`'s `resinfo[]` table, and NOT in `worker.c`'s `N_HOST_ATTRS` list) -- live-validated clean 9/9 MATCH on the default (no-args) set post-fix, no regression on `mem=5`-only or mixed accept/reject scenarios. Option (b) (keep the synthetic worker's session connected across all per-layer checks instead of `quit`ting) was investigated live in this same round and confirmed NOT a simple call-reordering fix: `xce_test`'s REPL has no command path from the `worker` context back to the `client` context without first calling `leave_ctx()` (via `..`/`cmd_up`), which performs the exact same disconnect `quit` does -- making option (b) work would require either a C-level `xce_test` change or a materially riskier bash coprocess restructuring, both out of scope. This class of issue (a host-attrs resource name queried after synthetic-worker disconnect) remains a live risk for any FUTURE task or user-supplied custom `resource=value` argument naming `type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` -- see buglog.json bug-1579 for the full analysis.
- **bug-1577 (FIXED, task 6.1): `xce-deploy-ha`'s `deploy-ha.sh` silently starts a REAL worker daemon (`xcecd_virtual`) whenever one happens to be built alongside the controller** (its own pre-existing opt-in-by-presence convention, predates resource-value-verification) -- a plain `/xce-build all` leaves `xcecd_virtual` present, so `xce-test-resources`'s own deployment picked one up unconditionally, contradicting Requirement 2.1 ("no real worker daemon process required") and risking bug-1572 (scheduler crash on a resource's 2nd periodic report) on a long-enough run. Fix, without touching deploy-ha.sh (out of that task's boundary): the generated `start-worker.sh` reads `${XCE_WORKER_BIN:-<real path>}` from its OWN inherited environment at run time -- export `XCE_WORKER_BIN=/bin/true` before invoking the generated `start-all.sh`, so the worker step exits immediately with zero persistent process. General lesson: when a skill's design deliberately avoids starting some component, verify what a shared deploy helper ACTUALLY launches by inspecting real running processes, don't assume your own new code is the only thing determining what ends up running.
- **bug-1578 (FIXED 2026-08-18): controller SIGSEGVs on worker disconnect if a scheduler ever connected AND that worker's resvals were pushed at least once.** Fix applied: `sched_worker_change()`'s dead `#if 0 ... #endif` block replaced with `w->host.nresvals_to_sched = 0; w->host.resvals_to_sched = NULL;` before `xlist_fini(&resvalist);` -- exactly mirrors `apiserver_worker_change()`. Verified two ways: (1) `xce-build.sh controller` clean build, 1102/1104 tests pass (2 failures are pre-existing bug-1561 overlayfs/fdatasync, unrelated); (2) live repro via `run-resource-verify.sh --keep-running` -- deployed full stack, joined synthetic worker, ran 9 `resupdate`s, disconnected (`quit`), confirmed the exact crash-trigger log sequence fired (`handle_worker_leave` -> `sched_worker_change` pushing resvals to scheduler twice) and the controller stayed alive ~2 minutes past disconnect with zero SEGV/ASAN/core-file (previously died within 1-10s every time). **UPDATE 2026-08-18 (follow-up task) -- the "no testable-static-function harness pattern exists" claim below was wrong; corrected so it isn't trusted stale in a future session.** Added `src/controller/testsuite/test_sched_worker_change.c` (new standalone binary `sched_worker_change_unit_test`, wired into meson.build mirroring `sched_push_unit_test`'s real-`sched.c`+`dataex.c`+`egress.c` linking strategy -- the exact same "drive a static hook function through the real installed `hooksys` via `xce_run_hooks()`" pattern `test_job_array_sched_push.c` already proved for the JOB hook works identically for the WORKER hook: `on_scheduler_join()` installs `sched_worker_hook_fun`, then `xce_run_hooks(&c.hooksys, XCE_CONTROLLER_HOOK_WORKER, HOOK_AFTER_EVENT, HOOK_DETAIL_WORKER_DEAD, &c, w)` reaches `sched_worker_change()` exactly as `handle_worker_leave()` does in production). Test 1 asserts `w->host.resvals_to_sched==NULL && nresvals_to_sched==0` after the hook fires. Test 2 is the stronger ASAN-strength proof: it calls the REAL production encoder `encode_Worker()` (`controller.capnp.c`, linked for free via `libxce_dep` -- no need to link `persist.c` at all, since `persist.c`'s own `capnp_encode_worker()` is just a 20-line pass-through to it) directly on the worker right after the hook fires, exactly what `persist_worker_hook_fun` does on disconnect in production. RED/GREEN-verified live: temporarily re-wrapping the fix in the original dead `#if 0 ... #endif` reproduced a genuine ASAN heap-use-after-free (`encode_NameValue_list`, `controller.capnp.c:3408`, freed by `xlist_fini<-sched_worker_change<-sched_worker_dead<-sched_worker_hook_fun`) -- then the approved fix was restored (confirmed byte-identical via `git diff` to the pre-revert state) and both tests went GREEN. Gotcha hit along the way: `makeResource()`/`makeString()` (`resobj.c`/`xstring.c`) need the global xstring intern table initialized first (`xstring_table_init(N)`, once, before first use) or `makeString()` returns an object whose `->name` is NULL, and the very next `key_add()`/`hash_string()` call SIGSEGVs on the NULL string -- easy to miss in a fresh standalone test binary since no other linked test file has already called it process-wide (existing precedent: `test_worker_join.c`'s own `RUN_TEST` call site comments this exact requirement). Original root-cause notes retained below. `sched_worker_change()` (`src/controller/sched.c`) aliases `w->host.resvals_to_sched`/`nresvals_to_sched` directly to a local `xlist_t`'s backing array, pushes it, then `xlist_fini()`s that array WITHOUT resetting the alias first -- the correct reset lives in the SAME function but is wrapped in a dead `#if 0 /*FIXME*/ ... #endif` (commit acfbec875, 2026-03-03 -- predates resource-value-verification by ~5 months; a later commit e5fde7b5 "fixed" this but edited code INSIDE the already-dead block, so its fix silently never ran). Leaves `w->host.resvals_to_sched` dangling for the worker's whole remaining lifetime. On disconnect, `handle_worker_leave -> mark_worker_unavail -> xce_run_hooks -> persist_worker_hook_fun -> capnp_encode_worker -> encode_Worker -> encode_NameValue_list -> encode_NameValue` dereferences the freed array -> SIGSEGV (si_addr=NULL). Only reachable once a scheduler has connected at least once (`on_scheduler_join()` registers `sched_worker_hook_fun` on a one-shot `static hook_installed` gate) -- a solo controller or controller+apiserver-only never triggers it, which is why the original A/B isolation matrix (lone controller alive 90s+, full stack + xce_test never run alive 100s+) didn't catch it: the missing variable wasn't "was xce_test run" but "did a scheduler ever join AND did that worker's resvals get pushed." Sibling `apiserver_worker_change()` (apiserver.c) does the reset correctly (not `#if 0`-wrapped) -- mirror that exact pattern for the fix, do NOT just uncomment the dead `free_name_val_list()` call (it already frees the array AND nulls the field itself; the very next `xlist_fini()` line would then double-free the same pointer -- almost certainly why the original author reached for `#if 0` instead of deleting it). Fully outside resource-value-verification's task boundaries (sched.c, untouched by tasks 2.1/2.2/6.1) -- tasks 2.1/2.2 are innocent, no re-review needed. See buglog.json bug-1578 for the full backtrace and fix diff.

- **`do_update_resvals()` (`src/controller/opupd.c`) fans out the FULL, UNFILTERED submitted resource list to the scheduler AND apiserver hooks BEFORE (and still, after) `update_resval_list()` does its own per-item validity check — the controller's own `w->host.resvals` write is filtered (rejects unregistered names via `key_find(c->resmgr,...)`), but the HOOK_AFTER_EVENT payload `sched_resource_hook_fun`/`apiserver_resource_hook_fun` (`sched.c`/`apiserver.c`) act on is the SAME raw list regardless.** Found live during resource-value-verification task 8.1 (bug-1583, NOT fixed): injecting one deliberately-unregistered resource name showed the scheduler's `core_events.c` independently re-validates against its own resource table and discards+WARNs on an unknown name, but apiserver's `OnResourceLocationUpdated` (`handler.go`) has zero such validation and unconditionally `store.ResourceLocations.Put()`s it — a genuine 3-way inconsistency (controller rejects, scheduler rejects, apiserver silently stores forever). General lesson: when a controller hook fans a raw request payload out to multiple downstream consumers, don't assume upstream validation (even a REJECT logged elsewhere in the same request) actually filtered what those consumers received — check each consumer's own hook function for whether it re-validates or blindly trusts the payload.
- **When a live crash won't reproduce under gdb (even a direct `gdb --args` child launch, which sidesteps this sandbox's `ptrace_scope=1` attach restriction), suspect gdb's ptrace overhead itself is masking a timing-sensitive bug rather than concluding "not reproducible."** bug-1578: a 100%-reproducible (bare launch) controller SIGSEGV did NOT reproduce within 20s under gdb for either the ASAN or debug(-O0) build, for both a plain `run` and a `--batch` launch. Fix: skip gdb entirely -- compile a tiny `LD_PRELOAD` shared library (constructor installs a `sigaction` handler with `SA_ONSTACK`+a pre-allocated altstack for SIGSEGV/SIGABRT/SIGBUS/SIGILL/SIGFPE that calls `backtrace()`+`backtrace_symbols_fd()` to stderr, then `signal(sig, SIG_DFL); raise(sig)` to preserve normal termination semantics) and launch the target with `LD_PRELOAD=./crash_handler.so <normal invocation, no wrapper>`. Near-zero runtime overhead (one constructor + a couple `sigaction()` calls), so it does not perturb thread scheduling/timing the way ptrace does -- reproduced the exact same crash reliably and captured a full backtrace. Resolve the raw addresses with `addr2line -e <binary> -f -C -p <offset...>` using the `(+0xNNN)` MODULE-RELATIVE offsets `backtrace_symbols_fd()` prints (not the raw runtime addresses, which include the PIE ASLR base and won't resolve against the on-disk binary directly).

- **Testing a bash function that both populates caller-scope globals AND prints via a `log()`-style echo (task 8.2, `aggregate_verdict()`): never capture its output with `$(...)` command substitution if you also need the globals it set.** Command substitution forks a subshell in bash — any associative-array/variable writes the function makes are lost the instant the subshell exits, even though the function's exit code and printed text survive. Fix: call the function directly with plain `>` file redirection (`fn ... > "$tmpfile" 2>&1; rc=$?`), which does NOT fork a subshell, then read `$tmpfile` separately for text assertions while the just-set globals remain live in the current shell. This is exactly `.claude/skills/xce-test-resources/verdict-logic-selftest.sh`'s `run_agg()` helper.
- **This codebase's established pattern for a persistent bash-tooling regression test is `.claude/skills/xce-test-ha/soak-verdict-selftest.sh`**: name it `<subject>-selftest.sh`, `source` (not execute) the target script to get its functions in scope, use a `check DESC EXPECTED ACTUAL` string-equality helper tallying a `fails` counter, end with a `SELFTEST PASS`/`SELFTEST FAIL (N failed)` line and exit 0/1. `run-resource-verify.sh` itself is written to support this — it has a `[[ "${BASH_SOURCE[0]}" == "${0}" ]]` guard at the bottom specifically so sourcing it never runs `main()`/starts a live deployment. Reused for `verdict-logic-selftest.sh` (task 8.2).

- **Sandbox capability update (2026-08-19): `sudo` now works passwordless here** (`sudo -n true` succeeds) — this unblocks anything previously gated on "this session/skill cannot sudo" (e.g. real `xcecd`, `worker/src/main.c`'s `check_user()`). Still true, and load-bearing for anything assuming a real production host: this sandbox has **no systemd, no dbus, no `/run` at all** (`ls /run` → No such file or directory; PID 1 is `s6-svscan`; no `dbus-daemon`/`systemctl` even under `nix develop`). Root access and a real init/dbus stack are independent axes — don't assume sudo working means systemd-dependent things (job dispatch via `org.freedesktop.systemd1.Manager.StartTransientUnit`, anything needing `/run/dbus/system_bus_socket`) will also work. `/run/systemd/system` existing is the standard `sd_booted(3)`-style signal for "a real systemd instance is running as this host's manager" — used in `src/tools/deploy-ha.sh` to auto-pick `xcecd` (real) vs `xcecd_virtual` per host (`XCE_REAL_WORKER=0|1` overrides).
- **A root-owned file created by a daemon under `sudo` is not automatically readable by the invoking non-root user** — confirmed live (bug-1592): real `xcecd`'s main log file comes out `root:root` mode `640` even though the SAME binary explicitly `chmod(..., 0666)`s its domain socket and its `xcecd_worker.log` sibling. Don't assume "the daemon already chmod's some of its output files permissive" generalizes to all of them — check each file's actual mode after a live run before writing a verification command that reads it as the invoking user; prefix with `sudo` where it doesn't.
- **The "rule out a concurrent competing build/soak" caution (line 16 above) includes SIBLING AGENTS in your own Claude Code session, not just unrelated external processes** (2026-08-23, scheduler-dispatch-commit task 3.1 GREEN verification). A peer subagent within the same session independently launched its own `xce-build.sh controller asan xce.build.green31 --incremental` + `meson test` against the exact build dir this session was told to use exclusively, overwriting the live `meson-logs/testlog.txt` in place mid-investigation. Identify this via `ps -o pid,ppid,cmd` to find the process tree, then `tr '\0' '\n' < /proc/<pid>/environ | grep CLAUDE_CODE_SESSION_ID` — a match against your own session's ID (visible in your own scratchpad path) confirms it's a sibling agent, not an unrelated build. Mitigation: copy/preserve any log file you need as evidence (e.g. `sed`/`python3`-clean it into your scratchpad) the MOMENT you capture it, before doing further slow diagnostic work that gives a peer time to overwrite the live file in place.
- **`controller_unit_test` TIMEOUT (300s SIGTERM, `Ok:4 Fail:0 Timeout:1`) is bug-1429, a pre-existing, NOT-yet-root-caused, systemic hang** — now reproduced 3 times across 3 completely unrelated diffs (delaymgr/persist.c changes; a pure version-string change; scheduler-dispatch-commit's job.c change) and 3 different hang locations each time (test_demotion_replay_yields_committed_only; the test_configack_* group; test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs). Before suspecting your own diff caused a hang here: (1) check whether the hanging test is even in a file your diff touched — if not, that's strong evidence against causation; (2) `/proc/<pid>/task/*/wchan` on the hung process (gdb attach is blocked by sandbox ptrace restrictions) — both/all threads in `futex_do_wait` confirms deadlock-consistent, not CPU-bound; (3) re-run the already-built binary directly (bypassing meson's 300s cap) with a generous `timeout` — if it reproduces in total isolation with zero concurrent activity, that rules out resource contention as the cause too. Update bug-1429's `occurrences`/`last_seen`/`fix` narrative rather than filing a new bug — do not conclude "regression in my diff" from this signature alone.

- **xce_resource_t (xce/types.h) has a `flags` bitmask (XCE_RESF_BUILTIN/DYNAMIC/GLOBAL/SHARED/EXTERNAL/RELEASE/DEFAULT_HOST/CONSUMABLE/LIMIT) distinguishing a genuinely cluster-wide resource (GLOBAL/SHARED, stored ONCE in `core->sharedInst`, a zarray_t indexed by the resource's `idx` -- the same pool RESVALS_FULL/RESVALS_DELTA event handling and snapshot save/load already treat as authoritative) from a host-based/per-worker-distributed resource (stored per-worker in `w->resvals`/`w->resInUse`).** Before this session, `repl_info_resource_one()` (scheduler REPL) conflated the two -- aggregated total/available across every worker's resvals for EVERY resource regardless of kind, a fabricated number for a host-based resource (cpus/mem) and redundant with `info worker`'s own correct per-host display. When adding any new resource-value display/log code, check `r->flags & (XCE_RESF_GLOBAL|XCE_RESF_SHARED)` first to decide which pool (`core->sharedInst` vs per-worker `resvals`) is the correct source of truth -- see bug-1593.
- **`repl_info.c` (scheduler REPL state-inspection dumps) was already OVER this project's 800-line cap (883 lines) before this session even started** -- accumulated from three prior sessions' bug fixes (bug-1587/1590/1591) each adding a little without ever triggering a new sibling-file split. A task-12.5 rewrite of the resource-display functions brought it down to 856, still over. Flag this to the user rather than silently ignoring it or unilaterally doing a 4th-generation file split (repl.c -> repl_break.c/repl_info.c -> repl_params.c precedent) without being asked -- a split touches meson.build, test_sim_isolation.c's per-file scan macros, AND every white-box test TU's #include list (real, mechanical work, not worth taking on speculatively).
- **A `repl_split_tokens()`-style whitespace tokenizer with zero quote-awareness silently breaks any downstream parser that requires a whole-string-anchored regex match** (e.g. `parse_jobid()`'s `pmatch[0].rm_so==0 && pmatch[0].rm_eo==len` check) when a user quotes an argument out of habit (common when copy-pasting a value from another command's own output). The numeric fallback path (`strtoull()`) then silently parses the leading quote character as "not a digit" and returns 0 rather than erroring -- producing a confusing wrong-answer, not a clear parse failure. Worth checking for in any REPL/CLI tokenizer this codebase adds. See bug-1596.

- **A pointer returned by `makeString()` (rtti-managed, `lib/xstring.c`) must always be released via `dereference()`, never `FREEUP()`/plain `free()`** — `makeInstance()`/`objAlloc()` returns an offset into a combined header+data block, not the block's own `malloc()`-returned base address, and the string is refcounted (freed only when the refcount hits zero), not directly owned. bug-1600 (2026-08-19, production, not a test fixture — see also bug-1503's earlier test-fixture instance of the identical mismatch): `lib/group.c`'s `makeGroup()` "existing group" update branch (hit every time a config-push re-applies a group name that already has members, e.g. any retried/duplicate config push — no thread concurrency required) called `FREEUP(m->u.name)` on a `makeString()`-allocated field, while the SAME FILE's `group_killer()` destructor released the identical field correctly via `dereference(m->u.name)`. This caused the real, reproducible (8 confirmed core dumps) `xce_scheduler` `free(): invalid pointer` SIGABRT crash during controller-failover config-push retries. **When adding or auditing any free/release path for a field populated by `makeString()`, grep the same file/struct for an existing destructor and match its release call — don't assume `FREEUP()` is safe just because it's used for adjacent plain-`calloc`'d fields in the same struct.**
- **`lib/group.c`'s `makeGroup()` "existing group" update path takes only `RTTI_LOCK_READ` (a genuine shared `pthread_rwlock_rdlock`, confirmed via `kvmgr.c`) while mutating shared group member-list state, and the scheduler's entire config-push handler chain (`sched_init_config()` → `plugin_config_update()` → ... → `makeGroup()`) holds no serializing lock at all — while `scheduler_req_handler()` genuinely runs on whichever of the scheduler's 4 vcpu worker pthreads picked up that vtask (`vtaskmgr_init(..., 4)`).** `xce_ctx_reconnect()` reuses the same `ctx->ep` struct and `endpoint_init()` unconditionally resets `ep->busy=0`, so a reconnect racing with a still-running config-push handler can let a second, concurrently-dispatched config-push run `makeGroup()` on the same group object at the same time as the first. This is a REAL latent concurrency bug distinct from bug-1600's allocator mismatch (which alone was sufficient to explain the observed crash and was the one fixed) — flagged as a follow-up, not yet fixed; would need a write-lock scope redesign since `init_group_by_config()`'s callees (`get_group()`) take their own internal read lock, so naively holding a write lock across the whole function would self-deadlock on the non-recursive rwlock.

- **`apply_committed_entry()`'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON dispatch case (`persist.c`) never stamped `start_time`/`finish_time`/`run_time`** — it only ever called the generic, state-only `apply_job_update_transition(j, state)` (2026-08-19, `xcecli joblist` showed `StartTime`=0 for every RUN/DONE/EXIT job, `FinishTime`=0 for DONE jobs). Fixed additively (user's explicit steering: "don't change the logic, just add extra information") by keying a `time(NULL)` stamp on `wal_type` right after `apply_job_update_transition()` succeeds, still under the same `lock(j, RTTI_LOCK_WRITE)` — `WAL_JOB_START` → `start_time`, `WAL_JOB_FINISH` → `finish_time` + `run_time = finish_time - start_time`. No change to `apply_job_update_transition()`'s signature or the dispatch structure. Mirrors what the richer (currently-unreached-on-this-path) `apply_job_start_transition()`/`apply_job_finish_transition()` cores in `job.c` already do for their own callers.
- **`test_config_repl.c`'s `test_conf_trim_keeps_only_n_versions`/`test_conf_get_saved_tag_no_config` failing with the exact same Expected/Was values plus a burst of `persist: flusher fdatasync FAILED (errno=5 Input/output error)` lines is bug-1561, a confirmed overlayfs-sandbox `fdatasync` artifact — 6 occurrences now across unrelated sessions/diffs, never once traced to a real code change.** If a `/xce-build controller` run shows exactly these 2 failures with that exact error pattern, it's environment noise, not something your change caused — don't spend time re-investigating, just note it and move on (though still bump bug-1561's `occurrences`/`last_seen` per buglog protocol).

- **Testing an "on a clean tree" scenario against a shared repo that's under continuous concurrent background editing (other agent sessions, OpenWolf hooks, etc.) cannot rely on the ambient working tree's `git status` ever actually being clean.** dirty-content-fingerprint task 6.1 (`src/tools/test-dirty-fingerprint-distinguishability.sh`, 2026-08-21) solved this by creating its own throwaway, fully-isolated `git worktree add --detach <tmp-dir> HEAD` before doing anything else — a fresh linked worktree checkout is clean by construction (zero uncommitted changes) and is a completely separate directory tree, so concurrent activity in the original shared worktree can never touch it. Bonus: `src/tools/xce-build.sh` resolves its own `SRC_DIR` from `$0`'s location and `cd`s there itself before running any `git` command — so simply invoking the ISOLATED worktree's own copy of `xce-build.sh` (not the shared tree's copy, and regardless of the caller's own cwd) automatically scopes every git command inside it (rev-parse/status/diff/ls-files) to the isolated worktree, with zero extra `-C`/`cd` bookkeeping needed by the caller. Clean up with `git worktree remove --force <dir>` (plain `rm -rf` alone leaves stale worktree metadata — needs a `git worktree prune` fallback too). General pattern: reach for an isolated `git worktree` (not `git stash`, not hoping the ambient tree happens to be clean) any time a test genuinely needs to observe a controlled clean/dirty state transition in a repo that isn't exclusively yours for the run.
- **A `local a="$1" b="$a/something"` — two assignments in ONE `local` statement where the second references the first — is a real bug under `set -u`, not just bad style (bug-1615, found live writing the script above).** Bash expands every word of a single `local` command before ANY of that command's assignments take effect, so `$a` in the second word's value expands against the OLD (not-yet-declared) `a`, producing `iso_src: unbound variable` (or silent empty-string under plain `set -e` without `-u`). Fix: split into two separate `local` statements on their own lines. Same family as the earlier-logged `fp="$(cmd)"` under `set -e` gotcha (cerebrum, `xce-build.sh` dirty_fingerprint note) — another case where a single-line-looking assignment has a non-obvious evaluation-order trap under bash's strict-mode flags.

- **`builtins.currentTime` is entirely unavailable ("attribute 'currentTime' missing") during a flake's default pure evaluation — not merely fixed to a dummy value, the attribute is removed from `builtins` outright.** Confirmed via `nix eval` both ways: a bare `nix eval .#pkg.someAttr` (or `nix build`) referencing `builtins.currentTime` anywhere it gets forced fails at eval time; `nix eval --impure .#pkg.someAttr` succeeds and returns the real epoch. This is unrelated to the derivation *build* sandbox (`sandbox = false` in this machine's nix.conf) — it's Nix's flake-level pure-eval restriction on impure builtins, and it applies regardless of where in the file the reference sits (laziness only delays the error until the value is actually forced, e.g. by string-concatenating it into `mesonFlags`). `self.shortRev`/`self.dirtyShortRev` (the existing `commit_id` mechanism) are NOT affected — those come from flake input/git introspection that Nix computes specially even in pure mode, not from an impure builtin. **RESOLVED (bug-1603, 2026-08-20): task 2.2's first attempt used `builtins.currentTime` in `src/flake.nix` per research.md's original decision, and was REJECTED by review because `nix build .#xcecore_debug` (no flag, the documented canonical command) failed at evaluation exactly as predicted above.** Fixed by moving the computation entirely out of Nix evaluation and into each derivation's own build phase instead — see the `preConfigure`/`mesonFlagsArray` entry below. Nix-eval-time impure builtins should be treated as generally unusable for anything that must survive a plain `nix build`/`nix eval` (no `--impure`), full stop — don't reach for `builtins.currentTime` (or similar) in a flake output again; use a build-phase shell hook.
- **nixpkgs' `meson.hook`/setup-hook.sh (`mesonConfigurePhase`, confirmed by reading the actual hook source in the locked nixpkgs tree at `pkgs/by-name/me/meson/setup-hook.sh`) runs `runHook preConfigure` FIRST, then builds its `meson setup` invocation via `concatTo flagsArray mesonFlags mesonFlagsArray`** — so a Nix derivation's `preConfigure` shell-hook attribute can compute a value at BUILD time (inside the sandbox, via stdenv's own bundled coreutils — no extra `nativeBuildInputs` needed) and `mesonFlagsArray+=("-Dfoo=$val")` it, and that flag reaches the real `meson setup` command line, all within the same shell process (stdenv's generic builder runs every phase/hook in one continuous bash script, no subshells between them) — confirmed `foo+=(...)` on a variable never declared as a Nix attribute (so genuinely unset, not just empty) still works fine even under stdenv's `set -eu` (`bash -c 'set -eu; foo+=("x"); echo "${foo[@]}"'` succeeds, auto-vivifies as an indexed array). This is the general pattern for "a nix package needs a value only knowable at build time, not eval time" (e.g. build-time-versioning task 2.2's `build_time`, fixed via this exact mechanism in `xce.nix`/`xcecli.nix` after `builtins.currentTime` broke pure-eval — see the entry above). Live-verified: `nix build .#xcecore_debug`'s build log (`nix log <drv>`) showed `mesonConfigurePhase flags: ... -Dbuild_time=2026-08-20T09:13:29Z` and meson's own summary printing `build_time : 2026-08-20T09:13:29Z`.
- **nixpkgs.lib has no epoch-seconds→ISO8601 formatter** (checked `lib/*.nix` in the locked nixpkgs source tree directly — no `strings.nix`/`trivial.nix`/etc. function does this). This is now moot for `build_time` specifically (see above — it's computed via shell `date -u` in a build-phase hook, not in Nix expression code at all), but still true in general for any future nix-side ISO8601 need.
- **`xce-build.sh` defaults `BUILD_DIR` to the same path (`xce.build`) regardless of component, and its non-`--incremental` `setup_build()` does `rm -rf "$BUILD_DIR"` + fresh `meson setup` on every invocation — two concurrent `xce-build.sh` runs for DIFFERENT components in the SAME working directory race on that shared path.** (bug-1604, 2026-08-20). Live-confirmed twice in one session: (1) a `scheduler asan` run's initial `rm -rf` hit `rm: cannot remove '.../controller_unit_test.p': Directory not empty` because a second, independently-running `xce-build.sh controller asan` process (same cwd, different session) was actively writing into `xce.build/` at that instant — simply retrying once the other process's own clean pass finished resolved it. (2) minutes later, the freshly-verified `xce_scheduler` binary (checked present + fresh mtime, `--version` run successfully) had VANISHED from `src/xce.build/scheduler/` — the same concurrent controller build's own `rm -rf` had wiped and was regenerating `xce.build/` for its component, silently invalidating the just-verified scheduler binary out from under the caller. **Before trusting any non-incremental `xce-build.sh` binary/mtime to still be valid, or before starting a clean build in a worktree another session might also be using, check `ps aux | grep xce-build.sh` for a concurrent run — different components sharing one worktree's default `xce.build` is a real, reproducible race, not environment flakiness (this was initially misdiagnosed as an overlayfs artifact before the concurrent process was found).**

- **Live smoke-testing apiserver's `-version`/startup-log output (build-time-versioning task 4.4) needs a real topdir, but a full `xce-deploy-ha` deployment is overkill.** Recipe: `mkdir -p $TOPDIR/config $TOPDIR/logs` then copy the four shipped templates `src/frontend/apiserver/config/{app,logger,auth,rbac}.yaml` into `$TOPDIR/config/` verbatim, then `timeout 10 xce_apiserver -d $TOPDIR`. The shipped `app.yaml` template has a live `backend: {address: localhost, port: 9901}` block with no controller actually running there, so the process logs `Attempting backend connection` → `WARN All configured backend candidates failed` → `ERROR Failed to connect to backend` and exits 1 within ~10ms — this is fine and expected for verifying only the startup log's FIRST lines (logger init, before backend connect is even attempted); no live controller needed. Log lands at `$TOPDIR/logs/apiserver_out.log` (from `logger.yaml`'s `outputPath: apiserver_out.log`, joined with `$TOPDIR/logs/` by `InitLogger()`). Separately, `xce_apiserver -version` (no `-d` at all) exits 0 with zero filesystem side effects (no `logs/` dir, no config read) — confirms the `-version` early-exit genuinely runs before `PromptForTopDir()`/`InitLogger()`, not just before the visible log line.
- **This codebase's apiserver Go code has ONE idiomatic zap-logging style: structured fields via `zap.String(...)`/`zap.Error(...)` appended to the `logger.Info/Error/Debug/Warn(msg, fields...)` call, never Sprintf-style interpolated messages** — confirmed by grepping every `logger.Info/Error/Debug/Warn` call site across `controllers/`, `middleware/`, `router/` (files.go, auth.go, rbac_check.go, validatehandler.go, routers.go, permissions.go all exclusively use this pattern). Even though `encoding: console` is configured in `logger.yaml`, the fields still render as trailing JSON on the log line (already documented above) — that's fine, it's still the right convention to follow for any new log call site in this package, don't reach for `fmt.Sprintf` into the message string instead.
- **`gofmt -l`/`gofmt -d` on `apiserver/cmd/main.go` reports a PRE-EXISTING import-ordering violation (`_ "apiserver/docs"` sorted before `"apiserver/backend"` instead of after, alphabetically) that predates build-time-versioning task 4.4 and is unrelated to it** — confirmed via `git show HEAD:.../main.go | gofmt -l -` on the pre-edit blob, same violation. Don't "fix" it as a drive-by inside a task whose boundary is a single file with no mandate to touch unrelated pre-existing formatting — flag it, don't silently reformat, since an unrelated reformat in the same diff makes the real change harder to review (mirrors the existing Do-Not-Repeat entry about not running `clang-format -i` on whole existing C files).
- **scheduler-sim-decision-parity task 1.2 (2026-08-22): white-box-including `resource.c` alongside the FULL driver+sim+recording stack (not just `core.c`, unlike test_resource_accounting.c's own deliberately-narrower pairing) works and needs exactly one NEW stub beyond what the driver/sim stack already required: `sched_ctx_get_uncomit_host()`** (resource.c's `sched_try_alloc_on_host_()`'s own link-time dependency, never actually called when the test only drives the public `sched_check_host_satisfy()`/`sched_charge_account()`/`sched_commit_job_alloc()` entry points directly). Two stubs test_resource_accounting.c's OWN narrower resource.c-only pairing needs (`plugin_prepare_allocation`, `sched_fill_limit_requests`) must NOT be redefined once `plugin.c` is also in the TU (test_sim_equivalence.c's set already includes it) — `plugin.c` provides a REAL `plugin_prepare_allocation`, and `resource.c` itself defines the REAL `sched_fill_limit_requests` (not test_resource_accounting.c's stub version) — redefining either is a duplicate-symbol link error, not a harmless no-op. Also: `sched_worker_release_host_ctx` must NOT be stubbed once resource.c is included (every OTHER file in testsuite/ stubs it because none of them pull in resource.c, which defines it for real).
- **A genuine, non-fabricated "capacity denied" fixture is achievable without driving scheduler.c's real `sched_allocator()`/resreq-DSL/plugin machinery at all**: a test-owned "stand-in allocator" thread (the SAME established pattern test_sim_equivalence.c already uses to replace `sched_allocator()`, there for the different reason of avoiding real allocation entirely) can instead call the REAL `sched_check_host_satisfy()`/`sched_charge_account()`/`sched_commit_job_alloc()` (resource.c) directly against a real `xce_sched_worker_t` with a hand-set finite `resvals` capacity (added via `sched_add_worker()`, `put_resvals()` — same residx-direct-bypass-of-resmgr convention `test_resource_accounting.c`'s `make_test_account()` already established) and a fixed per-job "wanted" lookup table standing in for resreq-DSL compilation. This produces genuinely-arithmetic-derived DISPATCH/PEND decisions (not fabricated) while staying at the same "don't pull in the untested resreq-DSL/plugin layer" scope every other white-box test in this directory already commits to. `sched_check_host_satisfy(xce_scheduler_t*, ...)` itself was NOT relocated by task 1.1 (only the accounting/charge functions were) — it still needs a real `xce_scheduler_t*`, not just a core, because `unsatisfy_()`'s TRACE line reads `sched->core.resmgr` (harmless when resmgr has no matching entry: the output pointer is pre-`NULL`'d and the TRACE format string degrades to `"?"`, no crash).
- **scheduler-sim-decision-parity task 2.2 (2026-08-22): a new `static` function added to `sim_loop.c` but not yet wired into any call site (by design -- the wiring is a LATER task) triggers `-Wunused-function` at production-object compile time (`ninja scheduler/xce_scheduler.p/src_sim_loop.c.o`), but this project has no `-Werror`/`warning_level` override in `src/meson.build` (default meson `warning_level=1`, no werror), so it's a harmless warning, not a build failure -- confirmed by building that exact object directly. Don't mistake this for a real problem needing an early premature call site or a `(void)` no-op reference; just note it as an expected, temporary CONCERNS item until the wiring task lands. Also confirmed the standard "RED phase for a brand-new function with no prior runtime behavior" technique: `cp` the modified source aside, `git checkout -- <file>` to revert to HEAD (pre-implementation), rebuild+run the new test to capture a genuine `implicit declaration of function` compile-time RED failure, then restore the saved copy -- cleaner than hand-editing the diff back out and back in, and git-safe (never touches the actual uncommitted work beyond a temporary revert/restore pair on one file).
- **scheduler-sim-decision-parity task 2.3 (2026-08-22): to prove an ASAN "no leak" claim for a white-box test in `src/scheduler/testsuite/`, do NOT trust the meson-registered test run alone** — every test target in this directory is registered with `env: ['ASAN_OPTIONS=detect_leaks=0']` (meson.build) because this whole test family deliberately never tears down its `xce_sched_core_t`/fixture allocations (documented file-header convention). Re-running the standalone binary with default `ASAN_OPTIONS` (leak detection ON) is the only way to actually see leak provenance, but doing so SWALLOWS Unity's own PASS/FAIL summary from stdout entirely (LeakSanitizer's report prints, then the process exits via a path that never flushes the earlier buffered Unity output) — so run it TWICE: once as `ASAN_OPTIONS=detect_leaks=0 ./binary` for Unity's real pass/fail summary (matches the project's own convention/exit code), once as a bare `./binary` (or `ASAN_OPTIONS=detect_leaks=1`) purely to inspect leak stack traces. When triaging those traces, filter for ones whose stack passes through the code you actually changed (`grep -B3 '<your_function>' `) — the other 100+ leak reports are the SAME pre-existing "fixture not torn down" noise on every test in the file (kvmgr/groupmgr/select-parser allocations from `xce_sched_core_init()`), not signal.
- **`free_host_account()` (`src/lib/misc.c`) never frees `acct->host`** (the `strdup(halloc->host)` `make_host_account()` sets, `xcealloc.c:119`) — confirmed via the above ASAN leak-provenance technique while verifying task 2.3's reconcile-then-release round trip. Pre-existing, NOT scheduler-sim-decision-parity-specific: equally leaks on the LIVE `scheduler.c` dispatch path (`sched_commit_decision()` → `make_job_account()` → same `free_job_account()`/`free_host_account()` release chain), previously masked only by this test suite's `detect_leaks=0` convention. Logged as bug-1621, not fixed (out of boundary for task 2.3 — `misc.c` wasn't in that task's file set). Correct fix if picked up later: add `FREEUP(acct->host);` to `free_host_account()`, mirroring `free_res_account()`'s adjacent `FREEUP(acct->resname);`.
- **scheduler-sim-decision-parity task 3 (2026-08-22): `resource.c`'s `get_resvals(zarray_t*, int nres, int*, float*)` returns `nres` on success (not 0)** — confirmed by reading its body (`return nres;` at the very end). Bit a new end-to-end test (`test_sim_round_boundary_reconciliation.c`) that wrongly asserted `TEST_ASSERT_EQUAL_INT(0, rc)` after a 1-resource query; briefly looked like the task-3 wiring hadn't worked (GREEN rebuild still failed) before the real cause (a test-authoring mistake, not a reconciliation bug) was found. Logged as bug-1622. Lesson: don't assume this codebase's near-universal "0/negative" return convention applies uniformly -- `get_resvals()` is a documented exception, check the actual body.
- **scheduler-sim-decision-parity task 3 (2026-08-22): to exercise Recorded-Decision Reconciliation's REAL `make_job_account()`/`make_res_account()` (xcealloc.c) path end-to-end — as opposed to every existing resource-contention-style test in this directory, which hand-builds `xce_job_account_t` with `residx` already resolved, deliberately bypassing this — you must handle that `make_res_account()` (xcealloc.c) always sets `residx = -1`, forcing `sched_update_res_accounts()` (resource.c) to LAZILY resolve `resname` -> `residx` via `key_find(core->resmgr, resname, ...)` at charge time (falling back to auto-registering the name via `add_vres()` if not found).** Don't gamble on which index that lazy auto-registration would land on (e.g. assuming a fresh resmgr's first-ever entry gets index 0) — instead, pre-register the resource explicitly via the real `makeResource(core->resmgr, &xce_resource_t{...})` (resobj.c) BEFORE adding worker capacity, and read back the returned object's own `->idx` to use consistently for both the worker's `resvals`/`resInUse` setup (`put_resvals()`) and every later `sched_check_host_satisfy()` `wanted`/`unmeet` vector. This is the only test file in `src/scheduler/testsuite/` (as of this task) that exercises this lazy-resolution path at all — every sibling test that touches `xce_res_account_t.residx` sets it directly, sidestepping the question.
- **In any script running under `set -euo pipefail` (e.g. `xce-build.sh`), a bare assignment whose RHS is solely a command substitution (`fp="$(cmd)"`) is NOT exempt from `-e`** — if `cmd` fails, that assignment's own exit status equals `cmd`'s, and since a standalone assignment statement isn't a context `-e` treats as "tested" (unlike being the condition of `if`/`while`/`&&`/`||`/`!`), a failure there aborts the WHOLE script, not just that value's computation. Only `if var="$(cmd)"; then ... fi` (or `cmd || true`) is actually exempt. This bit dirty-content-fingerprint task 2.1's `dirty_fingerprint` computation (sha256 over `git diff HEAD` + sorted untracked file content) in `xce-build.sh`: every intermediate command (mktemp, `git diff`, the `git ls-files | sort` pipe, each `cat` in the untracked-file loop, and the final `sha256sum | cut`) had to be wrapped in its own `if`/`elif ... else fp_ok=0` check rather than trusted to propagate failure naturally, specifically so a failure (e.g. `sha256sum` missing) degrades to an empty fingerprint (Requirement 5.1's "never fabricate") instead of crashing the entire build invocation. Verified live by temporarily hiding `sha256sum` from `PATH` in an isolated scratch repo — script printed `dirty_fingerprint=` and exited 0, not aborted.
- **scheduler-sim-decision-parity task 4.2 (2026-08-22): a job that stays genuinely PEND (never dispatched, never state-changed) is re-offered to the live picker and re-PENDed EVERY subsequent round it remains unresolved, not just the round it first arrives in** — `builtin_fcfs.c`'s `pendJobs` discipline only removes a job on dispatch/state-change (its own comment: "A non-pending job is left removed and not reinserted"), so a still-PEND job simply stays in its queue's `pendJobs` and `next_job()` keeps offering it every round. Confirmed independently by `test_repl_equivalence.c`'s own existing (untouched) fixture output: job 501 (pushed before session 1, never dispatched) recurs as a fresh PEND decision in sessions 2 AND 3 too. Bit a new multi-round `--sim-verify` regression test (`test_sim_verify.c`'s `test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatch`): the round-2 fixture initially recorded only the round's one FRESH job's PEND, producing a real MISMATCH (expected vs actual position mismatch) because the still-PEND job from round 1 also reappeared, unrecorded, in the live-drained stream. Fix: any multi-round sim fixture with a job that stays PEND across round boundaries must include that job's OWN recurring PEND in every subsequent round's recorded decisions list, ordered by `queue_time` ascending (same-priority ties break by arrival order) to match `next_job()`'s real dequeue order. General lesson: don't assume a job "handled" in round N (PEND, not DISPATCH) is done being decided — only DISPATCH/state-change removes picker candidacy, so a genuinely-unresolved job is a recurring cost across every fixture round.

- **scheduler-sim-decision-parity task 4.3 (2026-08-22, FINAL task): Recorded-Decision Reconciliation's SUCCESSFUL-MATCH path is structurally unreachable from ANY continuously, organically live-driven recording (real normal-mode OR simulated) — this is a hard mathematical fact about the design, not a limitation of test-authoring creativity, and it is worth knowing before attempting a "real, long, driven recording" fixture for this feature again.** `sim_reconcile_unaccounted_jobs()` (sim_loop.c) matches a RUN-state, `job->accounts==NULL` job against a DISPATCH decision recorded in THAT SAME replayed round's own decoded ReplaySession only (each round's `decoded_session` is decoded fresh and freed immediately after that round's own processing — no historical carry-forward across rounds exists anywhere in the loop). But a job cannot be BOTH "freshly PEND when this round's live picker session starts" (the ONLY way any live picker/allocator, real or simulated, can ever produce a DISPATCH decision for it) AND "already RUN before this round's live picker session starts" (reconciliation's own trigger precondition) in the SAME round — `builtin_fcfs_on_dispatch()`/`on_job_state_change()` make these two states mutually exclusive by construction (dispatch permanently removes a job from `pendJobs`; a state-change to RUN removes it too, via the SAME `on_job_state_change` guard, whichever fires first in a batch). Confirmed BOTH by direct source read (sim_loop.c/driver.c/builtin_fcfs.c, before writing any test code) AND empirically, live: a test built with 11 "organic" jobs (arrive PEND, get denied/recur, get genuinely dispatched by a real capacity-aware allocator thread in their OWN arrival round, get confirmed RUN via a JOB_STATE_CHANGE event in a LATER round) showed EVERY ONE of those jobs' real dispatches permanently invisible during replay (`job->accounts` stays NULL forever, per Requirement 4's own documented graceful-degradation path) — only ONE job, deliberately engineered so its arrival+RUN-confirmation events AND its own "originating decision" all land in the exact same round (built via real constructors — `make_charge_account`/`sched_commit_job_alloc`/`make_dispatch_decision` — fed through the real `core->decisions` cbuf, NOT hand-crafted capnp bytes), ever showed nonzero resInUse during replay. Every EXISTING test in this feature that exercises reconciliation's success path (`test_sim_round_boundary_reconciliation.c` task 3, `test_sim_contention_parity.c`'s "Part B" task 4.1) independently arrives at the SAME resolution (hand-construct the one round where the trigger and its matching decision coincide) for exactly this reason — it is the ONLY way to reach that code path at all. Lesson: if a future task needs "a job whose historical dispatch becomes visible via reconciliation," do NOT assume dispatch-now-confirm-a-few-rounds-later will work for ANY organically-arriving job — it structurally cannot, regardless of round count or job count; the trigger and its match must be engineered into the identical round, unavoidably requiring at least a minimally test-authored decision object for that one job/round. See buglog.json bug-1623 for the live test-authoring mistake this caused (initial checkpoint values assumed churn jobs' dispatches WOULD be reconciled; they never were).

- **(2026-08-22) Normal-mode resource accounting does NOT survive a round boundary for a dispatched-but-not-yet-worker-confirmed job — a real over-commit/double-booking risk, distinct from and NOT fixed by `scheduler-sim-eager-reconciliation`.** `sched_commit_job_alloc()` (`scheduler.c:1446`, called from `sched_commit_decision()`'s DISPATCH branch) synchronously charges `job->accounts` into `core->sharedInUse`/`worker->resInUse` at dispatch-decision time, regardless of `job->state` (still PEND — RUN only sets later via the separate `schedStartJob` notify path). But EVERY round, `xce_core_begin_session()` (`core_session.c:66-108`) unconditionally zeroes those same totals, and `sched_account_jobs()` (`scheduler.c:1258-1292`) only re-derives them from jobs where `JOB_IS_RUN(job->state)` is true (`scheduler.c:1276`) — silently skipping a still-PEND-but-dispatched job. Normal-mode rounds run on a 1-second cadence (`main.c:453-460`, `while(1){ ...round...; sleep(1); }`), so any dispatch-to-confirmation latency over ~1s (ordinary for real worker RPC+fork/exec) wipes that job's charge at the very next round and doesn't restore it until confirmation lands. The allocator's admission check (`sched_check_host_satisfy_`/`sched_check_shared_satisfy_`, `resource.c:252-370`, called from `seqjob_allocator.c:130`) reads exactly this same `resInUse`/`sharedInUse` — no separate reserved/committed pool exists (`xce_sched_job_t` only has `state`/`allocs`/`accounts`, `sched/types.h:96-116`) — so the resource looks free again and can be double-granted to a different job. Logged as bug-1624, NOT YET FIXED. `scheduler-sim-eager-reconciliation`'s boundary explicitly excludes normal-mode behavior changes, so this needs its own spec.

- **(2026-08-23) `scheduler.c`'s `sched_update()`/`sched_job_state_change()` (the notify-driven job-state-apply path) is DEAD CODE in the live production driver — do not reuse or trace through it when designing scheduler changes.** `main.c`'s production loop calls only `xce_sched_driver_run_normal_mode_round()`, which drains controller-originated events via `xce_sched_driver_translate_pending_events()` (`driver.c`) → `xce_core_push_events()` → `core_events.c`'s typed-event switch (e.g. `XCE_CORE_EVENT_JOB_STATE_CHANGE`) — a separate, independently-maintained mirror of `scheduler.c`'s logic. `handler.c`'s notify handlers still enqueue onto the same underlying `core->pend_updates` dataq both drain paths read from, so the ENQUEUE side is shared/live, but only `core_events.c`'s DRAIN/APPLY side is ever actually invoked. Confirmed via direct trace of `main.c`'s `while(1)` loop body (found during `scheduler-dispatch-commit` design). Any future scheduler design work touching state-change application must target `core_events.c`, not `scheduler.c`'s `sched_update()`.

- **(2026-08-23) Iterative /kiro-validate-design loops (fix → re-review → repeat) keep finding NEW, escalating-severity issues well past round 1 — don't stop at the first clean-looking pass.** For `scheduler-dispatch-commit`'s design, round 1 found 3 issues (a locking claim that didn't match the code, a dead field, a requirements-wording mismatch); fixing those and re-reading surfaced a genuine double-notify race in round 2 (one function unconditionally sent the same notify type on both success and failure); fixing THAT surfaced an rc-value classification ambiguity + a missed 6th failure site in round 3-4; fixing THAT surfaced a 4-site reference-count leak (bug-1627) and an unconditional pre-failure-check commit in the same function; a final full re-read caught the WAL apply logic only covering one of two apply functions (`apply_wal_entry()` recovery path vs `apply_committed_entry()` live path). Each fix changed the code enough to expose a DIFFERENT class of bug the previous read hadn't looked for. Lesson: budget for 4-5+ rounds on a design touching genuinely concurrent/durable code, not 1-2 — and re-read the FULL file after each fix, not just the section just edited, since fixes ripple (count/terminology drift across sections was itself a repeat source of new-looking "issues" that were really just staleness from earlier edits).

- **(2026-08-23) Weekly status reports are an established, continuous series in `reports/`, and their windows are defined by CONTINUITY, not by the calendar.** Filenames are `mmmDD-mmmDD-YYYY.md` (lowercase month, zero-padded day, end date's year) but the ranges deliberately drift — `jun29-jul04` (Mon–Sat), `jul19-jul26` (8 days), `aug08-aug15` (Sat–Sat) — because each report starts the day after the previous one ended and runs to the last completed day. **Always `ls reports/` and read the latest report's end date BEFORE picking a window**; the naive "last complete ISO week" is usually wrong. Concrete case: on 2026-08-23 the ISO default resolved to Aug 10–15, but `aug08-aug15-2026.md` already covered it — the genuine gap was Aug 16–22, so the default would have written a duplicate while leaving a whole week unreported. The current stable format (last 3 reports) is: `# XCE Project Status — <range>` → `## TL;DR` → `## Commits by day` (per-day `###` subsections each with a *theme*, not a commit dump) → `## Bugs opened/fixed this window` → `## Spec status changes` → `## Stats` → `## Open threads / to-do heading into next session` (numbered, carried forward and explicitly closed out in the next report). Two metric traps: quote **non-`.wolf`** line counts as the headline (`.wolf/token-ledger.json` churn made one week read +288,983 raw vs +28,657 real), and split `chore(wolf)` checkpoint commits out of the commit count (89 of 192 in one window).

- **(2026-08-23) `.wolf/buglog.json` currently has 39 DUPLICATE IDs from a counter restart on Aug 15–17, 2026 — do not trust `bug-NNN` as a unique key, and do not infer "old bug re-touched" from a low ID with a recent timestamp.** New entries were appended reusing the already-taken `bug-875`–`bug-949` range; e.g. `bug-908` is *both* a 2026-06-21 `raft_sim.c` guard-clause bug and a 2026-08-16 `config-upgrade-snapshot/spec.json` refactor note. When counting bugs for a report or analysis, split IDs at the prior window's max (genuinely-new vs collision) rather than trusting min/max. The Stop hook flags duplicates every turn until this is repaired, and repair needs a git-history-informed merge (identify the newcomer of each pair, renumber above the current max). This is the exact failure the OpenWolf bug-logging rule calls "unrecoverable without a git-history merge" — when appending, always compute the next ID as `max(existing numeric IDs) + 1`, never from a local or remembered counter.
- **(2026-08-23, scheduler-dispatch-commit task 5.2) A design.md state-model claim of "no reversal needed on failure" can be silently scoped to only the FIRST occurrence of an event, and a later requirement in the SAME design can anticipate a repeat occurrence the claim doesn't cover — don't trust such a claim without checking it against every scenario the design's OWN other requirements construct.** `dispatch_a_job_()`'s (`job.c`) four retryable-failure branches never write `job.state`/`job.allocs`, justified by design.md's "Controller Dispatch Commit" text: "on RPC failure, state is untouched (still PEND — never entered RUN, so no reversal is needed, per the Research Log finding)" — true ONLY for a job's first-ever dispatch attempt. Requirement 4.3's own text ("consecutive... failures with no intervening successful dispatch") and task 3.1's own observable-completion criterion ("a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once") explicitly anticipate a job that has ALREADY succeeded once (RUN, allocs committed) failing again later — a scenario where "never entered RUN" is false, so "no reversal is needed" doesn't hold. Building a real, end-to-end fail→succeed→fail test (not a manual-field-poke unit test — those all vacuously start-and-stay-PEND, which is exactly why 4 EXISTING passing tests never caught this) exposed that `job.state`/`job.allocs` genuinely never revert on this path — logged as bug-1631 (unfixed; fix requires a job.c change, out of a test-only task's boundary). The counter-reset itself (Req 4.3, task 1.3/3.1) IS correctly implemented and IS proven end-to-end — this is a narrower, separate gap (Req 4.1/4.4/4.5) than the one the counter test was written to prove. **General lesson: when a design document asserts an invariant with a specific justifying scenario ("never entered RUN"), check whether ANY other requirement in the same design describes a scenario that would falsify the justifying premise — don't assume the invariant holds universally just because the design states it as if it does.**
- **(2026-08-23) Technique: driving a real fail→succeed→fail sequence through a black-box test of a static dispatch function that never gates on the target object's own state** — `dispatch_a_job_()` re-resolves the worker via a fresh `key_find()` on every call and never checks `job.state` itself, so toggling the SAME `worker_t*`'s `->ep` field between a "broken" endpoint (task 3.3's `dfr_setup_broken_ep()` — real socket, unregistered in iomgr's epoll, `xce_startjob_req()` fails deterministically and repeatably) and a "working" one (task 3.1's zeroed/no-iomgr `endpoint_t` — enqueue always succeeds) between consecutive calls against the identical job/worker pair is sufficient to construct a real fail→succeed→fail cycle with no second fixture and no manual field pokes. Reusable pattern for any test needing multiple, alternating real outcomes from the same call site.
- **(2026-08-23, scheduler-dispatch-commit task 5.4) `controller_test.c`'s `-t <group>` filter operates at the GROUP-function level only (`TEST(test_commit_barrier_reply)` etc.), never at the individual `RUN_TEST()` case level inside a group's own entry function** — if a hang (e.g. bug-1429) sits at an earlier `RUN_TEST()` call than the specific test you need to isolate, WITHIN THE SAME group function, `-t <that-group>` still hits the hang first and never reaches your test; there is no finer-grained filter in this harness. Confirmed live: `test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs` (bug-1429's usual hang site) is registered at `test_commit_barrier_reply.c:3639`, BEFORE `test_jobdispatch_receipt_ack_no_wal_append` at line 3649, inside the same `test_commit_barrier_reply()` function — so `-t test_commit_barrier_reply` cannot isolate the latter if the former hangs. In practice this session bug-1429 did NOT reproduce in 2 consecutive full unfiltered `meson test --suite controller` runs (both completed clean in ~38s, 1133 tests, only the 2 known bug-1561 failures) — reinforces that it's genuinely non-deterministic/timing-dependent, not something to route around by default; try a full unfiltered run FIRST (fast when it doesn't hang) before reaching for `-t` group-exclusion workarounds.
- **(2026-08-23, scheduler-dispatch-commit task 5.4) bug-1598's "never-joined background thread + stack-local test fixture" ASAN stack-use-after-return pattern (`sim_picker_thread_fn` in `sim_loop.c`, reading a stack-local `xce_sched_sim_driver_t` after its owning test frame returns/is reused) has an UNFIXED sibling instance in `test_sim_verify.c`'s own separately-defined `run_sim_and_capture_stdout()` helper** — bug-1598's fix only converted the 3 stack-local fixtures in `test_sim_event_loop.c`; any OTHER file with its own stack-local `xce_sched_sim_driver_t sim;` calling `sched_sim_driver_run()`/`sched_sim_driver_start()` still carries the identical latent bug, non-deterministically triggered by whatever stack-layout/`MALLOC_PERTURB_` randomization a given test invocation happens to produce (fired once in a `/xce-build scheduler` run this session, then didn't reproduce across 3 standalone re-runs + 1 more full-suite re-run). Logged as bug-1632. Before trusting "the scheduler suite is at its known N/52 baseline," grep for other stack-local `xce_sched_sim_driver_t` fixtures beyond the ones bug-1598 already fixed — this class of bug is per-file, not fixed globally by one file's remediation.

- **(2026-08-25, bug-1685) `key_add()` on a kvmgr (`src/lib/kvmgr.c`'s `xkvmgr_add()`) that was created with a non-NULL `.datakiller` in its `kvmgr_params_t` ALREADY frees a rejected value for you on failure — the caller must NOT also free it.** `xkvmgr_add()` calls `vstore(kvm->data, value)` first (stores into the backing zarray), and only THEN `key_add(kvm->kvtab, key, idx)`; if that second call fails (e.g. `-EEXIST` for a duplicate key), it rolls back via `vclear(kvm->data, idx)` -> `xzarray_clear()` (`src/lib/zarray.c`), which — since the slot is in-use — fires `a->killer(a->data[idx].data)` on the just-stored value automatically. A caller whose kvmgr has `.datakiller` set (e.g. `limit.c`'s `privdata->limits`, `.datakiller = kill_limit`) that ALSO calls its own free function on `key_add()` failure double-frees — this was bug-1685's exact live SIGABRT (glibc malloc_printerr). Conversely, a kvmgr created with `.datakiller = NULL` (e.g. `job.c`'s `c->jobmgr`, per `restore_job_array()`'s own doc comment) has a no-op rollback, so THAT caller correctly must free the rejected value itself. **Rule: before writing/reviewing any `key_add()` error-handling path, check the target kvmgr's OWN `kvmgr_params_t.datakiller` at its construction site — that single field, not intuition about "who owns this pointer," determines whether the caller must free on failure.** This is the second confirmed real bug in this exact shape (job.c's `restore_job_array()` comment documents the mirror case) — worth grepping for every other `key_add(...)` call in the codebase whose target kvmgr has a non-NULL `datakiller` and confirming its own error path doesn't also free.

- **(2026-08-25, bug-1686) A lock that correctly protects a data structure DURING a single critical section does NOT protect a raw pointer CACHED across SEPARATE lock acquisitions from going stale if freed by another properly-locked call in between — mutual exclusion prevents concurrent corruption, not cross-call pointer lifetime bugs.** `builtin_fcfs.c`'s `fcfs_plugin_data_t.lock` (the bug-1611-era concurrency fix) makes every individual hook call — `on_queue_update()`, `next_job()`, etc — atomic with respect to each other; that part works. But `priv->cursor`/`priv->job_cursor` are raw `ordq_item_t*` values that PERSIST in `fcfs_plugin_data_t` across separate `next_job()` calls, each taking and releasing the lock independently. A later, individually well-locked `on_queue_update()` call (removing-then-reinserting a queue's node to re-sort it — `ordq_get` + `FREEUP(it)` + `ordq_put`) can legitimately free the exact node a cursor from an EARLIER, already-completed `next_job()` call is still resting on; the very next `next_job()` call then dereferences that freed node — a genuine heap-use-after-free, no concurrency/threading required to trigger it (single-threaded, sequential calls reproduce it deterministically, confirmed live under ASAN). Fixed by NULLing whichever cursor points at a node right before that node is freed, at every site that frees/removes a node the cursor might reference (`on_queue_update`'s reuse branch, `on_job_state_change`'s `!JOB_IS_PEND` removal, `fcfs_remove_job_everywhere`) — safe because `next_job()` already treats a NULL cursor as "exhausted" and `begin_session()` resets both fresh every round, so invalidation costs at most a one-round delay, never a lost job or a crash. **Rule: any struct field that is a raw pointer INTO a lock-protected structure, and that OUTLIVES the single critical section that produced it (i.e., read under one lock acquisition, dereferenced under a LATER, separate one), is a latent stale-pointer bug regardless of how correctly the lock itself is used — audit for "cache a pointer, release the lock, reacquire the lock later, dereference the cached pointer" as its own category, distinct from and not fixed by the mutex being correctly held everywhere.** Same file as bug-1611's original concurrency fix (`builtin_fcfs.c`) — worth re-checking any other "cursor"/"cached node"-shaped field in this codebase for the identical cross-call pattern.

- **(2026-08-25, bug-1687) This worktree can have MULTIPLE concurrent agent sessions actively editing it at once — always re-check `.wolf/buglog.json`'s max id and `git status` for files you never touched immediately before writing, not just once at session start.** Mid-session, `.wolf/buglog.json`'s max id moved from 1685 to 1686 (a different session's `builtin_fcfs.c` SIGSEGV fix, bug-1686) between my first check and my write — required a bulk `sed -i 's/bug-1686/bug-1687/g'` across 6 already-edited files (job.c, persist.c, persist.h, wal_types.h, the new test file, and a test-fixture fix) to renumber every in-code comment reference before writing the buglog entry. Also: `git status` mid-session showed `src/scheduler/src/plugins/builtin_fcfs.c`/`.wolf/cerebrum.md`/`.wolf/buglog.json` as modified by that OTHER session, and a full top-level `ninja`/meson build can fail on THEIR uncommitted, unrelated changes even when your own target component builds fine — that other session's own bug-1686 entry documents hitting the mirror image of this same collision (my job.c/persist.c changes broke ITS full-tree `ninja` build). **Never assume `.wolf/*` files or the working tree are exclusively yours in this environment; re-verify immediately before any write that depends on "current state" (next bug id, clean baseline, etc), and scope builds/tests to your own component when a full-tree build fails on files you didn't touch.**
- **(2026-08-25, bug-1687) When a WAL/wire fact type's payload grows past a previously-fixed size (e.g. adding trailing bytes to what used to be a bare fixed-size struct — the bug-1678/bug-1687 pattern), grep the WHOLE testsuite for any test helper that manually parses the raw WAL segment file and matches entries by an EXACT payload-length equality (`hdr.payload_len == sizeof(fixed_struct)`) — such a helper will silently start UNDER-COUNTING (returning 0 instead of the real count) the instant the payload legitimately grows, and if any OTHER test polls/waits on that count reaching a value, the undercounting can manifest as a genuine, reproducible DEADLOCK far downstream (all threads futex-blocked, near-zero CPU accrual) rather than a clean assertion failure at the actual call site. Found via `cbrp_count_jobstate_entries_ex()` in `test_commit_barrier_reply.c` (`hdr.payload_len == sizeof(wal_jobstate_t)` — fixed to `>=`, since jobid/state always live at the same fixed leading offset regardless of trailing bytes). This is a DIFFERENT, more dangerous failure mode than bug-1678 ever hit (that fix's own tests never counted WAL_JOB_DISPATCH via this exact helper, so it went unnoticed) — always grep for `sizeof(wal_jobstate_t)`-style exact-match parsers project-wide whenever changing a WAL fact's payload shape, don't just trust the direct callers/apply seams to be the whole blast radius.
- **(2026-08-25, bug-1689) The bug-1687 apply-seam pattern (split a shared-payload WAL fact type out of `apply_wal_entry()`'s shared `case` group into its own case, extend `apply_committed_entry()`'s existing inline `else if (wal_type == X)` branch, add a `restore_job_<type>_state()` sibling in job.c, add a `persist_submit_job_<type>_fact()` sibling in persist.c/persist.h) generalizes cleanly to a THIRD job-fact type (WAL_JOB_FINISH) with one twist: not every such fact needs Cap'n Proto.** `job_exitinfo_t` (`xce/types.h`) is a single `{int exitcode;}` with no existing capnp mapping (confirmed via grep — no `JobExitInfo` in `controller.capnp`), so the trailing payload was a small NATIVE (raw memcpy) fixed-size trailer instead of a capnp-encoded blob — no malloc needed either, since the whole payload is fixed-size (unlike WAL_JOB_START's variable-length JobPIDInfo or WAL_JOB_DISPATCH's JobAlloc). General rule going forward: before reaching for capnp on a new WAL trailing payload, check whether the struct being carried already has a capnp mapping AND has any variable-length/nested-pointer fields — if it's a small fixed POD with neither, a raw `memcpy`'d native trailer (mirroring `wal_jobstate_t`'s own style) is simpler and correct, matching this codebase's own "small native trailing struct" precedent for `wal_jobstate_t` itself. Also discovered mid-fix: `apply_job_finish_transition()` (the "richer" core that DOES set exit_info/finish_time/run_time correctly) is effectively DEAD CODE on the real production apply path today — raft-apply-decoupling's redesign routes all real WAL_JOB_FINISH mutation through the leaner `apply_job_update_transition()` (state-only) instead, so `exit_info` was never populated through the live commit-apply pipeline at all, restart or not, prior to this fix (worse than "only breaks on restart"). When auditing a "richer transition function exists but the apply seam uses a leaner one" pattern (already documented for WAL_JOB_START in bug-1687's persist.h comment), always check whether the richer function has ANY live caller at all, not just whether it's correct.
- **(2026-08-25, bug-1689) Reconfirms the bug-1687 "multiple concurrent sessions write `.wolf/buglog.json`" lesson with a fresh, independent collision — re-check the max id immediately before EVERY write, every time, not just once per session.** Checked max id at session start (1687, free), implemented the full fix using "bug-1688" throughout code comments/test file, then re-checked right before writing the buglog entry (per this project's own mandated protocol) and found bug-1688 had been claimed by a sibling session's unrelated scheduler REPL bug in the interim — required a `sed -i 's/bug-1688/bug-1689/g'` across all 5 touched files before writing the entry as bug-1689. Zero cost when caught by re-checking as instructed; would have been a duplicate-id buglog corruption if skipped.
- **(2026-08-25, bug-1687) `ptrace` is fully unavailable in this sandbox — `gdb -p <pid>` fails with "Operation not permitted" even with `dangerouslyDisableSandbox: true`, and `/proc/<pid>/task/*/stack` is also unreadable (no permission).** The only process-introspection signal available without ptrace is `/proc/<pid>/task/*/{wchan,status,comm}` (shows which kernel function a thread is blocked in, e.g. `futex_do_wait`, and its state, e.g. `S (sleeping)`) plus `ps -o pid,etime,time` (wall-clock ELAPSED vs. CPU TIME — a process with large ELAPSED but near-zero TIME is genuinely blocked/deadlocked, not slow-computing; this distinction is what first proved the bug-1687 hang was a real deadlock and not just a slow test run). To actually isolate WHICH test in a monolithic Unity binary is hanging when `-t <group>` only filters at group-function granularity (see the existing entry on this): temporarily insert an early `return;` inside the group's own driver function (e.g. `test_commit_barrier_reply()`) right after a candidate `RUN_TEST()` call, rebuild+run with a bounded `timeout N`, and binary-search by moving the `return;` — each step is a full, clean process exit (no ptrace needed) so the log's last line IS trustworthy for that step (unlike a genuinely-hung run, where stdio is fully-buffered when redirected to a file/pipe and the last visible log line can be UP TO ~4KB of buffered-but-unflushed output stale relative to where the hang actually occurred — do not trust "last line before a real hang" as the hanging test's identity without this bisection). **After finishing this kind of diagnostic bisection, grep the group's own driver function for accidentally-DUPLICATED `RUN_TEST()` calls before considering the file clean** — inserting new `RUN_TEST()` lines ahead of a temporary `return;` marker and then later deleting only the marker (not the inserted lines) leaves the original, further-down calls to the same tests still present, silently double-running them (caught here via `git diff` showing a nonzero `+RUN_TEST`/`-RUN_TEST` net and a suspiciously-inflated total test count).
- **(2026-08-25, bug-1691) `/tmp` in this shared sandbox can silently fill to 0 bytes free across many unrelated sessions, because `mkdtemp()`-based test fixtures across this whole codebase only `rm -rf` their scratch dir on the NORMAL end-of-test path — any run aborted mid-way by ENOSPC (or a crash, or a killed process) leaves its 64-128MB scratch dir behind forever, compounding: fuller disk -> more runs abort early -> more orphans -> fuller disk still.** Found live: a build+test run failed with `OSError: [Errno 28] No space left on device`, both from meson's own JSON/XML report writer AND, worse, from genuine mid-test `mkdtemp failed`/spurious `Expected Non-NULL` failures in UNRELATED test files once disk hit 0 during actual test execution — that second failure mode produces a CORRUPTED, untrustworthy pass/fail signal, not just a build-tool crash, so a run showing many unrelated failures right after an ENOSPC condition should be re-run after fixing disk, never trusted as real regression evidence. Root cause found via `ls -1 /tmp | grep -c '^xce_'`: 15,437 orphaned `/tmp/xce_*` scratch directories (~658GB) had accumulated from prior sessions. Verified none were held open by any live process first (`for p in /proc/[0-9]*; do ls -la "$p/fd" | grep <pattern>; done` — empty output = safe to delete; a live `xce-deploy-*` HA cluster's OWN workdir, e.g. `/tmp/xce-deploy-1679`, DOES show up in a live process's cmdline/cwd and must NOT be deleted). Fix: `rm -rf /tmp/xce_*` recovered 658GB instantly (359MB free -> 659GB free). **Whenever a build/test run in this sandbox fails with ENOSPC or `df -h /` shows near-0 available, check for this exact pattern before assuming it's a real disk-capacity problem or someone else's live process** — it is very likely safe, accumulated test-fixture debris, not a resource genuinely in use.
- **(2026-08-25, bug-1691) A shared-fixture footgun found copy-pasting a sibling WAL-fact test file's "drive the real production entry point through a real restart cycle" test: `persist_init(&c.persist, &cfg, NULL)` (the exact line every prior sibling test — bug-1687/1689's START/FINISH suites — uses) is WRONG for a hook-driven fact type, even though it is byte-identical to what those working sibling tests do.** `persist_start()` only registers `persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun` on `p->cluster->hooksys` when `p->cluster != NULL` (persist.c). DISPATCH/START/FINISH's production functions call `persist_submit_job_*_fact()` DIRECTLY, bypassing the hook system entirely, so those sibling tests' `NULL` 3rd-arg is harmless. But CKPT (like bug-1690's RUSAGE) is HOOK-DRIVEN: `on_job_ckpt()` only ever calls `xce_run_hooks(..., HOOK_DETAIL_JOB_CKPT, ...)` and relies ENTIRELY on `persist_job_hook_fun` being a registered subscriber to reach the WAL at all. With `NULL` passed, the real-call test silently appends NOTHING (no crash — `xce_run_hooks` on an unregistered detail is simply a no-op), so the restart-replay assertion fails with "ckptinfo NULL" even though the fix code itself was already fully correct — a genuine RED-signal red herring that looks exactly like a fix defect. Diagnosed by noticing the 6 hand-built-payload tests all passed (proving the apply-seam decode/adopt logic itself was right) while ONLY the real-production-call test failed, then re-reading `persist_start()`'s actual gating condition rather than assuming the copied fixture was correct because it was copied from a passing sibling. **General rule: before copying a sibling WAL-fact test file's "drive the real production function" fixture, check whether YOUR fact type's producer calls `persist_submit_job_*_fact()` directly (safe to pass NULL) or is purely hook-driven via `xce_run_hooks()` (MUST pass a real `cluster_t*` to `persist_init()`'s 3rd arg so `persist_start()` registers the hook) — grep the producer function's own body, don't assume parity from a structurally-similar sibling test.**
- **(2026-08-25, bug-1691) `controller_unit_test` (not just `xce_test`, extending the existing entry on this) also loses ALL buffered stdout (both PASS and FAIL lines) when run directly with output redirected to a file/pipe AND `ASAN_OPTIONS`'s default `detect_leaks=1` finds leaks** — LeakSanitizer's atexit-registered check appears to bypass libc's normal `exit()`-time stdio flush, so a process that ran 7/7 tests successfully to completion produced a log with ONE clean LeakSanitizer summary and ZERO printed PASS/FAIL text anywhere, first reading as "harness produced no output, something is badly wrong" before realizing it was a buffering artifact. `stdbuf -oL` does NOT fix it (worse: it breaks ASAN's LD_PRELOAD-ordering requirement, producing `ASan runtime does not come first in initial library list`) — the correct fix is `ASAN_OPTIONS=detect_leaks=0 ./controller_unit_test -t <group>` when reading stdout directly (matches meson's own test-harness invocation, which already sets this — the SAME binary run through `meson test` shows real PASS/FAIL text fine). The leaks themselves (spy-fixture jobs created via `makeJob()` into a `jobmgr` with no `.datakiller`, never individually torn down) are PRE-EXISTING and IDENTICAL across every sibling `test_wal_job_*_fact.c` suite (confirmed: `test_wal_job_rusage_fact` in isolation shows 78 leaked allocations too) — not a new leak introduced by any one fix, not worth chasing.

- **WAL field-parity fix family (bug-1678/1687/1689/1690/1691/1692, controller persist.c/job.c)**: when a hook-driven WAL fact (payload built inside `persist_job_hook_fun`, not a direct `persist_submit_job_*_fact()` call site) needs a new field added to survive replay: (1) check whether the field's own release function uses `FREEUP()`/`free()` (plain malloc/strdup/calloc — safe to free-and-replace, e.g. `JobAlloc`/`JobPIDInfo`/`JobPendReason`) vs `dereference()` (xstring/RTTI-backed, e.g. `ckptinfo`) vs "can be the SAME live object the apply pump re-applies moments later" (needs a MERGE, not free-and-replace — only `WAL_JOB_RUSAGE`'s `run_usage` needed this, because `update_host_runusage()` mutates it in place before the WAL fact is even appended); (2) check whether adoption must be gated on `JOB_TRANSITION_APPLIED` or unconditional — if the setter never changes `job.state` (true for CKPT/RUSAGE/PREASON, false for START/FINISH/DISPATCH), gating makes the fix dead code since every real replay hits `JOB_TRANSITION_NOOP_DUPLICATE`; (3) **check whether the field has a genuine CLEAR-to-NULL operation** (only `pend_reasons`/`do_clr_preason` in this family so far) — if so, "trailing bytes present or absent" (bug-1690/1691's convention) cannot distinguish a legitimate clear from an old-format payload; needs an explicit presence byte instead. Always re-verify (2) and (3) from the real setter code, never assume the newest precedent's shape applies unchanged.
- **capnp-c's generated decode path does not validate content, only bounds via `capn_getp`/`capn_resolve`**: passing well-formed-length-but-semantically-garbage bytes to a `decode_X()`/`read_X()` pair (e.g. `decode_JobPendReason`) does NOT return NULL or error — it silently produces a zero-valued struct. A "decode failure returns NULL" defensive branch (`decode_X() == NULL`) is only reachable via allocation failure or an upstream length/bounds guard (e.g. `payload_len` too short to even attempt the read) — never via corrupt-but-present bytes. Don't write a regression test assuming garbadge bytes will fail decode; it will silently pass through and the test either false-passes or fails for a confusing, misleading reason (`Expected 4 Was 0` — the "corrupted" value decoded as an all-zero struct, not a NULL).

- **(2026-08-25, bug-1695) `meson.build`'s `controller_unit_test_src` deliberately does NOT link `myapp.c` (or `main.c`) — only `startup_validation.c`, `ha.c`, `cluster.c`, `persist.c`, etc.** `myapp_run()` is documented as an "untestable integration entry point" (test_ha_startup.c's own Task 5.1 comment) requiring a live network socket, so it's excluded from the unit-test binary entirely. If you need to unit-test logic that currently lives inline inside `myapp_run()`, do NOT just add a non-static function to `myapp.c` and forward-declare it in a test file — it will compile but fail to LINK (`undefined reference`) because `myapp.c` itself never gets compiled into `controller_unit_test`. Instead, extract the logic into `startup_validation.c` (already linked into both `xce_controller`/`main.c` AND `controller_unit_test`, and explicitly documented as existing "for use in main.c and the controller unit test suite") and have `myapp.c` forward-declare + call it — exactly mirroring the pre-existing `validate_startup_config()` pattern (defined in `startup_validation.c`, forward-declared at the top of `myapp.c`, called from `myapp_run()`). Confirmed via a real RED/GREEN round-trip: extracting into `myapp.c` first produced a clean compile + link FAILURE across all of `controller_unit_test`'s ~90 test files; moving the same function body to `startup_validation.c` fixed it with zero other changes.
- **(2026-08-25, bug-1695) `ha_state_t` has TWO separate term fields that look similar but are written by different code paths and must not be conflated**: `ha->term` (top-level, `_Atomic uint64_t`, `ha.h:157`) is the adapter's own copy, written by `ha_restore_term()` at boot and by `raft_cb_persist()` on every `on_persist` callback; `ha->raft.term` (nested inside the embedded `raft_state_t`, `ha.h:198`/`raft.h:29`) is the raft LIBRARY's own internal term, seeded ONCE at `ha_init_raft()` time from `ha->term` (`rcfg.initial_term = atomic_load(&ha->term)`) and thereafter updated only by the library itself (`raft_start_election`/`raft_on_vote_request`/`raft_on_append_entries_request`, all via the SAME `on_persist` callback that also updates `ha->term`) — so they stay in sync after boot, but the ONLY correct place to read "the term as of right after this boot's restore" is `ha->raft.term` captured immediately after `ha_init_raft()` returns (not `ha->term`, and not before `ha_init_raft()` runs). myapp_run's bootstrap check reads `ha->raft.term` specifically (not `ha->term`) — a fix that snapshotted the wrong field would silently do nothing.
- **(2026-08-25, bug-1695) A `--chaotic` HA soak's per-round "wait_for_new_primary window" log line is a WEAK signal on its own — always cross-check the raw controller logs (`grep -n "CTRL_PRIMARY\|CTRL_STANDBY\|PreVote round" logs/ctrl-*.log`) before trusting it.** The chaotic driver's own per-round polling window (~90s, 30×3s probes) can time out ("NO new primary within window") even when a genuinely NEW node DID win a real election shortly after — the log line only reflects whether the driver's OWN bounded poll happened to catch it, not whether the underlying HA mechanism worked. In this session's soak, the fixed run showed "NO new primary within window" on 5/6 rounds (superficially similar to the unfixed baseline's 6/6) while the RAW logs proved something structurally different was happening: term genuinely progressing 1→2→3→4→7 and a different node (`ctrl-2`) achieving a real, log-verified `CTRL_PRIMARY` transition via a won PreVote — vs. the baseline where term was frozen at a single value for the entire run and NEITHER peer ever showed any PreVote/election activity at all, anywhere. **The A/B technique that resolved this ambiguity**: `git stash push -- <the exact files you changed>` (reverts just those files, keeps everything else including the same freshly-built binaries' surrounding tree state), rebuild, re-run the IDENTICAL soak invocation against a fresh scratch dir, compare raw logs side by side, then `git stash pop` to restore the fix. Far more conclusive than trusting either run's summary line alone.
- **(2026-08-25, bug-1695) A chaotic-soak `expect-job`/`verify_classify` "COMMITTED JOB LOSS" verdict is NOT self-verifying — cross-check the raw WAL with `xce_test`'s `waldump <data-dir>` REPL command (built into `xce_test`, not a separate binary) before treating it as a real controller data-loss bug.** `verify_classify` reads through the apply-pump/materialized-jobmgr layer (an RPC-level `expect-job` query), which per already-documented bug-1487 can legitimately lag behind the raw committed WAL for several seconds after a genuinely-new election (the new leader's OWN-term commit-barrier gate, `ha_arm_own_term_gate`, requires a fresh majority ack for its own-term no-op before folding in prior-term entries) — a RETRY-EXHAUSTED "not found" there does not by itself prove the entry is actually missing from the durable log. `walcompare`'s `committed_divergence` count (also built into `xce_test`) is the AUTHORITATIVE data-safety signal — it stayed 0 in this session's soak even while `verify_classify` counted 6 losses, and a direct `waldump` of all 3 nodes' `data/wal/seg-*.wal` files confirmed the ONE specific jobid in question genuinely does not appear on ANY node's WAL at all (not divergent between nodes — universally absent), tracing it to the cluster's very first cold-boot config-bootstrap window rather than anything this specific fix touched. **CRITICAL CAVEAT discovered later the same day (bug-1696): do not stop at "walcompare says 0 divergence" for a soak run driven by the bug-1695 fix specifically — that fix newly enables a restart-then-immediately-self-promote path that hits a SEPARATE, far more severe bug (WAL truncated to empty on promotion, see bug-1696 below). A truly clean soak needs BOTH committed_divergence=0 AND a raw `waldump` entry count that matches the real number of jobs submitted, not just "the specific jobid I went looking for is present somewhere."**
- **(2026-08-25, bug-1696, CRITICAL — Leader Completeness / bug-1060-guard defeated by boot ordering) `raft_cb_become_leader()` (ha.c:1615) computes the promotion WAL-truncation floor and evaluates the ENTIRE promotion-eligibility gate using `persist_durable_floor()`/`persist_recovery_truncated()`/`persist_apply_verdict()` BEFORE `cluster_init_phase2()` — the only call that invokes `persist_recover()` for this process's first-ever boot — has run.** `is_first_boot` handling (`cluster_init_phase2()` call) sits at ha.c:~1873-1899, AFTER the safe_committed_lsn computation and `persist_truncate_wal_above()` call (~1637-1662) and AFTER the full eligibility gate (~1699-1823) in the SAME function. Before recovery, a freshly-`persist_init()`'d `persist_t` reports `durable_lsn=0` — not the real on-disk tip. So on a node that RESTARTS with real committed data on disk and then wins its OWN promotion on that same boot (a path bug-1695's fix newly makes reachable — previously a restarted node's stale term made it always defer to a peer instead), `ha_promotion_truncate_floor(safe_lsn≈0, durable_tip≈0)` computes ≈0 and `persist_truncate_wal_above(p, 0)` physically destroys the real on-disk WAL before it's ever been read. bug-1445's comment (ha.c:1901) already flagged this EXACT "gate evaluated blind before recovery" hazard for one flag (`apply_resource_fenced`) with a narrow re-check-after-phase2 patch — but never for the far more consequential `durable_floor`/truncation computation, which runs even earlier and physically deletes data rather than just making a stale eligibility decision. **General lesson: any promotion/election-safety code that reads `persist_durable_floor()`/`persist_recovery_truncated()`/`persist_apply_verdict()` must be checked against whether `cluster_init_phase2()` (this process's `persist_recover()`) has actually run yet THIS boot — `is_first_boot` is the tell; a function that both checks `is_first_boot` and reads recovery-derived persist state earlier in its own body than the `is_first_boot` phase2 call is almost certainly reading pre-recovery garbage.** Root-caused via source trace (not guesswork) after direct `waldump` evidence (entire on-disk WAL = 1 NOOP entry despite 21+ committed jobs, no snapshot dir, zero checkpoint log lines) plus the smoking-gun log line `ha_compute_safe_committed_lsn: safe_committed_lsn=0 ... (from 2 granted voters)` immediately preceding `raft_cb_become_leader: → CTRL_PRIMARY` on ctrl-1's SECOND boot. Fix pending — see bug-1696 in buglog.json for the exact proposed reorder and required A/B soak proof.

- **(2026-08-26, bug-1688) When a zarray's `.count` field does something numerically impossible (doubles across a repeat push carrying the exact same keys, or drifts above `.caps`), do NOT jump straight to "something external corrupted the array between the two calls" or "the same key must be resolving to two different indices" — first check whether the CURRENT call's own writes are storing NULL.** `zarray_put_ex()` (zarray.c:267-303) only increments `.count` when the slot being overwritten currently reads NULL; it has NO NOTION of "is the new value itself NULL" — writing NULL into an already-occupied slot leaves `.count` unchanged while silently zeroing that slot's data, and a LATER write to that now-NULL slot (even with a perfectly stable index) then legitimately increments `.count`, looking exactly like "the array got corrupted between calls" if you only watch `.count`/the slot's occupancy and not the actual VALUE being written. Root-caused here only after adding a `zarray_put_ex()`-level watch-trace (a temp global pointer comparison, since the bug lived in library code called from many sites) that printed `data[i]` itself, immediately revealing `new_data=(nil)` on every write of the "corrupting" push — the two more exotic hypotheses (idx-resolution bug; core_reset_workers()/session-boundary code clearing the array) were both live-traced and definitively ruled out first, costing significant time that a "trace the actual value being written, not just count/occupancy" instinct earlier would have saved. Separately: `decode_Value_ptr()` (controller.capnp.c:2966) legitimately decodes to NULL whenever the wire's capnp pointer for that field is `CAPN_NULL` — this is not a decode bug, and any C struct field populated via `decode_X_ptr()` from a capnp-generated decoder should be treated as "can legitimately be NULL even when the enclosing struct/list entry itself is non-NULL," with call sites guarding accordingly (the same discipline this file's existing `nv == NULL`/`nv->name == NULL` guards already applied to the OUTER struct, just not to this one nested pointer field).

- **(2026-08-26) `controller_unit_test`'s meson `test()` registration sets `env: ['ASAN_OPTIONS=detect_leaks=0']` (src/controller/meson.build:239) — running the built binary DIRECTLY (bypassing `meson test`) without setting that same env var yourself makes ASAN's default leak detection fire at process exit, reporting ~7400 pre-existing allocations (~954KB) across dozens of unrelated test files as "leaked" and returning a nonzero exit code, even when the actual Unity test count (`N Tests M Failures`) is 100% clean.** This is NOT a regression signal — it's the project's own accepted, documented convention (also referenced in `xce-build.sh`'s own "skipped tests" hint: `ASAN_OPTIONS=detect_leaks=0 ./$BUILD_DIR/controller/controller_unit_test`). Always grep the target's own `meson.build` `test()` block for its `env:` before treating a direct-binary-run's exit code or ASAN output as ground truth; the Unity summary line (`"<N> Tests <M> Failures <K> Ignored"`) is the real pass/fail signal, not the process exit code alone.
- **(2026-08-26) A Unity `TEST_ASSERT_*` failure `longjmp`s out of the CURRENT test function immediately — skipping any teardown call written AFTER the assertion in that same function body.** If that teardown would have stopped a live background thread (e.g. `persist_stop()`/`persist_fini()` tearing down a real `fsync_worker` pthread), the thread leaks alive holding a pointer into that test's now-returned (and ASAN-poisoned) stack frame — the NEXT test in the same binary that happens to touch overlapping stack memory can then crash with a `stack-use-after-return` ASAN report that looks totally unrelated to the real, intentionally-failing assertion (confirmed while proving bug-1697's RED phase: a deliberately-reverted assertion failure in `test_wal_job_start_fact.c` cascaded into an unrelated ASAN abort deep inside `test_dispatch_commit.c`, later in the same run). This is a known, already-documented pathology in this codebase (see `test_upgrade_snapshot_apply.c`'s own header comment: "gather every assertion-relevant observable into a plain local variable FIRST, call teardown UNCONDITIONALLY, and only THEN assert") but most test files in this suite do NOT follow that discipline (assert-then-teardown is still the norm). When a RED-phase test run produces both the EXPECTED assertion failure(s) AND a seemingly-unrelated ASAN crash later in the same run, check whether the crash is this cascade (a live thread from an earlier failed-and-un-torn-down test) before treating it as a new bug — capture the actual observable into a variable, run teardown, THEN assert, if you need a RED-phase test to survive to completion without corrupting siblings.
- **(2026-08-26) When a live HA soak or other long-running deployment is using `src/build-asan` (or any shared default build dir) from a concurrent session, do NOT run `/xce-build` (or any `xce-build.sh`/`meson compile`) against that same directory — even for a "just checking" build — since a concurrent `ninja`/`meson setup` can race the other session's own build, and relinking a binary a live soak's processes still hold open risks corrupting or confusing that process's on-disk image.** Always pass an explicit, unique custom `build_dir` (`xce-build.sh <component> <profile> <custom-dir>`) for any build done "on the side" of another session's live deployment — check `ps aux | grep xce_controller` (or the relevant daemon name) FIRST to see which build dir(s) are actually live before choosing where to build. Also verify a dispatched build `Agent` subagent's build_dir argument actually took effect (compare the binary's own mtime against your most recent source edit's mtime) rather than trusting its self-report — one such subagent in this session silently ended up building into the script's own default dir (`xce.build`, from a malformed/dropped custom-dir argument) and initially reported stale/incorrect RED results from a build that predated the real edits; independently re-running the build directly (not via a sub-dispatch) and cross-checking mtimes caught it.
- **(2026-08-26) `deploy-ha.sh <binary> <top> <peers>`'s `peers` argument is a plain comma-separated `host:port` list — do NOT prefix each entry with its own `name=` (e.g. `ctrl-1=localhost:19991`); the script adds that prefix itself internally.** Passing an already-prefixed peers string produces a doubled `ctrl-1=ctrl-1=localhost:19991` in every generated start script, which silently breaks worker/scheduler cluster-join (`Failed to join cluster, retrying in 5 seconds` in the worker log) while the controller itself still starts and elects fine (its own `--peers` parsing tolerates/ignores the malformed self-reference for a single-node cluster) — the failure only surfaces once you try to verify a worker/scheduler joined, not at controller-startup health-check time. Confirmed the fix by re-deploying with a bare `host:port` peers string.
- **(2026-08-26) `xce_test`'s REPL command set is SPLIT by connection mode — `jobsub`/`joblist`/`jobinfo`/`wait-primary`/etc. live under `client` mode's command table, while `jstart`/`jfinish`/`jfail`/`resupdate` live under a SEPARATE `worker` mode's command table (test.c's `client_commands[]` vs `worker_commands[]`).** Typing `jstart` while still in `client` mode fails with a generic `"jstart: No such command for xce test."`, not an argument or connection error — easy to misdiagnose as a jobid-format problem. To send a synthetic worker-side fact (e.g. to test a WAL_JOB_START collision scenario against a job a REAL worker is also concurrently racing to finish), you need a SEPARATE `xce_test` invocation that switches into `worker` mode (its own `join`) first — you cannot mix `jobsub` and `jstart` in one client-mode session. The `$jobid`/`$jobcount` REPL auto-variables (populated by the most recent `jobsub`/`joblist`) are real and usable across subsequent lines in the SAME script (test.c:166-179), useful for exactly this kind of immediate-follow-up scripting once you're in the right mode.

- **(2026-08-26, bug-1700/bug-1701) `ha_state_t.primary_host`/`primary_port` (ha.h:161-162) are NEVER WRITTEN by any production code path — they are only ever set by test fixtures manually poking the struct.** `fill_redirect_reply()` (join.c:27) and anything that calls it (OPCode_join's handler at join.c:277, and bug-1700's new myreq_handler gate-rejection call) is therefore a NO-OP in a real deployment: the redirect it sends is always an empty host / zero port, which the xceadmin client correctly (per its own documented contract) treats as "no redirect." This was independently confirmed by grepping every `.c`/`.h` in `src/` for an assignment site (none outside tests) and is already explicitly named in `ctrlstatus.c`'s own doc comment ("the vestigial, never-written ha->primary_host") and in the already-closed bug-854's root-cause writeup — but nobody had connected the dots that this ALSO silently breaks `fill_redirect_reply()`'s OTHER caller, `join.c:277`, not just whatever new caller you're adding. **Before wiring any new caller of `fill_redirect_reply()`/relying on `ha->primary_host` for anything live, verify with a real 3-node deploy (not just a unit test that manually sets the field) that the value is actually populated on a standby** — a unit test proves the helper copies the field correctly, not that the field ever holds anything real in production.
- **(2026-08-26) When a task's own "critical correctness check" section asks you to verify a field's value on a standby "that has NOT yet learned of any live primary (e.g., very early boot, mid-election...)", read that as a hint to check the PERMANENT-vs-transient distinction, not just the transient one it names.** bug-1700's task prompt explicitly asked this question but framed it as a transient edge case; the actual answer was "permanently empty in every state, forever" — a much bigger finding than "empty during a brief startup window." Don't let the framing of a verification question limit how far you check.
- **(2026-08-26, bug-1701 FIX) `ha_state_t.primary_host`/`primary_port` (bug-1700/bug-1701's vestigial-field finding above) is now actually populated in production, closing the gap.** Fix is a pure LOCAL lookup in `handle_ctrlhello()` (config_repl.c:2170-2246, inside the existing `primary_ctrl_fd == -1` idempotency guard) — no new wire field needed, because every node already statically knows every candidate's `host:port` from boot (`c->myconf->peers_cfg`, parsed once from `--peers`). Match `req->from->name` (populated for EVERY incoming request at `request.c:379`, `req->from = ep->peer;`) against `peers_cfg.candidates[]` by name and copy the matched candidate's `hostname`/`port`. **General pattern worth remembering**: when a per-connection identity field (`req->from->name`) needs to resolve to a richer static record, check whether that record is ALREADY fully known locally (a config parsed at boot) before assuming you need a new heartbeat/wire-protocol field to carry it — `ha_init_raft`'s existing `voted_for_name`-to-index lookup (ha.c:7822-7829) was the exact precedent pattern to copy, found by grepping for other `peers_cfg.candidates[i].name` comparisons in the same file family. Also: `ha.h`'s field-level doc comments (e.g. `pthread_mutex_t lock; /* Protects primary_host/port + voted_for_term/name */`) are authoritative for which lock protects a field — trust that over inferring locking convention purely from a nearby unrelated field's (`primary_ctrl_fd`'s) actual call-site behavior, which can itself be under-protected without it being the intended convention for the field you're adding.
- **(2026-08-26) `xce-build.sh <component> <profile> <build_dir>`'s `build_dir` argument is resolved AFTER the script's own `cd "$SRC_DIR"` (`cd "$(dirname "$0")/.."`, i.e. `src/`) — passing an already-`src/`-relative path like `src/xce.build` from the repo root produces a bogus DOUBLED path (`src/src/xce.build`) that silently creates a whole second build tree instead of erroring.** Caught immediately (binary just wasn't at the expected path) but wasted a build cycle. When explicitly passing `build_dir` to `xce-build.sh`, give it relative to `src/` (e.g. bare `xce.build`), or omit it and let the profile default (`asan`→`xce.build`, `debug`→`build.dbg`, `release`→`build.release`) resolve it.
- **(2026-08-26) To produce a clean "before/after" pair of binaries for live E2E verification when OTHER unrelated files have uncommitted in-flight changes you must not touch or rebuild against**, use `git stash push -- <only the files for this fix>` (explicit pathspec, NOT a bare `git stash`) to isolate exactly the fix's own diff — rebuild for the "before" binary, `git stash pop` to restore the fix, rebuild again for the "after" binary. This leaves every other uncommitted file (the unrelated in-flight `ha.c`/`job.c`/`persist.c` work in this session) completely undisturbed throughout, unlike a bare `git stash` which would have stashed everything. For a PYTHON-only fix specifically, this same before/after swap works without any rebuild at all — just `cp` the two scheduler.py variants in and out of `src/xceadmin/src/xceadmin/` and re-run the SAME editable-install venv's `xceadmin` entry point; a `pip install -e` venv reads live from the source tree, no reinstall needed between swaps.
- **(2026-08-26, bug-1713) In this sandbox's local-loopback 3-node deploy, the FIRST-BOOT election-settling window varies wildly run to run (sometimes near-instant, sometimes ~10-12s via a term-1 election-timeout-then-PreVote-failover fallback path) — a fixed "wait for start-all.sh's health-check, then fire N requests" tight loop is NOT a reliable way to catch a transient `E_XCE_NOT_READY`/`E_XCE_NOMASTER` window: by the time start-all.sh's own 1s-granularity health-check first observes `CTRL_PRIMARY` in a log and returns, `ha_has_write_quorum()`'s `standby_endpoints` (populated synchronously inside `raft_cb_become_leader` from the raft library's own connected-voter set) is typically ALREADY satisfied, so the gate is already open. A REAL FAILOVER (kill -9 the current primary, then hammer the survivors with NO sleep, no health-check wait) reproduced the exact live symptom reliably instead — got a genuine bare `-10013` at ~t+8.6s post-kill on an unfixed client, and confirmed zero such failures surfaced across the equivalent window on the fixed client. The bug's own root-cause writeup already named this as an equally-valid trigger ("...or any time a real failover is in flight") — prefer the failover trigger over racing first-boot timing when a live repro is needed and cold-boot timing proves flaky.
- **(2026-08-26) A hand-rolled multi-connection mock TCP/TLS server in a test file needs a genuinely FINITE accept loop (`for handler in handlers:`), not `while self._handlers:` guarding a list that's never mutated — the latter is an always-true condition that leaves the server thread blocked in `accept()` indefinitely once the test's real client stops connecting, which does not fail the test but silently adds `thread.join(timeout=N)`'s FULL timeout to every such test's teardown (measured: 3 tests each added a flat 5s of pure teardown wait with zero effect on pass/fail).** Match the mock's handler-list length exactly to the number of connections the test actually expects (assert on `connection_count` for the "stops on its own budget" property instead of over-provisioning handlers "just in case") and additionally set a short `socket.settimeout()` on the listener as a safety net so a hypothetical future regression that causes MORE connections than expected still fails within one timeout window instead of silently reintroducing the same multi-second stall.

## Do-Not-Repeat
- **A TDD-implementer SUBAGENT (spawned by a kiro-impl coordinator) has NO external wakeup mechanism — arming a `Monitor` tool call or a `Bash run_in_background` call and then ending the turn to "wait for a notification" just stops the subagent forever; the coordinator has to manually resume it.** (2026-08-26, xce-test-rc-file task 1.2, recurred across multiple tasks in this same spec before being called out explicitly.) Only the top-level session genuinely receives async task-notification events between turns; a subagent's turn ending IS the end of its execution for that dispatch. When a validation command (e.g. `controller_unit_test`) risks exceeding the default 120-300s Bash timeout, pass an explicit large `timeout` (e.g. 300000-600000ms) to a PLAIN FOREGROUND `Bash` call and let the tool call itself block until real output returns — never `run_in_background`, never `Monitor`, never end a turn on an unfinished command in this role. If a command is auto-moved to background because the timeout already elapsed, immediately re-run it in the foreground with a longer timeout rather than trying to "wait" for the backgrounded one.
- **When translating a `fp_ok`-tracked, per-step-gated failure-detection pattern (like `xce-build.sh`'s dirty-fingerprint computation) into a Make `$(shell sh -c '...')` one-liner, do NOT collapse two commands into `{ cmd1; cmd2; } >"$$fp_tmp"` and gate the whole group with one `if`** — a POSIX sequential group's exit status reflects only its LAST command, so `cmd1`'s failure is silently discarded whenever `cmd2` still succeeds (2026-08-21, bug-1612: `git diff HEAD` failing was swallowed because the trailing `git ls-files | sort | xargs cat` pipe still exited 0, producing a fabricated fingerprint instead of degrading to empty). Give each step its own explicit `if ! step; then fp_ok=0; fi` (or `&&`-chain), exactly mirroring the reference script's discipline, even inside a cramped one-liner. Caught by adversarial code review with a targeted failure-injection test (wrapper `git` on `PATH` that fails only the targeted subcommand) — that test pattern is the reusable way to verify this class of fix.
- Don't run `clang-format -i` on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via `git checkout HEAD -- <file>` on a file with other uncommitted work destroys that work irrecoverably.
- **(2026-08-22, scheduler-sim-decision-parity task 2.3) Actually ran `clang-format -i` on two files mid-task despite this rule already being logged above — caught it via `git diff` before finishing, not before running it.** If this happens again: do NOT `git checkout` the file (destroys real work). Instead: `git diff -- <file>` and read every hunk; any hunk whose ONLY change is whitespace/line-wrapping on code you did NOT intentionally write/edit this turn is collateral reformatting — manually `Edit` just that hunk back to its pre-format text (find each such hunk, revert by hand). Verify success with `git diff -- <file> | grep -c '^-[^-]'` on regions you didn't touch dropping to 0, then rebuild+retest to confirm nothing broke. Cheap enough to be a non-event once caught, but the real fix is: check this Do-Not-Repeat list (or just never run `clang-format -i` without `--dry-run`/`-output-replacements-xml` first to preview scope) BEFORE running it, not after.
- Before citing a bug as "still open" or a spec as "blocked," cross-check `.wolf/buglog.json`'s actual fix status and the spec's own tasks.md/commit history — blocked_on notes and prior narrative claims (including this file's own past entries) can be stale. **Self-caught recurrence (2026-08-19)**: while reconciling OTHER stale doc references, wrote a new tasks.md status line asserting "tasks 2.1-2.6 still carry no `_Verified:` annotations" by trusting a 2026-08-07 validation finding's own wording verbatim, without re-checking the file — they'd actually been backfilled the very next day (commit `d7b49690`, 2026-08-08). A dispatched verification agent caught it via `git log`/direct read. Lesson holds even for a claim you are ABOUT to write, not just one you're citing from memory: grep/read the actual current state of the specific lines before asserting their status, every time, no exceptions for "this one's probably still true."
- Before trusting an "all tests pass, GO" claim on a stale/uncommitted tree, re-run the suite yourself — a buglog "FIXED" note can describe the *intended* fix while the code on disk is mid-revert from RED-phase construction.
- A generic retry/failover classifier that judges retriable-vs-terminal by a raw return value's magnitude (comparing against known errno values) is unsafe unless every wrapped call site shares the same success convention — a call that legitimately returns a variable count on success can numerically collide with an error code.
- When two independent "is it ready" signals exist server-side, a wait/convergence helper gating on only one does not imply the other is satisfied — check what the actual raced precondition depends on.
- Nix fixed-output-derivation hash mismatches from an upstream input pinning a *mutable* branch: re-locking flake.lock is a dead end once already at the latest commit; fix via a local `overrideAttrs` pinning `src` to an explicit immutable commit in the consuming repo.
- `pkill`/`pgrep` by bare process name (unscoped) can kill or match a *different* concurrent invocation's processes, including a sibling session's build/soak — always scope process matching to something unique to the current run (e.g. a workdir path).
- **Stacked/masking bugs in never-before-reached code paths: when a bug prevents a code path from EVER executing, fixing it can immediately expose 1+ MORE bugs that were always there but never triggered.** bug-1569 thread B was "no job ever gets scheduled" -- fixing cause #1 (scheduler.c's sched_init_queues() never calling picker_iface->on_queue_update() for builtin_fcfs, so its candidate registry stayed empty forever) let a job reach the allocator for the first time ever, which immediately SIGSEGV'd on cause #2 (core_events.c's core_build_job() never constructing job->schedobj, a documented-but-wrong assumption from the scheduler-core rewrite). Fixing #2 let a job reach genuine dispatch+resource-charge accounting for the first time ever, which immediately SIGSEGV'd on cause #3 (resource.c's put_resvals() had an off-by-one, `idxs[i] <= 0` treating resource idx 0 -- "slots" -- as an invalid sentinel). Lesson: after fixing a "nothing ever happens" bug, don't declare victory on the first crash/error that surfaces -- it may be the SECOND (or third) bug the first one was masking, not a regression from your own fix. Verify via gdb `bt full` on the actual live crash, not guesswork from reading code alone.
- **`xce_scheduler -i`/`--sim` (the interactive REPL) starts PAUSED at the initial snapshot** and never advances the simulated clock past a recorded `ReplayEvent`/`ReplaySession` until you send an explicit `step [N]` or `continue`. Piping `"info worker <name>\nquit"` with no `step`/`continue` first only ever shows the pre-replay baseline state — this cost a large chunk of bug-1569 investigation, wrongly appearing to show a second "periodic resources still 0.00 after the host-mismatch fix" bug when the fix was already fully correct and sufficient; `step N` (N large enough to pass the relevant event, confirm via `xceadmin scheduler dump <file>` first if unsure of event ordering) revealed the true, correct values. Before concluding a replay-derived state is wrong, always `step`/`continue` first.
- **A dead `#if 0 /*FIXME*/ ... #endif` block is a signal the disabled code is unsafe as written, not merely "not yet wired in" — don't just uncomment it.** bug-1578: `sched_worker_change()`'s disabled block called `free_name_val_list(&w->host.resvals_to_sched, &w->host.nresvals_to_sched)` right before `xlist_fini(&resvalist)` — but `resvalist.data` and `w->host.resvals_to_sched` are the SAME pointer (the earlier line aliased one to the other's backing array), and `free_name_val_list()` already frees that array itself; uncommenting as-is would double-free. The safe fix null-and-reset the aliased fields directly (`w->host.resvals_to_sched = NULL; w->host.nresvals_to_sched = 0;`) and let `xlist_fini()` be the sole owner of the free — the pattern already proven correct in the sibling `apiserver_worker_change()` (apiserver.c). General lesson: when re-enabling dead code, first work out WHY it was disabled (git blame + read what it actually does to shared/aliased memory), don't assume `#if 0` means "harmless, just needs a flip." Other call sites still doing `free_name_val_list(&(w->resvals_to_sched), ...)` on this same field (dataex.c:435, opget.c:128, reply.c:66, apiserver.c:51) were NOT audited for the identical aliasing hazard as part of this fix — flagged, not yet checked.
- **`xce_test`'s `!<shell-command>` escape (`preprocess_and_run()`, single call site in `main()`'s top-level REPL loop, `test.c` ~line 950-998, backed by a plain blocking `system()`) works IDENTICALLY in every REPL context, including `worker` — no `leave_ctx()`/disconnect happens, unlike moving between contexts via `..`/`cmd_up`.** This is the general pattern for "run an out-of-band check while a stateful REPL session must stay connected": bake a `!`-prefixed line into the script fed to the session (before its `quit`) that shells out to ANOTHER invocation of your own driver script in a hidden internal mode (e.g. `! bash "$0" --internal-<mode> <args>`), intercepted via a dispatch check placed BEFORE the normal arg-parsing entry point at the very bottom of the file. Used to fix bug-1579 (resource-value-verification): the controller-side resval check now runs nested inside the injecting worker session's own `!` escape instead of as a separate xce_test process launched after `quit` already disconnected the worker (which reset host-attrs resources to baseline first, an always-false MISMATCH). Gotchas: (1) the nested process must generally return exit 0 even on partial/expected failure conditions — a nonzero exit becomes the outer session's `g_last_status`/`$?`, and in non-interactive/batch mode `preprocess_and_run()` ABORTS the REPL loop before any later command (including `quit`) runs; surface real failures via a sentinel/marker file the OUTER script checks AFTER the whole session completes, not via the nested process's own exit code. (2) `system()`'s child inherits the invoking process's environment but NOT its parent bash's local (non-exported) shell variables — bake every value the nested invocation needs (paths, host lists, names) as literal, `printf '%q'`-quoted arguments directly into the `!` line's command string, don't rely on env-var passthrough through the `nix develop`/ASAN-wrapped intermediate hops. (3) since accept/reject-style bookkeeping computed by the REPL session itself is only visible as TEXT PRINTED to that session's own stdout — not accessible to a freshly-spawned nested process — have the nested step collect RAW, unfiltered data for every candidate (not just ones later known to be "accepted") and defer accept/reject filtering to the outer script, after it has parsed the whole session's captured output post-`quit`. (4) live-verified no orphan/zombie processes result even when a SIGTERM interrupts the outer driver script WHILE the nested `!`-escape's own sub-session is actively running — bash defers a pending trap until the current foreground command (the whole piped xce_test invocation, nested check included) returns, so the child process tree either completes and exits cleanly on its own or the trap fires right after; confirmed via a full `/proc/[0-9]*/cmdline` scan, not just `pgrep`.
- **Piping scripted commands into `xce_test` (the controller REPL/testsuite binary) with default `ASAN_OPTIONS` loses ALL stdout output when redirected to a file/pipe.** `xce_test` is an ASAN build; `detect_leaks=1` is the default, and this binary has small pre-existing intentional leaks (e.g. `set_resval`'s `calloc`, `cmd_resupdate`'s `strdup` for `resname` — never freed by design, REPL-lifetime scratch data). LeakSanitizer's exit-time report appears to bypass/precede the normal stdio flush when stdout is non-interactive (fully block-buffered), so a redirected-output smoke test showed ONLY the ASAN leak report and none of the actual REPL command output (join/resupdate/resval results), even though the commands genuinely executed (confirmed via the leak stack trace itself). Fix: always run `xce_test` with `ASAN_OPTIONS=detect_leaks=0` (matching the meson test harness's own invocation, which already sets this) whenever piping/redirecting its stdout for a live smoke test — e.g. `nix develop ./src --command env ASAN_OPTIONS=detect_leaks=0 xce_test < cmds.txt`.
- **A fork/subagent dispatched in a prior session that claimed "build/tests in progress, will report back" is NOT reachable in a new session — `ListAgents` will not show it, and its self-report (if logged to memory before the session ended) describes an *intent*, not a completed result.** 2026-08-19: a fork agent from the previous session was recorded as "actively writing tests... then building via xce-build skill and reporting final metrics," but the session ended before it reported. On resuming ("continue"), the scheduler test binaries in `src/xce.build` were still dated Aug 18 14:27 while the source edits were Aug 19 09:18-09:27 — proof the claimed build never actually happened in this environment. Always verify via binary/source `stat` mtime comparison (or just re-run the build) before trusting a cross-session "in progress"/"dispatched" claim, same discipline as the existing "stale/uncommitted tree GO claim" entry above but for cross-session fork continuity specifically, not just tree staleness.
- **Go's `-ldflags -X importpath.name=value` is silently a no-op (not a build error, not even a printed warning) when the target `main` package has no such package-level string var declared** — confirmed live (build-time-versioning task 4.1, 2026-08-20): built `xcecli/cmd/main.go` (which does NOT yet declare `GitCommit`/`BuildTime`, those land in task 4.4/4.5) with `-ldflags "-X main.GitCommit=abc123 -X main.BuildTime=..."` via both a direct `go_build` invocation and the full `xce-build.sh xcecli` meson path — exit 0 both times, working ELF binary produced, `grep -i "warning\|no such"` on the full captured output found nothing beyond an unrelated nix git-dirty-tree line. This means task 4.1 (the `-ldflags` injection in `go_build`) has NO real ordering dependency on tasks 4.4/4.5 (the consuming vars) — safe to land and validate standalone, contrary to what a cautious reading of Go linker docs might suggest.
- **A REPL/replay test fixture that reuses the SAME jobid for both a "real" event (job push) and a "cheap padding" event (job state change, added purely to exercise a second event type) accidentally mutates the real job's state, not just decorates the log.** bug-1595 test regression (task 12.5, 2026-08-19): `test_step_echoes_events_before_session_decisions` pushed job 6161 then sent a JOBSTATECHANGE event for the SAME jobid 6161 (PEND->RUN) purely to get a second event type echoed — this genuinely transitioned the real job to RUNNING before the session ran, so the picker's `next_job()` never offered it and the session's own PEND decision (which the test also asserted on) was never produced. The sibling test `test_event_breakpoint_pauses_repl_at_matching_event` already documents the correct pattern in its own comment (`append_jobstatechange_event() establishes: core_events.c WARN+skips an unknown jobid rather than fail, so ... below needs no real job to exist`) — it deliberately used unrelated jobids (111/222) precisely to keep the padding event inert. When adding a "just need any event of type X to be logged" filler event to a fixture that also has a real job in flight, always target an unrelated/nonexistent jobid unless the state transition itself is the thing under test.
- **`[ -n "${arr[key]+x}" ]` (associative-array key-EXISTENCE) is NOT a safe proxy for "the value is meaningful/reachable" when the producer unconditionally sets that key even on failure.** bug-1582 (resource-value-verification task 7.1, round-1 REJECTED, round-2 fixed): `main()`'s record-building loop derived `sched_ok`/`api_ok` from whether `SCHEDULER_OBSERVED[$vname]`/`APISERVER_OBSERVED[$vname]` existed — but `query_scheduler_resvals()`/`query_apiserver_resvals()` (already-approved tasks 6.3/6.4) always set that key, setting it to `""` on a genuine UNREACHABLE/timeout rather than leaving it unset. Key-existence was therefore always true, silently masking every real UNREACHABLE as a MISMATCH (comparing the injected value against an empty string). Fix: derive the "is this layer reachable" signal from the layer's own pre-classified status string (`SCHEDULER_CHECK_RESULTS[$vname]`/`APISERVER_CHECK_RESULTS[$vname]`, which DOES distinguish `UNREACHABLE:*` from `MATCH`/`MISMATCH:*`) instead of from a sibling data array's key presence. General lesson: before using `${arr[k]+x}`/`-v arr[k]` as a "was this checked / did this succeed" signal, verify the producer actually leaves the key UNSET on failure — if it sets it to an empty/placeholder value instead (a common pattern for "always populate every requested key so downstream code doesn't need existence-checks"), key-existence collapses to "was this key in the requested set" a wholly different, weaker signal than "did this succeed". Also: a synthetic test that only calls the pure aggregation function directly with hand-built records (task 8.2's original approach) cannot catch a bug living in the data-marshaling code that BUILDS those records — this class of bug needs the marshaling code itself extracted into a directly-callable function (`build_verdict_records_from_tsv()`) and exercised by its own test.

- **(2026-08-26, xce-test-rc-file task 1.1, bug-924) When `controller_unit_test` invoked DIRECTLY (not via `meson test`) dies instantly (well under 1s) with a plain exit code 1 and NO crash/ASAN signature — the raw log just stops mid-file, often right after some earlier, unrelated `test_*.c` group's last line — check `df -h /tmp` before suspecting a real regression.** This sandbox's `/tmp` accumulates `mkdtemp()`-created `xce_*` fixture directories across EVERY session indefinitely (no test file in `testsuite/` cleans up its own `/tmp/xce_*` dirs); once the filesystem backing `/tmp` hits 100% used, `mkdtemp()` fails with ENOSPC inside whichever test file's fixture setup runs next, and several such files (e.g. `test_repl_apply.c`'s `make_test_persist_for_apply()`/`make_test_persist_in()`) hard-`exit(1)` on that failure rather than reporting it through Unity — silently killing the ENTIRE combined-binary test process before any later `test_*` group (including one you just added) gets a chance to run, with no signal distinguishing it from your own new code being broken. Confirmed live at 14731 leftover dirs / 653.6G / 100% used, 0 available — `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` reclaimed 630G (100%→29%) and the exact same binary then ran to completion (1208 tests) cleanly. Also check for and kill any orphaned `controller_unit_test`/`xce_controller` process left over from an earlier invocation in the SAME session (`ps aux | grep controller_unit_test`) before re-running — one was found stuck in `futex_do_wait`, presumably retry-looping a WAL fsync against the full disk, and could confuse a fresh run's process/port state. See buglog.json bug-924 (now updated with this occurrence) for the original, smaller-scale sighting.
- **This coding-agent harness's interactive shell aliases plain `grep` to a PCRE-capable (ugrep-backed) wrapper via a `.bashrc`-sourced shell FUNCTION — but that function is interactive-shell-only and is NOT inherited by a script invoked as `bash script.sh`, which falls back to whatever the real `grep` binary on PATH is (in this environment: BusyBox, no `-P` support at all).** (2026-08-20, build-time-versioning task 6.2): `echo x | grep -oP '(?<=x)y'` succeeded fine when typed directly as a Bash-tool command, but the identical `grep -oP` call inside a standalone `.sh` file, run via `bash thefile.sh`, failed with `grep: invalid option -- 'P'` (BusyBox usage dump). Confirmed via `type grep` in an interactive call (shows a `_cc_a`-prefixed shell function shelling out to the Claude Code binary itself acting as ugrep) vs. `which -a grep`/`/sbin/grep --version` showing BusyBox as the real underlying binary. **Any shell script file meant to be run standalone (not just typed inline as a Bash-tool command) must assume POSIX-only grep/sed (BRE, `-E`, `-F`, `-o`) — no `-P`/PCRE, no lookbehind/`\K`.** Use `sed -n 's/.../\1/p'` (POSIX BRE, works under BusyBox) instead of `grep -oP '...\K...'` for field extraction inside any committed script.
- **A direct `go build` (even a real, working one) without `-mod=readonly` can silently rewrite a COMMITTED `go.sum`, adding unrelated indirect-dependency checksum lines as a side effect of resolving the full module graph — even though `go env GOFLAGS` was empty and go.mod itself was never touched.** (2026-08-20, build-time-versioning task 6.2): building `apiserver/cmd/main.go` directly (bypassing the network-hung `go_build` wrapper, see below) added ~65 new `/go.mod`-hash-only lines to `src/frontend/apiserver/go.sum` on the first attempt — caught only via a post-hoc `git status --porcelain` sweep, not by the build itself (exit 0, no warning). Go 1.16+'s `-mod=readonly` default blocks go.mod EDITS but does NOT by itself block go.sum EXTENSION for modules already implied by go.mod's requirements. Fix: pass `-mod=readonly` EXPLICITLY on every ad-hoc `go build` invocation that isn't supposed to touch repo state — confirmed live that the same build still succeeds under explicit `-mod=readonly` with zero go.sum diff, so nothing was genuinely missing; the extra entries were pure side effect. **Any script/agent that runs `go build` directly against a checked-out repo (not via the project's own wrapped build path) must always pass `-mod=readonly` and verify `git status --porcelain` on go.sum/go.mod afterward, every time, no exceptions.**
- **`src/tools/go_build`'s hardcoded `GOPROXY=https://goproxy.cn` + a fresh-per-invocation `GOCACHE`/`GOPATH` under meson's own `$TMPDIR` sandbox can hang indefinitely in this environment — not fail fast, HANG: an established (not refused/reset) TCP connection to goproxy.cn:443 sits in `futex_do_wait` with near-zero CPU time for 8-10+ minutes across repeated attempts.** (2026-08-20, build-time-versioning task 6.2): `xce-build.sh apiserver <profile> <dir>` (which builds apiserver AND xcecli together via this wrapper) reliably hung on xcecli's fresh module downloads; confirmed via `ss -tnp` showing `ESTAB` to `8.45.176.179:443` (=goproxy.cn, independently reachable fine via plain `curl`/`/dev/tcp` from the SAME shell) with the `go` process's CPU time frozen — a genuinely stuck data path, not slowness, since it never progressed no matter how long left running. Root cause isolated to `go_build`'s own env overrides (`export GOCACHE=$TMPDIR/go-cache; export GOPATH=$TMPDIR/go; export GOPROXY=https://goproxy.cn`), which bypass the ALREADY-POPULATED default `$GOPATH/pkg/mod` cache (`go list -m all` from an ordinary `nix develop` shell, no wrapper, resolved apiserver's full module graph in under a second, zero downloads). **Workaround (do NOT modify go_build/Makefile — this is this specific sandbox's networking quirk, not a real bug in those files): invoke `go build` directly with the default module cache (`GOPROXY=off` is fine since everything needed is already cached) plus the same CGO_CFLAGS/CGO_LDFLAGS pkg-config derivation `test-backend`/apiserver's own meson.build already uses (see bug-1597's documented fix) whenever a real apiserver/xcecli build is needed standalone in this sandbox and `xce-build.sh apiserver`/`make build-server` hang or fail.** Do not blindly increase the timeout and retry — a truly stuck TCP data path does not resolve itself no matter how long you wait; diagnose via `ss -tnp`/`/proc/<pid>/wchan` first.
- **Refinement to the above (dirty-content-fingerprint task 4.4, 2026-08-21)**: when the Go-side change under test lives ONLY in `apiserver/cmd/main.go` (or another `cmd`-level file) and doesn't need the real cgo backend, skip bug-1597's full CGO_CFLAGS/CGO_LDFLAGS pkg-config reconstruction entirely — `apiserver/backend/` already ships a `//go:build !cgo` stub (`stub.go`) alongside every `//go:build cgo` file, so `CGO_ENABLED=0 GOPATH=/home/claude/go GOPROXY=off go build ./cmd/main.go` (matching the Makefile's own `test` target convention, `GOFLAGS := CGO_ENABLED=0`) compiles cleanly with zero C dependencies and produces a real, runnable binary — confirmed live across `go vet`, `go test`, and 5 distinct `-ldflags`-injected `-version` scenarios plus a real (if CGO-stub) daemon-start log capture. Only reach for the full CGO reconstruction when the task actually needs the real backend/libxce integration exercised (e.g. bug-1597's own `build-server` scenario). `-mod=readonly` was NOT needed this time (checked `git status --porcelain` on go.sum/go.mod after every build — zero diff both times), likely because the module graph was already fully resolved in `/home/claude/go/pkg/mod` from prior sessions; still worth checking every time per the entry above, since whether it triggers depends on cache state, not code.

- **This sandbox's `nix build` (not `nix develop`) dispatches to a REMOTE builder over SSH (`ssh://rshen@172.31.109.18`), which is NOT reachable directly via SSH from inside this sandbox (connection times out — same network restriction family as the goproxy.cn DNS failures).** (2026-08-21, dirty-content-fingerprint task 6.3): confirmed via `nix build`'s own stderr ("The failed build directory was kept on the remote builder due to `--keep-failed`... re-run with `--builders ''` to disable remote building"). Practical consequence: `--keep-failed`'s leftover build directory (containing real, successfully-linked binaries from a partially-failed multi-target derivation) is NOT locally inspectable/runnable without either `--builders ''` (forces a full local rebuild from scratch — expensive, full ASAN tree) or SSH access to that remote host (unavailable here). When a task needs evidence from a `nix build` that can't fully succeed (e.g. the known bcctool/goproxy.cn limitation), prefer `nix eval` (fast, eval-time, no builder involved) cross-checked against the real build LOG's `mesonConfigurePhase flags:` line (via `-L`) rather than trying to extract-and-run an actual compiled artifact — this is also literally what task 2.2 already did.
- **This sandbox's `mktemp` is BusyBox's implementation, not GNU coreutils — it requires the TEMPLATE to literally END in `XXXXXX` with nothing after it (no `--suffix` flag, no trailing literal suffix like `.XXXXXX.log`).** (2026-08-21, dirty-content-fingerprint task 6.3): `mktemp "path/name.XXXXXX.log"` fails with a bare `mktemp: Invalid argument` (no other explanation) under BusyBox; `mktemp --help` confirms ("TEMPLATE must end with XXXXXX"). Always end mktemp templates in this repo's scripts with `XXXXXX` and nothing else — append any desired suffix to the returned path afterward if truly needed, don't rely on GNU's suffix support.
- **A self-check that greps a script's OWN SOURCE for a literal flag string (e.g. confirming `--impure` is never used) will false-positive on the script's own comments/log messages if those comments discuss the flag by name (which good documentation often does).** (2026-08-21, dirty-content-fingerprint task 6.3): a blind `grep -q -- '--impure' "$0"` matched this script's own header prose explaining why `--impure` is deliberately avoided. Narrow such self-checks to lines that begin an actual invocation of the relevant command (e.g. `grep -A2 -E '^[[:space:]]*nix (eval|build|develop|log)\b' "$0" | grep -q -- '--impure'`), not a whole-file grep — this generalizes to any "assert we never do X" self-check in a script whose own documentation legitimately mentions X.
- **Reinforcement of the bug-1615 same-statement `local a=... b="$a..."` gotcha**: recurred independently in a brand-new file (`extract_flag_from_json()`, dirty-content-fingerprint task 6.3, logged as bug-1616) less than a day after being first documented — this pattern is easy to reach for out of habit when writing a "compute one derived value from a parameter" helper. When writing ANY new bash function with `local x="$1" y="...$x..."`, default to splitting the dependent assignment onto its own `local` statement, don't rely on remembering the gotcha in the moment.
- **scheduler-sim-decision-parity task 1.1 (2026-08-22)**: relocating an `xce_scheduler_t*`-signature function to `xce_sched_core_t*` (the `sched_resolve_job_picker()` precedent, now repeated for `sched_account_jobs()`/`sched_charge_account()`/`sched_uncharge_account()`/`sched_update_account()`/`sched_update_res_accounts()`/`sched_commit_job_alloc()`) has a hidden fan-out cost: EVERY testsuite file that white-box `#include`s the driver/scheduler source and provides its OWN link-satisfying spy/stub for that function (grep the whole `testsuite/` dir for the function name, not just the file(s) named in the task boundary) must have its stub's signature updated too, or the build fails with a type mismatch/duplicate-declaration error — 6 separate test files needed this for `sched_account_jobs()` alone (test_driver_decision_cb.c, test_resval_ownership.c, test_admin_request.c, test_sim_equivalence.c, test_record_lifecycle.c, test_driver_translate.c), none of them named in the task's own boundary list but all mechanically required by it. This project's codebase convention on the relocated parameter's name is to rename it from `sched` to `core` (confirmed via `core.c`'s own `sched_get_job`/`sched_add_worker`/etc., not just keeping the old name with a retyped pointer) — match that, don't just retype in place.
- **Testing "does function X behave identically given a core from container A vs container B" (Req 1.4-style cross-container proof) does NOT require white-box `#include`-ing the heavyweight driver file that defines the thin iterate-and-call wrapper** (here, `scheduler.c`'s ~20-line `sched_account_jobs()`) if the real accounting logic lives one level down in a lighter, already-provenincludable file (`resource.c`, per `test.c`'s own existing `#include "../src/resource.c"`) — no test in this project's `testsuite/` white-box-includes `scheduler.c` at all (confirmed via grep before writing), because its link-time dependency surface (controller-join, capnp publish, HA candidates) is much larger than `resource.c`'s. Test the real lower-level functions directly instead (`sched_charge_account()`/`sched_uncharge_account()`/`sched_commit_job_alloc()` here) and document in the new test's header comment exactly why the thin wrapper itself isn't independently exercised (already covered elsewhere, e.g. by an existing ordering-proof spy test) — this is honest, still exercises 100% real relocated code, and avoids a huge stub-writing burden.
- **Editing `.wolf/buglog.json` (or any large existing JSON file with unicode em-dashes/arrows in prose fields) via Python's `json.load()`/`json.dump()` reformats the ENTIRE file, not just the new entry — `json.dump`'s default `ensure_ascii=True` rewrites every existing `—`/`→` character anywhere in the file into a `\uXXXX` escape, producing a diff hundreds of lines larger than the actual change.** (2026-09-01, worker-rejoin-resync task 4.1 self-caught, then recurred anyway in a dispatched implementer's own task-5 edit before being caught and fixed forward as its own commit): appending one ~30-line bug entry this way produced a 400+-line diff touching ~400 unrelated pre-existing entries. Always pass `ensure_ascii=False` to `json.dump()` when writing back a file that may contain non-ASCII prose — or better, avoid `json.load`/`json.dump` round-tripping entirely for a pure append (insert the new entry as raw text right before the file's closing `]\n}`) so untouched entries are never re-serialized at all. If this recurs a third time, brief every implementer prompt that touches `buglog.json` with this exact gotcha up front rather than relying on self-catching it after the fact.
- **(2026-08-26) Before treating a fresh HA deploy's anomalous behavior (term jumping impossibly fast, a standby permanently failing to load config, a node never converging with peers) as a real product bug, check for and kill ANY leftover controller/scheduler/worker processes from EARLIER in the SAME session — a same-session soak/investigation left running and forgotten (`ps aux | grep xce_controller`) can bind the exact same default ports (`9991`/`9992`/`9993`, this project's `/xce-test-ha` skill default) a brand-new deploy also uses.** Concretely: an abandoned `xce-ha-chaotic2` 3-node cluster from an earlier bug-1696 investigation, still running hours later, was still bound to 9991-9993 when a fresh `run-ha-soak.sh --sequential` deploy was launched at the same default ports; the new cluster's vote traffic cross-talked with the ancient zombie (which had organically reached a high term over its own long runtime), producing an apparently-impossible term jump (term=1→23 within 50ms of a genuinely fresh first boot) and a standby that could never load cluster_config for the whole test — both fully explained once the zombies were found (`ss -tlnp | grep -E "999[123]"` showing FOUR processes across two unrelated deploys all bound to the same three ports) and killed; a clean re-run on the exact same code, same binaries, same script passed cleanly (`PASS: 2 rounds, 12 jobs, zero job loss; all 3 nodes led`). **Always run `ps aux | grep -i "xce_controller\|xce_scheduler\|xcecd"` (and `ss -tlnp` for the ports you're about to use) before launching a "fresh" HA deploy in this sandbox, especially after any earlier soak/investigation in the same session — a `/xce-test-ha` skill soak that "completed" or whose agent died without explicit cleanup does not guarantee its processes actually exited.** This directly extends the earlier bug-1487-era lesson ("a chaotic-soak verdict is not self-verifying, cross-check the raw WAL") one level further back: an anomalous LIVE symptom from a fresh-looking deploy is not self-verifying either until same-session process/port contamination is ruled out first.
- **`xce_sched_core_init()` is safe to call twice in the same test process/function to build two independent, simultaneously-alive `xce_sched_core_t` instances for a side-by-side comparison** — established precedent `test_snapshot_save_load.c` (`core_a`/`core_b`) already relies on this; it does re-init a couple of process-global singletons internally (`xstring_table_init()`) but that's a harmless re-init, not a crash/corruption risk, confirmed by that pre-existing test passing. White-box-including `../src/core.c` + `../src/resource.c` (NOT `core_events.c`/`core_session.c`/`decision.c`/`plugin.c`, unneeded if the test only calls `xce_sched_core_init()`/`sched_add_worker()`/`sched_charge_account()`-family functions directly, bypassing the event-push/session-begin/decision-drain APIs entirely) needs exactly 7 link-satisfying no-op stubs: the `kill_sched_job/queue/worker/user/project` family + `kill_sched_resval` (core.c's kvmgr/zarray datakillers) + `plugin_unregister` (core.c's plugins-kvmgr datakiller) + `plugin_prepare_allocation`/`sched_ctx_get_uncomit_host` (resource.c's `sched_try_alloc_on_host_()`'s own callees, needed only because that whole file is pulled in, even though this test path never calls it) — the last two are the exact same stub pair `test.c`'s own resource.c-only white-box test already establishes needing.
- **Never use `date -d` with a RELATIVE expression in this environment (`-7 days`, `+5 days`, `last monday`) — `date` first on PATH is BusyBox v1.36.1, and its failure is silent-by-default rather than loud.** (2026-08-23, `/weekly_report` skill authoring, bug-1628): BusyBox date *does* parse absolute dates (`date -d 2026-06-29 +%A` → `Monday`), which masks the gap during casual probing, but rejects all relative arithmetic with `invalid date` **on stderr** — so `START=$(date -d "-7 days" +%F)` captures only stdout and yields an EMPTY string. The empty value then flows into `git log --since="" --until=""`, which git treats as *no time filter at all* (unbounded history) instead of erroring, so a broken date computation silently produces a plausible-looking-but-wrong result. Note `/bin/date` IS GNU coreutils 9.8 — the nix profile bin dir simply precedes `/bin` on PATH. **Do date arithmetic in `python3` (`/bin/python3`, always present) via `datetime`/`timedelta`, emitting `VAR=value` lines consumed by `eval "$(python3 - <<'PY' ... PY\n)"`;** call `/bin/date` by absolute path only if GNU date is genuinely required. Corollary that generalizes beyond dates: whenever a computed shell variable feeds a `git log --since/--until`-style range filter, assert it is non-empty first — git's "empty arg means no filter" behavior turns an upstream failure into silent wrong output rather than a crash. Same BusyBox-shadows-GNU family as the `grep -P` and `mktemp` entries above.

- **A scheduler resource-accounting test fixture that lets a job's `xce_res_account_t.residx` resolve LAZILY (left at -1, relying on `sched_update_res_accounts()`'s own first-use `key_find()`/`add_vres()` fallback) can SEGV inside `zarray_get_ex()` instead of failing loudly.** (2026-08-23, scheduler-dispatch-commit task 2.1, bug-1629): a minimal test fixture that never pre-registers a resource name in `core->resmgr` (no real resource-config load, unlike production) hit a silent failure path — `sched_update_account()` discards `sched_update_res_accounts()`'s return value entirely, so if that inner call's `key_find()`/`add_vres()` resolution doesn't complete, `residx` silently stays `-1` with no error surfaced anywhere up the call chain (`sched_commit_decision()` still returns success). `zarray_get_ex()` (`lib/zarray.c:332`) only guards `idx >= array->caps`, NOT `idx < 0`, so passing that leftover `-1` to `get_resvals()`/`put_resvals()` reads `array->data[-1]` — an underflow read ASAN reports as a high-address SEGV, not a clean bounds error. Fix: in any test fixture exercising real charge/accounting code without loading real resource config, pre-register the resource explicitly first via `add_vres(&vres, core.resmgr, name, 0)` and use its returned (non-negative) index for `get_resvals()`/`put_resvals()`, rather than trusting the production lazy-resolution path to run to completion inside a minimal fixture.

- **(2026-08-26, bug-1715) Running xceadmin's pytest suite under `nix develop --command bash -c "... PYTHONPATH=src python3 -m pytest ..."` with a bare `PYTHONPATH=src` (no `:$PYTHONPATH`) silently DISCARDS the whole nix-provided PYTHONPATH (pycapnp/click/jsonschema/pytest itself all live there, per the PYTHONPATH-entries Key Learning above) instead of extending it — fails with `No module named pytest`, easy to misread as "pytest isn't installed in this devshell" when it actually is.** Always use `PYTHONPATH=src:$PYTHONPATH` (prepend, don't replace) when adding `src/xceadmin/src` to the path for a one-off `python3 -m pytest`/script invocation inside `nix develop`.
- **(2026-08-26, bug-1715) The AUTHORITATIVE, complete list of every literal `sched_admin_reply_t.error = "..."` string the real controller/scheduler can send (confirmed by grepping `src/controller/` and `src/scheduler/` for the assignment, then reading each site's surrounding code — not from requirements.md's own worked examples, which under-lists them):** transient/connection-timing (self-resolving, safe to retry): `"scheduler disconnected"` (cluster.c:1062, bug-1693's disconnect path), `"scheduler not connected"` (cluster.c:958, `c->sched.ep==NULL`), `"request timeout"` (cluster.c:1078, `SCHEDADMIN_TIMEOUT_MS` wait expired), `"server busy, try again"` (cluster.c:937, `vtaskmgr_blocking_acquire()` admission-reservation exhaustion — NOT previously documented as transient anywhere, found by reading the admission-gate comment, not the string alone). Permanent/domain (must never retry): `"already recording"` / `"not recording"` (driver.c:390/426, `rc==-EALREADY`) and `validation_error_text()`'s six filename/path-rejection strings (driver.c:298-316: `"invalid record file name"`, `"record directory not found"`, `"record directory path is not a directory"`, `"permission denied accessing record file"`, `"a non-file entry with this name already exists"`, `"record file path too long"`), plus a dynamic (non-literal, can't exact-match) `"cannot write to %s: %s"` I/O-failure message (driver.c:398-404). When classifying an error-reply reason string as retry-safe vs not, read the ERROR-SITE's surrounding code/comments, not just the string's English wording — "server busy, try again" was only correctly classified transient by reading `vtaskmgr_blocking_acquire()`'s admission-gate design comment, not from requirements.md.
- **(2026-08-26, xce-test-rc-file task 3.3, bug-1719) `src/tools/xce-build.sh <component>` with NO `--incremental` does a full `rm -rf "$BUILD_DIR"` + fresh `meson setup` by design (per its own top-of-file comment), then `meson compile` only rebuilds the ONE requested component — so building a second component (e.g. `scheduler`) into an `xce.build` dir that already has a different component's binaries (e.g. `controller`'s `xce_controller`/`xce_test`) silently WIPES the earlier ones.** Always pass `--incremental` when you need a component built alongside another already-built one in the same dir, and never pass an explicit build_dir positional arg unless it matches the script's own default relative-path resolution (`xce.build`, resolved from `src/tools/xce-build.sh`'s own cwd) — passing a repo-root-relative path like `src/xce.build` as the 3rd positional creates a bogus nested `src/src/xce.build` instead (harmless/gitignored via meson's own per-builddir auto `.gitignore: *`, but wastes a build cycle). Rebuilding the wiped component with `--incremental` restores it without disturbing the other.
- **(2026-08-26, xce-test-rc-file task 3.3, bug-1720) `xce-test-resources`'s apiserver layer reporting UNREACHABLE for every resource does not by itself mean a propagation regression — check for a stale bind conflict on :8080 first (`lsof -i :8080` / `pgrep -af apiserver`).** An orphaned `/tmp/apiserver` process from an unrelated, long-concluded prior session (its own deploy dir already deleted, 7.5h uptime) was still bound to :8080, so THIS run's fresh apiserver instance failed `listen tcp :8080: bind: address already in use` and never logged anything — surfacing identically to a genuine trace-line-never-appeared UNREACHABLE. `run-resource-verify.sh`'s own trap-based cleanup only reaps processes IT started, not orphans left by an earlier, already-ended unrelated script/session.
- **(2026-08-31, worker-rejoin-resync design phase) Do NOT reach for `ScheduleWakeup` to wait on background subagents launched via the `Agent` tool.** Called it twice in a row ("wait for these 3 research subagents") purely to pass the time — the tool's own description explicitly says not to ("when harness-tracked work finishes, you are re-invoked automatically"). The subagent completion notifications did arrive on their own regardless, so the calls were pure waste; worse, each one plants a recurring `<<autonomous-loop-dynamic>>` wakeup that keeps firing (with nothing to do) even after the awaited work is long done, until explicitly stopped (`ScheduleWakeup({stop:true})`). Correct behavior when launching parallel `Agent` calls and there's nothing else to do meanwhile: just end the turn (or do other useful reads) — do not call ScheduleWakeup at all unless the user is actually running `/loop` or genuinely polling an external system the harness can't track (a CI run, a deploy).

## Decision Log
- HA/consistency model follows Raft + CP strictly: majority ACK is the hard commit barrier for all state-changing writes (reject on timeout, never local-apply-and-degrade); the own-term commit rule (not a 4th "COMMITTING" state) is what gives a new leader safety.
- A controller process must NEVER self-exit/self-abort as a designed response to an operational condition (only a genuine crash or admin/supervisor kill) — an unrecoverable/fenced member logs loudly, stays permanently non-promotable, stays alive, and keeps retrying.
- Commit-first-then-reply: job/queue/host control ops validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply; no success reply before commit. Worker-reported fact entries instead follow pure Raft semantics — retained/retried indefinitely, no commit timeout.
- No client reply-wait timeout as a fix for a hung request — fail loud at the real transport/source instead of masking with a deadline backstop.
- Product is pre-release: no backward/mixed-version wire compatibility required; wire format changes may be made freely.
- `resource-value-verification` task 7.1 (2026-08-18): `aggregate_verdict()`/`build_verdict_record()` (`run-resource-verify.sh`) re-derive MATCH/MISMATCH/UNREACHABLE from raw `*_OBSERVED` values via `controller_values_match()` at aggregation time, rather than trusting each layer's already-rendered `CONTROLLER_CHECK_RESULTS`/`SCHEDULER_CHECK_RESULTS`/`APISERVER_CHECK_RESULTS` strings from tasks 6.2-6.4 -- deliberate, so the SAME classification code path is exercised by both the live-run wiring and task 8.2's synthetic-record tests (a synthetic test that only fed pre-classified strings would just be testing string formatting, not the comparison logic). A rejected resource name is modeled as its own failure category (`REJECTED`, layer=`injection`) distinct from a per-layer `MISMATCH`/`UNREACHABLE`, and always forces overall FAIL, per design.md's Error Handling ("unrecognized resource name is a run-level failure") -- this is NOT implied by Requirement 6.2/6.3 alone (those only speak to "checked layers"), it's a deliberate design-level extension. The script's bottom dispatch block is now guarded (`[[ "${BASH_SOURCE[0]}" == "${0}" ]]`) so it can be `source`d to get `aggregate_verdict()` et al. in scope with zero side effects (no `main()`/deployment triggered) -- this is the intended integration point for task 8.2's dedicated test harness.
- `resource-value-verification` task 7.1 round 2 (2026-08-18, bug-1582, round-1 REJECTED -- see Do-Not-Repeat entry for the mechanism): `main()`'s OWN record-building loop (feeding `aggregate_verdict()`, distinct from `aggregate_verdict()`/`build_verdict_record()` themselves, which were and remain correct) had the actual bug -- `sched_ok`/`api_ok` derived from `*_OBSERVED` array-key existence instead of the `*_CHECK_RESULTS` classification strings, masking every genuine scheduler/apiserver UNREACHABLE as MISMATCH. Fixed by switching to a leading-`UNREACHABLE:` check on `SCHEDULER_CHECK_RESULTS`/`APISERVER_CHECK_RESULTS` (the controller layer's `ctrl_ok` derivation from `CONTROLLER_OBSERVED` array-key existence was independently confirmed fine and left untouched -- that layer has no separate UNREACHABLE concept). Also extracted the loop into a standalone `build_verdict_records_from_tsv()` function (populates global `VERDICT_RECORDS`) specifically so a synthetic test can exercise `main()`'s actual data-marshaling code, not just `aggregate_verdict()` called with hand-built records -- the round-1 gap was that no test exercised the code that BUILDS records from the `*_OBSERVED`/`*_CHECK_RESULTS` globals, only the pure aggregation function itself. Live-reproduced by killing the scheduler process right after `confirm_deployment_up()` returns (before `inject_resources()`/`query_scheduler_resvals()` ever run) -- deterministic, no race window, and confirmed the controller/apiserver layers are unaffected by an isolated scheduler kill (each layer's check is independent).
- `build-time-versioning` task 4.2 (2026-08-20): `apiserver/meson.build`/`xcecli/meson.build`'s existing `CURRENT_SOURCE_DIR="@CURRENT_SOURCE_DIR@"`/`BUILD_ROOT="@BUILD_ROOT@"` command-array entries (passed to `env_program = find_program('env')`, no shell involved) put literal double-quote characters INTO the env var value seen by the child process — meson's `command:` list is exec'd directly, so shell-style quoting around a placeholder is not stripped by anything; `go_build`'s own `fix_input_envs()` explicitly `sed -e 's@"@@g'`s those two vars to compensate, proving this empirically. When adding new KEY=VALUE entries to this SAME mechanism (`XCE_GIT_COMMIT`/`XCE_BUILD_TIME`, sourced from `get_option('commit_id')`/`get_option('build_time')`, task 1's options — no meson placeholder substitution needed since they're already resolved strings), deliberately did NOT wrap them in quotes (`'XCE_GIT_COMMIT=' + get_option('commit_id')`, not `'XCE_GIT_COMMIT="..."'`) — `go_build`'s task-4.1 code reads `$XCE_GIT_COMMIT`/`$XCE_BUILD_TIME` directly with no quote-stripping, so quoting these would have embedded literal `"` characters into the linked `-X main.GitCommit=...` value. Live-verified via `DEBUG=1` (`go_build`'s own `set -x` trace mode) meson-driven build: clean values (`18ae0cc3`, ISO8601 timestamp) reached `go build -ldflags`, and `xce-build.sh apiserver`/`xcecli` end-to-end produced binaries with `-X main.GitCommit=<hash>-dirty -X main.BuildTime=<ts>` in their `go version -m` build info, no stray quotes. General lesson: "mirror the existing mechanism" doesn't mean copy-paste every byte of an existing established pattern — check whether the DOWNSTREAM consumer (here, `go_build`) actually expects/strips the same decoration before reusing it verbatim.
- `resource-value-verification` spec (2026-08-17): verify scheduler/apiserver resource-value propagation via a one-line trace/debug log at each layer's point of application (`core_events.c`'s RESVALS_DELTA case; apiserver's `OnResourceLocationUpdated`), grepped live, instead of record+replay+REPL or a REST/CLI query — matches how bug-1568/1569 were actually root-caused, and is a stronger read of "the scheduler's own resource state" than a replayed reconstruction. Trade-off accepted: doesn't exercise the real query/REST path, only storage correctness. Load-bearing gotcha: `TRACE`=`tlog_debug3` needs `-vvv`; apiserver's `logger.Debug` needs `logger.yaml`'s `logging.level: debug` — both silently no-op otherwise, producing false timeouts. Also: extend `xce-deploy-ha`'s `deploy-ha.sh` (opt-in, conditional-on-binary-present, mirroring its existing worker/scheduler pattern) plus a generated `start-all.sh`, rather than building a parallel standalone deploy script — avoids re-implementing the same deploy conventions twice.
- bug-1690 (2026-08-25, WAL_JOB_RUSAGE/job.run_usage restart-drop fix): established a NEW apply-seam pattern, distinct from bug-1678/1687/1689's "decode, free-and-replace" shape — **merge, don't replace**, whenever the field being restored can also be the job's LIVE structure (a fact type whose live-mutation call site runs synchronously BEFORE persisting, the same "mutate-then-persist" shape DISPATCH/START/FINISH/RUSAGE all share) and any of that structure's nested pointers can originate from something other than a plain `malloc()` (here: `makeString()`'s xstring pool). `merge_job_runusage()` (job.c) is the reference implementation: overwrite scalar fields unconditionally in place, match-and-update existing nested entries by key (never freeing their pointers), only adopt genuinely-new nested entries by ownership transfer. Also established: for a fact type whose LIVE mutation almost never itself changes `job.state` (a rusage report keeps reporting the same RUN state), gate the type-specific field adoption on nothing narrower than "trailing bytes present" — do NOT reuse START/FINISH/DISPATCH's `result==JOB_TRANSITION_APPLIED` gate, since that would make the new field's adoption dead code for the realistic repeated-report case. Both of these are genuine deviations from the just-established sibling-bug precedent, found by reading the real code (`update_host_runusage()`, `persist_apply_pump_run()`'s doc comment, `apply_job_update_transition()`'s guard) rather than pattern-matching the fix shape — worth treating "mirror the precedent" instructions as a strong prior to verify against the actual code for THIS field, not a template to copy blindly, especially around allocator provenance and state-transition gating.
- **This project's existing home for a standalone, CLI-runnable regression/integration check that needs a REAL build or live deployment (not a meson `test()`) is `src/tools/test-<topic>.sh`** — `test-commit-durability.sh` and `test-heartbeat-tolerance.sh` are the established precedent (shared header-comment style documenting the invariant under test + its spec, `green()`/`red()`/`yellow()` colour helpers, a `pass()`/`fail()` counter convention). Followed this for `build-time-versioning` task 6.1's `test-build-version-regression.sh` rather than inventing a meson test() (two full `rm -rf`+`meson setup`+compile cycles per profile is far too expensive for `meson test`) or a new top-level test-framework directory. When a task explicitly says "use judgment / follow existing convention if the design doc doesn't pin down an exact path," check `src/tools/` first for this class of check before reaching for `testsuite/` or inventing something new.
- **`xce-build.sh` has `set -euo pipefail`, and every profile's `RUN_TESTS` block (asan only, currently) runs the FULL component meson test suite unconditionally as the LAST step of the invocation — so ANY single failing test in that suite (even a pre-existing, unrelated one, e.g. bug-1560's scheduler symlink-escape flake) makes the WHOLE `xce-build.sh <component> <profile> ...` invocation exit non-zero, even though `meson compile` already fully succeeded and the real binary exists on disk.** A caller/wrapper script that treats a non-zero `xce-build.sh` exit as "the build failed, binary unusable" will incorrectly skip/abort real post-build work (e.g. checking `--version` output, as `test-build-version-regression.sh` needs to). Fix pattern: never bare-call it under your own `set -e` — capture the exit code via `cmd || rc=$?` (NOT `if cmd; then ...; fi` with no else, which loses the real nonzero code — POSIX defines an `if` with a false condition and no `else` as exiting 0, discovered live re-deriving this), print a WARNING distinguishing "compile succeeded, only the bundled test run failed" from a genuine failure, and let a SEPARATE binary-existence check be the actual pass/fail signal for "did the build produce something real."- **A `sched_account_jobs()`-shaped no-op stub already present in SOME white-box test files that `#include "../src/sim_loop.c"` directly does NOT mean it's present in ALL of them — adding a genuinely NEW production call inside a white-box-#include'd file breaks every sibling TU that lacks the matching stub, and the only reliable way to find them all is to build and read the linker errors, not to grep-and-guess.** scheduler-sim-decision-parity task 2.1 (2026-08-22): `sim_loop.c`'s round boundary gained a real call to `sched_account_jobs(&(sim->core))` (previously never called there at all, bug-1613). `test_sim_equivalence.c`/`test_sim_resource_contention.c` (task 1.2-era files) already carried a no-op stub anticipating this; 8 OTHER sibling files that also `#include "../src/sim_loop.c"` (`test_sim_event_loop.c`, `test_sim_verify.c`, `test_sim_alt_events.c`, `test_sim_record_output.c`, `test_repl_breakpoints_inspection.c`, `test_repl_snapshot_params.c`, `test_repl_execution_control.c`, `test_repl_equivalence.c`) did NOT, since none of them ever needed the symbol before. `grep -rl '#include "../src/sim_loop.c"' testsuite/` first, then just build — the linker's `undefined reference to sched_account_jobs` errors enumerated the exact fix list precisely, faster and more reliably than reasoning about it from source alone.
- **`xce_request()`/every `xce_*_req()` wrapper (`src/lib/request.c`) is a genuinely non-blocking, fire-and-forget send — it capnp-encodes the request and `endpoint_enq()`s it onto `ep->outq`, then returns; it never waits for a reply itself** (the synchronous `xceapi.c` wrappers like `xce_hello`/`xce_jobsubmit` layer a separate `cbuf_timedget()` wait on top, but the raw `xce_*_req()` call a controller-side handler like `dispatch_a_job_()` makes does not). `endpoint_enq()` (`netio.c`) succeeds unconditionally against a **zeroed `endpoint_t` with `ep->iomgr == NULL`**: `dataq_enq()` has no failure path at all, and `endpoint_update_epoll()` is a pure no-op whenever `ep->iomgr == NULL` (skips the whole `iomgr_update_poll` branch). Net: to make a controller-side `xce_*_req()` call "succeed" in a unit test, you do NOT need a live peer, a synthetic reply thread, or a socketpair — a `static endpoint_t` (or embedded field) zeroed with `.fd = -1` and left with no `iomgr` is sufficient, and packets just accumulate on `ep->outq` for the test to `endpoint_deq()`/`free_packet()` afterward. This exact pattern is already established/proven live by `test_commit_barrier_reply.c`'s `g_wkr_disp_ep`/`cbrp_arrange_jobdispatch()` fixture (its own file header even documents it: "a zeroed static endpoint_t with no iomgr... packets stay queued") — reused directly for `scheduler-dispatch-commit` task 3.1's `test_dispatch_commit.c` success-path tests, no mocking/round-trip machinery needed.
- **`controller_unit_test` (and presumably every other `argp`-based Unity runner built the same way in this project) supports a repeatable `-t <group-name>` (or `--run <group-name>`) CLI filter** (`controller_test.c`'s `TEST(x)` macro checks `check_test(&arguments, #x)` before calling `x()` — with zero `-t` args every group runs, with one or more only the named `TEST(...)` group(s) run, matched against the literal `TEST(test_foo)` identifier as a string). Invaluable for isolating ONE new test group from a suite-wide hang/flake without touching any source: run the already-built binary directly, e.g. `./controller_unit_test -t test_dispatch_commit`, or build a `-t` list for "everything except the one broken/hanging group" via `grep -oP '(?<=^extern void )test_\w+(?=\(void\);)' controller_test.c | grep -v '^test_broken_group$'` and pass each as its own `-t name`. This bypassed a pre-existing, already-tracked hang (bug-1429, `test_commit_barrier_reply.c`'s `test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs`, unrelated to the diff under test) cleanly — confirmed a new test group's 3 tests all passed AND that every other group (91 of 92) still matched the pristine-tree failure count exactly, without needing to touch/skip/comment out any test source.
- **bug-1429 (`controller_unit_test` TIMEOUT 300s/SIGTERM, NOT root-caused) has now recurred a 3rd time (2026-08-23, `scheduler-dispatch-commit` task 3.1), each time with a COMPLETELY unrelated diff and a DIFFERENT hang location** (`test_demotion_replay_yields_committed_only` → `test_configack_*` group → `test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs`, all in `test_commit_barrier_reply.c`/`test_commit_barrier_rollout.c`). Before concluding a suite-wide TIMEOUT is caused by your own change, check `.wolf/buglog.json` for bug-1429 first, then get an independent data point on a genuinely pristine tree (`git stash -u` your own diff, rebuild+retest once) — a pristine run completing cleanly while your-diff runs hang 2-for-2 does NOT by itself prove causation when the hang location has zero code-path relationship to your diff (traced directly: the hanging test never calls the function you changed) and the SAME symptom is already logged against two prior, unrelated diffs. The `-t` filter (above) is the fastest way to get a real, uncontaminated pass/fail answer for your OWN new tests when a pre-existing hang blocks the full suite.
- **In the current (pre scheduler-sim-decision-parity tasks 2.2-2.4) codebase, a RUN-state job's `job->accounts`/`job->allocs` is ALWAYS NULL in simulation mode — there is no live path that ever sets it.** Two independent, compounding reasons, both confirmed by reading the real code: (1) `sim_picker_thread_fn()`'s (`sim_loop.c`) own picker thread unconditionally PENDs every candidate (`picker->picker_iface->on_pend()` + `make_pend_decision()`) — it never emits a live DISPATCH (documented "Known scope boundary": real allocation needs a live `xce_scheduler_t*` the sim driver deliberately doesn't have, belongs to the not-yet-designed scheduler-allocation spec). (2) Even if it did dispatch, `sched_commit_decision()` (`scheduler.c`) — the ONLY function that ever does `job->allocs = decision->u.alloc; job->accounts = make_job_account(...)` — is only ever called from `sched_allocator()` (scheduler.c's own real allocator thread), which sim mode never runs; sim's own decision-drain loop (`sim_loop.c`) reads `core->decisions` directly and just records/prints/REPL-checks each one, with no `sched_commit_decision()` call anywhere in that path. Consequence for task 2.1: `sched_account_jobs()`'s new round-boundary call (which only charges a job whose `job->accounts != NULL`) has genuinely nothing to charge yet in isolation — it only becomes behaviorally meaningful once Recorded-Decision Reconciliation (tasks 2.2-2.4) populates `job->accounts` from `decoded_session->decisions`. Task 2.1's own regression test therefore has to prove the WIRING (is `sched_account_jobs()` called, once per round, inside the session-active window) via a spy, not an end-to-end "resInUse becomes nonzero" assertion — that end-to-end behavior only becomes true after 2.2-2.4 land.
- **`sched_sim_driver_run()` (`sim_loop.c`) is a single monolithic entry point — unlike `driver.c`'s decoupled, single-round `xce_sched_driver_run_normal_mode_round()` — that opens the replay file, spawns its OWN picker thread (`sim_picker_thread_fn`) and bootstraps the real `builtin_fcfs` plugin (`sim_bootstrap_picker()`, `sim_picker.c`) internally, all inside one call.** This means `test_driver_session_ordering.c`'s cross-thread wbuf-rendezvous technique (substitute a test-controlled picker plugin, spawn it in a test-owned thread, call the driver's one-round function directly) is NOT reproducible against the sim driver without either modifying `sim_picker.c` (routinely outside a narrowly-scoped task's boundary) or restructuring `sched_sim_driver_run()` into smaller testable pieces (not what design.md's File Structure Plan asks for — it's described as an inline round-boundary edit, not a new decomposed interface). Pragmatic substitute used for task 2.1's `test_sim_session_ordering.c`: since `xce_core_begin_session()`/`sched_account_jobs()`/`xce_core_signal_go()` are three unconditional, sequential, same-thread calls in `sim_loop.c`'s own source (no branch can reorder them), the only genuinely runtime-observable fact is "does it actually get called, and does it see `core->session_active==1`" (proven via a spy) — NOT a full cross-thread ordering race, which doesn't exist to prove in the first place given `cbuf`'s already-tested blocking rendezvous semantics.
- **`controller_unit_test`'s notify() capture (`test_commit_barrier_reply.c`'s `g_notify_*` arrays) is `static`/file-local, so no OTHER test file in the same binary can read it directly — even though that file is the sole `notify()` definition linked into the binary (`dataex.c`'s real one is excluded).** scheduler-dispatch-commit task 3.3: to let a new test file (`test_dispatch_fail_retry.c`) observe `notify_dispatch()` (job.c) actually firing, added a handful of tiny non-static `cbrp_notify_capture_*_ext()` accessor functions to `test_commit_barrier_reply.c` (count/event/jobid/code by index, plus a reset wrapper) rather than duplicating a second `notify()` stub (impossible — duplicate-symbol link error across a single binary) or skipping the observation entirely. Read-only, additive, does not touch that file's own capture/reset logic or any of its existing tests — a low-risk way to extend an existing test's shared mock surface to a sibling file in the same binary.
- **`dispatch_a_job_()`'s `key_find(c->workermgr, ...)` for a genuinely UNREGISTERED worker key returns `rc>=0` with `w` left NULL (hits `XCE_DISPATCH_FAIL_SITE_WORKER_NULL`), not `rc<0` (`XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP`)** — confirmed empirically (task 3.3), matching `test_dispatch_commit.c`'s own prior comment. A genuine `key_find()` internal failure (rc<0, the WORKER_LOOKUP site) has no known black-box trigger from a test without reaching into librtti's own (external, opaque) internals — `test_dispatch_fail_retry.c`'s `test_worker_notfound_resolves_retryable` deliberately documents that it covers "whichever of WORKER_LOOKUP/WORKER_NULL the runtime takes" rather than claiming to isolate one, since both sites share the exact same classify/resolve code shape and the test's assertions hold for either.
- **A real, unregistered-with-its-iomgr socket fd (`socket(AF_INET,...)` + a live `iomgr_t` via `iomgr_init()`, `ssl_handshake_done=1`, `flags=0`, never `EPOLL_CTL_ADD`'d) is a reusable, deterministic technique for forcing an `xce_*_req()` call to fail** — `EPOLL_CTL_MOD` on the unregistered fd fails ENOENT every time (`endpoint_update_epoll()`), and since `ep->flags` never advances past 0 on failure, the SAME broken endpoint can be dispatched against repeatedly in a loop for a bounded-retry-escalation test, no per-attempt endpoint reconstruction needed. First established by `src/lib/testsuite/test_netio_arm_failure.c`/`test_request_send_propagation.c` (bug-646/650); reused directly at the controller layer for `test_dispatch_fail_retry.c`'s `XCE_DISPATCH_FAIL_SITE_RPC_FAILURE` tests (task 3.3) by pointing a `worker_t.ep` at the broken endpoint instead of the established "zeroed, no-iomgr ep always succeeds" fixture task 3.1 uses for the success path.
- **A prior task's own test file can pre-emptively flag a specific assertion as "this will need to change once task N.M lands" in its own comment/message string — grep for the LATER task's own number in sibling test files before assuming "task boundary == only my new file changes."** Task 3.1's `test_dispatch_commit.c::test_dc_worker_not_found_leaves_job_pend_and_alloc_uncommitted` asserted `dispatch_fail_count` stayed unchanged with the message "incrementing it is task 3.3's job, not 3.1's" — a deliberate, anticipated staleness, not an accidental regression. Confirmed via the mandated RED (git-stash job.c only, rebuild, run new tests → fail as expected) → GREEN (restore, rebuild) cycle: the GREEN pass surfaced this ONE pre-existing sibling-test failure (seeded counter 2 → now correctly 3), fixed by updating that one assertion + its message, not by touching my own new file. This is explicitly within "test infrastructure under testsuite/ may change" boundary language even when the task brief's boundary section only names the NEW file — a task boundary permitting "test infrastructure" changes covers fixing a sibling test's own foreseen staleness, not just adding new tests.
- **A "mirror the precedent's free-and-replace pattern exactly" instruction (bug-1678/1687/1689's own established shape) can be actively WRONG for a new field if that field's live-mutation path sources any nested pointer from `makeString()` (bug-1690, WAL_JOB_RUSAGE/job.run_usage).** `update_host_runusage()` (misc.c) builds `xce_host_runusage_t.host_name` from `makeString()` — an RTTI-managed, refcounted, HEADER-BEFORE-DATA pointer (confirmed by reading librtti's actual source in the nix store, `RTTI_DATA_TO_OBJ`: `((meta_obj_t*)(((char*)v) - RTTI_OBJ_HEADER_SIZE))`), NOT a plain `malloc()`. `persist_apply_pump_run()` (the LIVE online-apply pump, confirmed via its own doc comment to run continuously on the primary itself, not just boot replay) re-applies every committed fact — including one the SAME process just live-mutated moments earlier via the synchronous "mutate-then-persist" pattern (the SAME pattern bug-1678's `dispatch_a_job_()` uses for `job.allocs`). For DISPATCH this is safe because `job.allocs`'s nested `host` field is genuinely capnp-wire-decoded end-to-end (arrives from the scheduler already deserialized) — `free_JobAlloc_ptr()` on it is always consistent. For RUSAGE it is NOT safe: calling the capnp-generated `free_JobRunusage_ptr()` on the LIVE (xstring-backed) structure would `free()` a non-malloc'd pointer — guaranteed ASAN heap-corruption abort on the very first live report for a new host. **General check before reusing this "free-and-replace at apply time" precedent for any NEW WAL fact type: grep the field's LIVE-mutation call site for `makeString()` (or any other non-malloc allocator) on any pointer the apply seam might later free — if found, replace with a MERGE (overwrite scalars in place, only adopt genuinely-new nested pointers by ownership transfer, never free an existing one) instead of a free-and-replace.** Proved with a real regression test using an actual `makeString()`'d pointer and asserting POINTER IDENTITY survives the merge (not just value equality) — a value-only check would not have caught this class of bug. Also found, but explicitly NOT fixed (separate file/component, out of scope): `job_destroy()` (`src/lib/jobj.c`)'s own existing manual `FREEUP`-based walk over `run_usage` has this exact same latent host_name/xstring mismatch, independent of and pre-dating this fix.
- **This project's Unity build has `TEST_ASSERT_EQUAL_DOUBLE` silently non-functional (fails every call with "Unity Double Precision Disabled") because `unity_dep = dependency('unity')` is a prebuilt nix package with no `-DUNITY_INCLUDE_DOUBLE`** (Unity's own default is double support OFF unless explicitly opted in; float support is ON by default). Grep any existing test file comparing a C `double` field before assuming `TEST_ASSERT_EQUAL_DOUBLE` works — the established codebase convention (e.g. `test_apply_committed_entry.c`, `test_dispatch_commit.c`, `test_worker_join.c`) is `TEST_ASSERT_EQUAL_FLOAT` even for genuinely-`double`-typed fields (implicit narrow to float; fine for clean/round test fixture values). Caught only by actually running the RED build and reading the real failure message, not by reasoning from Unity's public API alone.
- **When offloading a newly-discovered delaymgr/iomgr blocking-I/O violation, "vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK)" is NOT automatically the right destination even though it's this codebase's most commonly cited fix-pattern name — check whether the SAME pool already carries a lock the new work also needs, via a DIFFERENT already-offloaded caller** (bug-1694, checkpoint_task_fn offload, 2026-08-25). This codebase actually has TWO distinct offload destinations, easy to conflate: the vtaskmgr N-thread pool (`vtask_enq_classified`, VTASK_LIVE_BLOCK/VTASK_REQ_BLOCK — used for *bounded* handler work, e.g. `persist_apply_pump_kick`'s apply-pump drain and `commit.c`'s `dispatch_continuation`) vs. the dedicated SlowLane (`slowlane_submit`, `c->slowlane`, `n_workers` separate pthreads — used for genuinely *long/unbounded* blocking I/O: snapshot send, config-push, reconnect, catch-up-streaming, demotion-truncate). `ha-async-io.md`'s own table already states this split ("vtaskmgr must NOT do: long/unbounded blocking I/O ... starves the pool") but it's easy to miss which category new work falls into. Concretely: `persist_apply_pump_run()` (the live WAL-apply walk) holds `apply_pump_lock` across its ENTIRE walk and is ITSELF dispatched onto the vtask pool via `persist_apply_pump_kick()`'s `vtask_enq_classified(..., VTASK_LIVE_BLOCK)` — so any OTHER unbounded, lock-holding work (checkpoint's `write_snapshot()`, also under `apply_pump_lock`) put on that SAME pool would risk starving it (a worker parked on the lock for the whole snapshot-write duration, stealing capacity from commit-continuation/apply-pump liveness work) — not a classic deadlock, but the exact "starves the pool" failure mode the split exists to prevent. The fix used the slow lane instead, exactly mirroring `ha_snapshot_do_send()`'s pre-existing (already-in-production, default-ON) call to the SAME function (`persist_checkpoint()`) from a slow-lane thread — i.e. the safest new offload destination is often "wherever an already-proven-safe sibling call already runs the identical function from," not a fresh pattern match on a fix-recipe's literal wording. **General technique: before picking an offload destination, grep every existing caller of the SAME blocking function/lock for its current thread context — if a sibling caller already runs it safely from candidate destination X, reuse X; if a DIFFERENT already-offloaded consumer of the same lock lives on candidate destination Y, that's evidence AGAINST Y.**
- **A buglog entry's own "root cause: NOT YET ROOT-CAUSED" (or similarly hedged) status is a real signal to re-derive the causal claim from primary evidence before citing it as settled, especially when later re-occurrences of the SAME symptom show it firing on completely unrelated diffs.** bug-1429 (cited by `config_repl.c`/`ha_convergence.c`'s catchup/demotion offload-flag comments as "the vtask pool that deadlocked controller_unit_test on 2026-07-10") is itself logged as never root-caused, and its OWN later occurrences (2026-08-20, 2026-08-23) reproduced the identical TIMEOUT/hang signature on diffs that touched neither the vtask offload code nor persist.c's checkpoint path at all — "strongly corroborates this is a pre-existing, systemic, timing/order-dependent hang in the controller_unit_test harness," per the buglog's own later entries, not proven caused by VTASK_LIVE_BLOCK dispatch specifically. Don't repeat a downstream comment's causal claim as fact in a new design doc/comment without checking the buglog entry it cites — the comment can be more confident than the evidence it's citing.
- **`controller_unit_test` (the main shared meson.build `controller_unit_test_src` binary) does NOT link the real `sched.c`/`dataex.c` at all** (bug-1693, 2026-08-25) — `sched.c`'s own `on_scheduler_join()`/`collect_confvars()` and `dataex.c`'s `notify()` collide at link time with fake local definitions several OTHER files in that same binary already provide (`test_commit_barrier_reply.c`, `test_worker_join.c`), so `test_config_repl.c` instead ships a trivial no-op `handle_scheduler_leave(cluster_t *c) { (void)c; return 0; }` stub purely to satisfy the linker. **Any new/changed behavior added to a real `sched.c` function is therefore SILENTLY UNTESTED by any `controller_unit_test`-based test that calls that function by name** — the call resolves to the stub, not the real implementation, with no compile/link error to flag the mismatch (confirmed empirically: a new regression test asserting sub-second wake timing after calling `handle_scheduler_leave()` passed compilation and linking fine, then failed by taking the full 5s deadline — the stub had silently swallowed the call). `nm '<obj>.o' | grep ' T <symbol>$'` across every `.o` in the target's build dir is the fast way to confirm which TU actually provides a given symbol in a given binary before trusting a test's result. The two standalone binaries that DO link the real `sched.c` (`sched_push_unit_test`, `sched_worker_change_unit_test` — see their own `meson.build` comments for the collision rationale) don't link `cluster.c`, so neither binary alone can exercise a fix that spans both files for real; bug-1693's fix split verification across both (a direct-call test against `cluster.c`'s real new function in `controller_unit_test`, plus a small counting-spy test in `test_sched_worker_change.c` proving `sched.c`'s real caller invokes it) rather than inventing a third binary that links both (would need to duplicate most of `controller_unit_test_src`'s already-large dependency list to satisfy `cluster.c`'s own huge fan-out). General lesson: when a fix spans a function in `sched.c`/`dataex.c` AND a function in any of the ~15 other files real-linked into `controller_unit_test_src`, check with `nm`/grep for a same-name fake stub in a sibling test file in that binary BEFORE trusting a test that calls the `sched.c`/`dataex.c` side by name — it may be exercising a stub, not your change.
- **(2026-08-25, bug-1695) A bootstrap "is a leader already known" fallback check must compare a term against a snapshot taken at THIS boot's start, never against a bare constant like `> 0`, whenever the compared value is restored from durable on-disk state that only ever increases (Raft terms, monotonic counters, etc.).** `myapp_run()`'s `raft.term > 0` fallback was trivially true forever after the cluster's first-ever election because `ha_restore_term()` restores a persisted, never-decreasing term at every boot. Fixed by capturing `term_at_boot = ha->raft.term` immediately after `ha_init_raft()` seeds it (right after restore, strictly before any new activity could legitimately advance it) and comparing `> term_at_boot` instead of `> 0` — the general pattern: "is X known/true" checks against monotonic restored state should ask "did X change SINCE I booted," not "is X nonzero," whenever a nonzero-forever steady state is a normal, expected outcome of prior successful operation (not an error state).
- **(2026-08-25, bug-1697, CRITICAL — live, no-restart data bug) Whenever a semantic change makes one code path (e.g. a scheduler dispatch decision) start ANTICIPATING a state transition another, later code path (e.g. a worker's ack) used to be the sole author of, re-audit EVERY `result == JOB_TRANSITION_APPLIED`-gated (or equivalent "did this actually change anything" gated) side effect on that later path — the earlier path's own transition now permanently makes the later one a same-state no-op, silently killing every effect gated on "this was a real transition."** `dispatch_a_job_()` (job.c:1686, scheduler-dispatch-commit task 3.1) was changed to set `state = JOB_STAT_RUN` synchronously at DISPATCH time (the controller now decides RUN, it no longer waits for a worker ack) — but `apply_committed_entry()`'s WAL_JOB_START case (added by bug-1687, persist.c ~4594-4637) still gated its `start_time`/`pidinfo` wire-decode adoption on `result == JOB_TRANSITION_APPLIED` from `apply_job_update_transition()`, which now ALWAYS returns `JOB_TRANSITION_NOOP_DUPLICATE` for WAL_JOB_START (job.c:2795-2799, state already == RUN from dispatch) — so start_time/pidinfo silently never get set, for every job, live, no restart needed. The exact same class of bug had ALREADY been fixed for WAL_JOB_RUSAGE/CKPT/PREASON in this identical function (their own comments describe the identical hazard) — a fix for one wal_type in a shared-transition-guard case group does not imply sibling types sharing the SAME guard got the same fix; grep the whole case group whenever changing which code path is allowed to "arrive first" at a shared target state. bug-1679's own comment (job.c:1709-1719) had ALREADY independently hit and worked around the identical root collision for a different symptom (apiserver update-notification hook not firing) months/fixes apart from this one — the two were never connected as one root cause until a live "start_time is always 0" report forced re-deriving it from scratch. **General technique: after any change that makes state transition X reachable from a NEW, earlier code path, grep every existing consumer of "was this transition genuinely applied" (`result == *_APPLIED`, `state != cur`, etc.) that used to assume X's only author was the OLD path — each is a candidate for silently going permanently dead.** **FIXED (2026-08-26):** moved `start_time` ownership to the dispatch-time mutation itself (`WAL_JOB_DISPATCH`'s payload extended with the new `WAL_JOB_DISPATCH_HAS_START_TIME_FLAG` `_pad` bit, mirroring `WAL_JOB_FAIL_STAGE_FLAG` — NOT a length heuristic, since `JobAlloc`'s own variable length makes old-vs-new payload length ambiguous); `pidinfo` stays sourced from the worker's ack but its adoption is now unconditional like RUSAGE/CKPT/PREASON. RED confirmed by temp-reverting only the two defect lines (job.c's `start_time` stamp; persist.c's pidinfo `result==APPLIED` gate) and rebuilding — 5 genuine new failures at the exact production call sites (`dispatch_a_job_test`, `apply_committed_entry` seeded already-RUN). GREEN confirmed after restoring: "1198 Tests 3 Failures 0 Ignored" 3x direct runs, matching known baseline exactly.
- **(2026-08-26, bug-1718, xce-test-rc-file task 3.1) A `python3 -` (or any "read program from stdin") invocation placed on the SAME command as a `<<HEREDOC` redirect while ALSO receiving piped input (`upstream | python3 - args <<EOF ... EOF`) silently loses the piped data, no error.** The heredoc redirect wins stdin over the pipe for that command; `python3 -` consumes the heredoc's content as its own program source, so by the time the script's own `sys.stdin.read()` runs, stdin is already at EOF — it just returns `""` with zero diagnostic. Caught by re-reading the draft before ever running it, not by a failing run. **Fix pattern: write the embedded program to its own temp file ONCE (`mktemp` + `cat >"$DRIVER" <<EOF...EOF`, cleaned via an `EXIT` trap), then invoke it as `upstream | python3 "$DRIVER" args...`** — a file argument doesn't compete with the pipe for stdin, unlike `-`. Applies to any interpreter's "read program from stdin" mode (`python3 -`, `perl -e -`, etc.) combined with a heredoc AND a real piped payload in the same pipeline.
- **(2026-08-26, xce-test-rc-file task 3.1) `xce_test`'s `cmd_set(argc==1)` (`set` with no args) is the generic, no-live-cluster-required readback for ANY setting** (`test.c:3684`, format `"%-12s:%s default: %s\n"`, `KVAL()` prints `"<not set>"` for a NULL current value) — the right primitive for any integration test that needs to observe an rc-file/`set`-driven setting change (e.g. `masters`) without a live controller connection. Don't hardcode the exact column padding when grepping its output (fragile) — extract via `sed -E 's/^masters[[:space:]]*:(.*) default:.*/\1/'` (greedy `(.*)` correctly captures the embedded-space `"<not set>"` sentinel too, since there's only one literal `" default:"` substring per line to anchor against).
- **(2026-08-26, xce-test-rc-file task 3.1) `xce_test` uses `libedit`'s `readline()`/`isatty(STDIN_FILENO)`-gated `g_interactive`, so a genuine interactive-mode integration test needs a REAL pty, not a piped stdin** — confirmed working via a small embedded Python `pty.fork()` driver (write commands to the master fd with a short settle delay after each, drain with `select()`, detect child exit via non-blocking `waitpid`, `os.waitstatus_to_exitcode()` for the real exit code). Same technique task 2.2's own implementer used for manual verification; this task turned it into a reusable scripted harness. The prompt itself (`fill_prompt`, top-level context) renders as literal `"xce>"` — grepping for that substring in the pty transcript is a good "this really was a TTY session, not a piped fallback" sanity check, distinct from the actual assertion under test.