myxce/.wolf/cerebrum.md

# Cerebrum

> OpenWolf's learning memory. General, reusable knowledge only — bug-specific history lives in .wolf/buglog.json, not here.
> Do not edit manually unless correcting an error.
> Last updated: 2026-08-16 (condensed from a ~40k-token file; see .wolf/buglog.json for full bug history)

## User Preferences
- Run soak tests with specific round counts and deploy directories for precise regression tracking.
- Prefer parallel agent runs for independent analysis work; sequential for dependent tasks.
- Keep test logs for analysis, don't skip intermediate output even if a test hangs.

## Key Learnings
- **Build entry point**: `flake.nix` lives in `src/`, not the repo root. Always `nix develop /abs/path/to/src --command bash -c '...'` with an explicit path — Bash cwd isn't stable across calls, and a bare `cd src && nix develop` can silently short-circuit.
- **Test coverage gaps**: documented full-suite commands (`xce-build.sh <component>`, `make test`) each cover only a subset — e.g. a component's default meson run misses newly-added standalone test binaries unless wired into its `RUN_TESTS` case, and the CGO/apiserver backend suite needs `make test-backend` with CGO_CFLAGS/LDFLAGS reconstructed via pkg-config (package is `c-capnproto`, not `capnp_c`). A green default build is not proof of full coverage — verify every new test binary actually runs.
- **Unity harness**: use `UnitySetTestFile(__FILE__)` (not nested `UNITY_BEGIN()/UNITY_END()`) when multiple test-group files share one binary — nested BEGIN/END resets global counters and can mask most of the real suite exit code. Authoritative green = grep raw testlog for `:FAIL:` / word-boundary `\bTIMEOUT\b`, never trust an aggregate `Fail:0` alone. Also rule out a concurrent competing build/soak against the same build dir before treating a timeout/crash as a regression.
- **apiserver zap `encoding: console` still renders structured fields as trailing JSON**: `logger.yaml`'s `console` encoder is only for the message/level/time/caller prefix — any `zap.Field`s passed to `logger.Debug/Info/...` still render as a single trailing `{"key": val, ...}` JSON object at end of line (e.g. `... resource-location stored {"resource": "r15m", "host": "resverify-worker", "value": 1.5}`), in the exact field-append order from the call site. Don't assume `encoding: console` means key=value text — grep/parse trailing zap log fields as a JSON tail block (index()/substr() in bash is fine, no JSON lib needed for a fixed, known field order) regardless of the configured encoding name.
- **capnp/typemap conventions**: every schema struct gets `$C.mapname(...)`; for a wire-only concept with no live C struct, hand-define a small matching struct in `typemap.h`. A `Data`-typed field's generated composite codec is a silent no-op in this backend AND the generated `_ptr` wrapper doesn't zero-init — an unset field is written as literal uninitialized stack garbage (real crash risk). Prefer `List(UInt8)` + a `$C.maplistcount` field, which this backend does generate real code for. A nested-struct getter can return an unresolved far pointer after a segment round-trip — call `capn_resolve()` before reading its scalar fields (Text/Data getters resolve internally and mask this).
- **C comments**: never write a bare `type*/` (pointer type immediately followed by `/`) inside a `/* */` block comment — it silently closes the comment early, producing a cascade of unrelated syntax errors elsewhere. Recurred repeatedly; grep new/edited comments for `[A-Za-z0-9_]\*/` before building. Same failure mode, different trigger (caught before building this time, writing `test_sched_worker_change.c`'s file header): quoting a piece of source code that itself contains `/* ... */` (e.g. describing the literal dead `#if 0 /*FIXME*/ ... #endif` block from `sched.c`) inside your own outer `/* */` doc comment closes the outer comment at that inner `*/` too — C block comments never nest. When a doc comment needs to reference literal `/* */`-bearing source text, paraphrase it (e.g. "a dead #if 0 (FIXME-commented) block") instead of pasting the real comment markers.
- **Thread/lock hazards on async-ified single-threaded code**: a library/struct built for serialized calling (Raft state, SSL* objects, an epoll batch-dispatch loop) that later gets a "drive it on a thread pool" feature needs its own lock/refcount around the *shared* state — per-endpoint or per-callback serialization does not imply serialization across different endpoints. A same-thread same-batch use-after-free (one event's handler freeing something a later event in the identical batch still references) can look identical to a genuine cross-thread race.
- **makeString() (lib/xstring.c)** is rtti-backed, not malloc/strdup — only free it via its own module's rtti-aware teardown, or don't free it at all; a generic `free()`/capnp `free_X()` on it is heap corruption. Use `strdup()` for fields a generated `free_X()` will free directly.
- **Threaded test fixtures**: teardown for a fixture with real background threads must run *unconditionally* before any `TEST_ASSERT_*` — a failed assertion longjmps past later code, orphaning the thread and corrupting the next test's fixture memory.
- **Live-soak validation discipline**: a unit-test-green fix can still be unreachable/dead code in production — always re-run the exact live scenario end-to-end, not just the mechanism fixed. A later, differently-shaped soak failure after a fix isn't automatically a regression from that fix — check whether that failure point was ever exercised before.
- **Sandbox quirks**: `/tmp` can be overlayfs with intermittent fdatasync EIO — durability-sensitive tests need a real fs (e.g. `~/tmp` on ext4). This sandbox's `date` may resolve to BusyBox (no `%N`, silently passes it through instead of erroring) — use bash's `$EPOCHREALTIME`/`SECONDS` builtins for timing instead.
- **HA election timer discipline (bug-1562)**: every early-return branch in a function driven by a one-shot timer (`ha_on_primary_ctrl_fd_rdhup`'s liveness-watchdog callback) must either re-arm the next timer itself or verify some OTHER live timer is actually still scheduled before skipping the re-arm — checking only elapsed-time-since-last-event (the 5.10b debounce branch) is not the same as checking an election is still in flight (the 5.6 guard's actual role/pre_vote_in_progress check). A silently-dropped re-arm freezes the node with zero CPU/log activity forever (looks identical to a clean idle state, not a crash) — diagnose via `/proc/<pid>/task/*/wchan` (all threads in `futex_do_wait`/`do_epoll_wait` with no fds beyond listen+epoll = dropped timer, not deadlock) before reaching for gdb (ptrace is blocked in this sandbox, `yama ptrace_scope=1` + no ancestor relationship to the target).
- **Reproducing HA bootstrap bugs**: match the user's actual inter-node start stagger exactly — a tight ~0.5s stagger (the `/xce-deploy-ha` skill's default) can mask a bug that only triggers when a node sits solo for tens of seconds (real peers answering fast enough prevents the trap window from ever closing). Always ask/check the real timestamps before concluding "could not reproduce."
- **A soak's own "excluded"/PASS verdict is not a license to stop reading (bug-1563)**: `/xce-test-ha --chaotic`'s verdict deliberately excludes `bug924_fdatasync` (overlayfs EIO) from pass/fail — but a huge excluded count (300K+ in one run) or a different errno than the documented one (EINVAL, not EIO) is worth a manual look anyway. Found: `fsync_worker`'s retry loop only backs off when NOT pending, and a sustained fdatasync failure makes "pending" permanently true (durable_lsn never advances on failure) — a real busy-spin/livelock, invisible to the soak's own job-loss/divergence checks since the other 2-of-3 standbys still satisfy majority. Lesson: skim excluded/noise counts for magnitude and errno before trusting a documented exclusion covers them.
- **Fixing a busy-spin/livelock can unmask a latent notification gap (bug-1564)**: throttling an unbounded retry loop (bug-1563's fix) removed an *accidental* fast-repoll that had been quietly covering for `persist_flush()` never firing `on_durable_advance` (only the async `fsync_worker` did). A deterministic (3/3) new test failure appeared immediately after — always suspect "the old code accidentally covered for a real gap via sheer retry speed" when a correctness fix (removing a spin/race) makes a previously-green test start failing 100% of the time; that's a signal to look for a missing notification/wakeup, not to revert the fix.
- **nix devShell Python packages are `PYTHONPATH` entries, not merged into `python3`'s own site-packages** (found building a standalone-installable xceadmin wheel): `python3 -m venv --system-site-packages` only inherits the base interpreter's OWN site-packages dir, which is empty in this project's nix shell (`python3Packages.click`/`pycapnp`/etc. are each a separate `/nix/store/...` path added via `$PYTHONPATH`, confirmed via `nix develop … --command sh -c 'echo $PYTHONPATH'`). A `--system-site-packages` venv therefore silently CANNOT see them (`ModuleNotFoundError` at runtime, discovered live). Fix: skip venv/pip entirely — extract the wheel directly (`python3 -m zipfile -e`) and bake the *resolved* `$PYTHONPATH` string into a wrapper script's `export PYTHONPATH=...` at generation time (expand now, not `${PYTHONPATH:-}` at runtime) — mirrors `src/xceadmin/meson-install.sh`'s existing nix-packaging wrapper, which already solved this correctly; I should have grepped for that pattern before reaching for venv the first time.
- **`src/tools/xce-build.sh`'s `setup_build()` does `rm -rf "$BUILD_DIR"` for every single-component invocation** (bug-1565, FIXED): `/xce-build xceadmin` after `/xce-build controller` in the same dir used to silently delete `xce_controller`/`xce_test`/etc. Fixed by an opt-in `--incremental` flag (`meson setup --reconfigure` in place instead of wiping) — deliberately NOT the default, since this project's capnp-generation `custom_target` has known missing-dependency-edge races that a full wipe papers over as a loud, retry-fixable failure (skipping the wipe could turn the same gap into silent stale-generated-code instead). `/xce-build all` (build everything together) remains the simplest option when you don't specifically need incremental speed.
- **A never-joined background thread reading a stack-local test fixture is a latent ASAN stack-use-after-return, exposed only by test ORDERING, not by the thread's own logic** (2026-08-19, scheduler-core design-review follow-up). `sched_sim_driver_run()` (sim_loop.c) spawns `sim_picker_thread_fn` on a real pthread that is, by this codebase's own documented design, "never joined or cancelled... no sim-driver teardown/fini function exists yet" — it loops forever on `cbuf_get(&sim->core.wbuf, ...)`. Every existing white-box test in `test_sim_event_loop.c` already called this with a stack-local `xce_sched_sim_driver_t sim;`, which was ALWAYS a latent stack-use-after-return once the test function returned — it just hadn't been caught, because no test's stack frame had yet been reused by a later frame in a way ASAN's shadow-memory detector caught. Simply INSERTING one new test between two existing ones (no logic change to the existing tests) shifted the stack-reuse pattern enough to make it fire deterministically (reproduced twice). Fix: change the fixture from stack-local to `static` storage — safe here specifically because (a) `sched_sim_driver_start()` unconditionally `memset()`s the whole struct as its first action so no stale-data leak across runs, and (b) each affected test function is called exactly once per process (verified via grep, no re-entrancy) so a second call's memset can never stomp a still-live zombie thread's cbuf mutex/condvar state from a first call — `static` would be UNSAFE to apply to a helper called more than once while its first zombie thread might still be alive. General lesson: when a background thread spawned by code-under-test is documented as "never joined," any test giving that code a stack-local fixture is a ticking ASAN time bomb regardless of whether it has fired yet — check for exactly this "never joined/cancelled" phrasing in the target function's own comments before adding a new test that changes stack layout/ordering in the same file.
- **A pipeline into `head -N` can SIGPIPE-kill the upstream script early** under `set -o pipefail` (bit twice this session: once with `timeout N cmd | head -N` leaving a stale background process running for hours after the harness reported the task "complete", once with `xce-build.sh ... | head -3` silently truncating a build before it finished writing its wheel). If you need to sanity-check just the first few lines of a long-running command's output, redirect to a file and `head` the file afterward instead of piping directly into `head`.
- **`sim_loop.c` (and its `*_isolation`-audited siblings) genuinely has zero line-budget headroom** — its own file header says so explicitly. When a fix needs new logic there, split a new sibling file (bug-1567's `sim_picker.c`, following the exact precedent chain `sim_loop.c`→`sim_record.c`→`repl.c`→`repl_break.c`/`repl_info.c`→`repl_params.c`) rather than growing it even slightly. This requires wiring the new file into: `scheduler_srcs` AND `sched_sim_isolation_lib` (both `src/scheduler/meson.build`), a new `XCE_SIM_*_PATH` macro (`testsuite/meson.build`) consumed by `test_sim_isolation.c`'s 3 per-file text-scan functions, AND every white-box test TU that `#include`s `sim_loop.c` directly (9 files this session — grep `#include "../src/sim_loop.c"` in `testsuite/` to find them all) needs the new file `#include`d too or the link fails on the new extern. Mechanical but real work — budget for it.
- **`controller_unit_test`'s intermittent SIGSEGV (bug-1553, ~1-in-3 full runs) is a known pre-existing flake**, not a new regression — reproduce standalone (`./src/build.dbg/controller/controller_unit_test`, no `meson test` wrapper) a few times before suspecting your own change; `meson test`'s `MALLOC_PERTURB_` randomization per invocation is what makes it flaky, not deterministic per-build.
- **bug-1568 (FIXED) had TWO independent, unrelated root causes stacked on the same symptom** ("info worker" showing 0.00 for real resource values) — worth remembering the general pattern: (1) `xce_core_push_events()`'s `WORKERS_PUSH` case called `sched_add_worker()` (a plain `key_add()`, add-only) unconditionally on every push; a worker connecting sends multiple pushes in quick succession, so any push after the first for an already-registered name silently -EEXIST-discarded its resource values — fixed via `core_upsert_worker()`, mirroring the existing `core_upsert_queue()` add-or-update precedent (`core_events.c`). (2) Separately, `repl_info.c`'s display code compared `xce_val_t.which` against `XCE_VAL_NUMERIC` (`xce/types.h` `#define`, value 1 — a *resource-definition* value-type constant, `xce_resource_t.valtype`'s own enum) instead of `Value_u_fvalue` (`controller.capnp.h`-generated, value 5 — the real tag every producer in this codebase actually sets on `xce_val_t.which`) — always false, so it displayed 0.00 regardless of whether (1) was fixed. **Lesson**: when value-level TRACE at every pipeline hop (push → apply → snapshot save → snapshot load/decode) all show the correct number but the final displayed/returned value is still wrong, the bug is one hop further than "the pipeline" — trace the DISPLAY/formatting code too, don't stop at the data layer. And don't assume two enum families with parallel-looking names (`XCE_VAL_*` vs `Value_u_*`) are interchangeable just because both appear near `xce_val_t`/`.which`/`.valtype` — check which one a given producer actually sets before trusting a comparison against it.
- **`IFS=$'\t' read` (or any single-tab IFS) silently COLLAPSES adjacent/empty fields, just like the default whitespace IFS does** -- bash classifies tab (along with space/newline) as "IFS whitespace" for splitting purposes regardless of what IFS is actually set to, so `a\tb\t\tc\td` (an empty 3rd field) reads back as 3 fields (`a`,`b`,`c\td` collapsed) not 4, silently shifting every field after the gap. Found live building resource-value-verification task 7.1's `aggregate_verdict()` (`run-resource-verify.sh`) -- its own required synthetic self-test caught it immediately (an UNREACHABLE-layer record's empty `*_observed` field shifted every later field by one, corrupting the apiserver classification). Every OTHER tab-delimited TSV already in that file happens to never carry a genuinely-empty field, which is why this never surfaced before. Fix: when a delimited record format can legitimately have an empty field, use a non-whitespace-class separator (e.g. ASCII Unit Separator `$'\x1f'`) instead of tab -- `IFS=$'\x1f' read` does NOT collapse/strip, tab-like or space-like IFS always does.
- **This Bash tool aborts a multi-line script at the first command returning nonzero**, even with `; true`/`|| true` on that exact line — observed repeatedly with `pkill` matching zero processes (a completely normal, expected outcome) silently killing every following line in the same call. Don't chain a possibly-zero-match `pkill`/similar with follow-on commands in one Bash call — verify via `pgrep` first (it's just as prone to this, so treat ITS nonzero "no match" the same way) or issue the pkill as its own standalone call and check the result before proceeding.

- **apiserver's `main.go` only calls `RegisterToBackend` when `BackendYamlConfig.Address != ""` — checked BEFORE `Peers` is ever consulted** (`cmd/main.go:58`). Setting `app.yaml`'s `backend.peers` to a fully-populated HA candidate list while leaving `backend.address`/`backend.port` blank (reasoning "Peers takes priority inside RegisterToBackend anyway, per `backend/config.go`'s own doc comment") is NOT sufficient — the apiserver silently boots in permanent "stub mode" (`logger.Info("Backend not configured")`) and never even attempts a connection, `peers` or not. Caught live while building `xce-deploy-ha`'s new apiserver step (resource-value-verification task 3.1): always populate `address`/`port` too (e.g. the first `--peers` candidate's host:port) even when also setting `peers` — belt-and-suspenders, not redundant, because the two fields gate two different things (whether to connect at all, vs. which candidate(s) to try).
- **resource-value-verification task 4.1's scheduler trace-log exact format** (`xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case, `src/scheduler/src/core_events.c`, right after `zarray_put(&(w->resvals), res->idx, rv->val)`): `TRACE("resval applied host=%s resource=%s value=%f", ...)` for numeric (`Value_u_fvalue`) resources, or `value=%s` for `Value_u_svalue` ones — live-confirmed via `xce-deploy-ha` + `xce_test` worker-context `join`/`resupdate -n r15m -v 12345.5` producing exactly `resval applied host=task41-worker resource=r15m value=12345.500000` in `scheduler.log` at `-vvv`. Task 6.3 (scheduler-side log-tail check) MUST grep for `resval applied host=<H> resource=<R> value=<V>` — labeled `key=value` fields, no other fixed text to depend on. `%f` on a normal resource value produces trailing zeros (e.g. `12345.500000`), not the input's original decimal precision — task 6.3's comparison should parse-and-compare as float, not do a literal string match against the injected value string.
- **bug-1572 (NOT FIXED, pre-existing, discovered live task 4.1 round-2): `xce_scheduler` double-frees on a worker's SECOND periodic resource report for the same resource(s), silently dying with no ASAN output** (backgrounded via `setsid nohup ... >/dev/null 2>&1`, so the crash report is lost -- only visible as scheduler.log stopping mid-stream + `handle_scheduler_leave` in the controller log ~1s later). Root cause: `core_events.c`'s `XCE_CORE_EVENT_RESVALS_DELTA` case does `zarray_put(&(w->resvals), res->idx, rv->val)` with NO `MOVE_PTR` (unlike its sibling `core_apply_config_resvals()`, which does `MOVE_PTR(val, nv->val)` first) -- the same `xce_val_t*` stays aliased in the decoded request. `driver.c`'s `release_translated_payload()` then unconditionally frees that request's resvals list even when `applied=1` (its `XCE_SCHED_OP_UPDATE_RESVALS` case has no `if (!applied)` guard, unlike the sibling `XCE_SCHED_OP_UPDATE_CONFIG` case which explicitly skips the free when applied). Net effect: `w->resvals[idx]` is left dangling immediately after every successful RESVALS_DELTA batch; the SECOND update to the same index triggers zarray's killer to free the already-freed pointer. Confirms identically on round-1's code (unrelated to the enum-branch bug fixed in that round) -- a live smoke test that only runs one resupdate quickly after deploy will NOT hit this; anything that waits past one real worker's report interval (xcecd_virtual reports ~every 15s) will. When live-testing anything touching RESVALS_DELTA, either run fast (single update, no wait) or expect this crash and don't misattribute it to whatever fix is actually under test.
- **`src/frontend/Makefile`'s `build-server` target is missing CGO_CFLAGS/CGO_LDFLAGS (bug-1597, NOT FIXED)**: unlike `test-backend` (bug-1558) and `apiserver/meson.build`, `build-server`'s recipe is a bare `go build` with no `-I`/`-L` for `client.h`/`controller.capnp.h`/`libxce.a`, so `nix develop ../ --command make build-server` fails with `fatal error: client.h: No such file or directory` regardless of source changes. Confirmed via git-stash A/B that this reproduces identically with or without an unrelated diff, so don't misattribute it to whatever change is under test. Workaround to actually verify compilation: reconstruct CGO_CFLAGS/CGO_LDFLAGS the same way `test-backend` does (pkg-config for librtti/c-capnproto/openssl/tinylog + `-I$BUILD_DIR/lib`/`-L$BUILD_DIR/lib -lxce`, `BUILD_DIR` defaulting to `src/xce.build`) and run `CGO_ENABLED=1 CGO_CFLAGS=... CGO_LDFLAGS=... go build -o /tmp/apiserver ./cmd/main.go` directly inside `nix develop ../` — see bug-1597 for the exact one-liner.
- **`test.c`'s `set_resval()` leaves `xce_resval_t.name` (and `.val->which`) uninitialized for the `Value_u_svalue` fallback path** (`-n type`/`-n model`/explicit `-t string` on an unknown name) — its switch statement has no `case Value_u_svalue:` arm, and `vals[]` is a non-zeroed stack array reused across REPL invocations, so `.name` can be garbage, not NULL. Pre-existing (predates resource-value-verification's task 2.1/2.2), explicitly flagged as out-of-scope-to-fix in both tasks' code comments (Requirement 1.4's "preserve byte-for-byte" constraint). When building task 2.2's per-name verification loop on top of this, the safe pattern was to track the submitted name in a separate parallel array (`subnames[]`) captured directly from the getopt loop's own `resname` variable (always valid — that pointer is deliberately leaked/never freed, so it stays alive for the whole REPL call) rather than trusting `vals[i].name`.

- **Trap-based cleanup that must survive an async signal (SIGTERM/SIGINT) should combine multiple `pkill -9 -f` calls into ONE alternation pattern, not several sequential invocations, plus a retry-until-verified (`pgrep`) loop** (bug-1575, resource-value-verification task 5.1's `run-resource-verify.sh`). Four separate sequential `pkill -9 -f -- "pattern"` calls inside `cleanup()` worked reliably every time when reached via normal control flow (a direct `fail()`/`exit` call), but non-deterministically left 1-2 of the four target processes running when `cleanup()` instead ran from the EXIT trap fired by an externally-delivered SIGTERM translated via a separate `trap 'exit N' TERM` handler -- the exact failure point (which of the 4 pkill calls was "last to complete") varied run to run, consistent with a timing race in the trap-handler code path rather than a pattern-matching bug (each pattern was independently verified correct). Fix: one combined `pkill -9 -f -- "pat1|pat2|pat3|pat4"` call (fewer fork/exec round trips through the trap handler) wrapped in a loop that re-checks via `pgrep -f` and retries (up to ~2s total) until genuinely clean. Verified reliable across 5+ repeated live SIGTERM-mid-deployment interrupts after the fix (zero leftover processes every time), vs. non-deterministic leftovers before it. General lesson: don't trust a cleanup routine's reliability from its NORMAL-completion path alone -- explicitly live-test the signal-interrupted path too, since bash's async-signal-trap execution context can behave differently from the same code reached via ordinary top-to-bottom flow or a plain function call/exit.
- **`pgrep -f`/`pkill -f` run interactively via this Bash tool can self-match (and `pkill` can self-KILL) the invoking Bash-tool call's own wrapper process** (bug-1576, discovered repeatedly while diagnosing bug-1575 above -- cost significant back-and-forth before recognized). This harness wraps every Bash call as roughly `bash -c "... && eval '<submitted script text>' ..."`, so the wrapper's own `/proc/<pid>/cmdline` contains the FULL LITERAL TEXT of whatever was submitted -- including any search pattern typed directly into that same script (unavoidable, since testing a pgrep/pkill pattern means typing it). A `pgrep -f -- "sometext"` that also appears as literal text elsewhere in the SAME submitted command block will match the wrapper too; `pkill -9 -f` in the same situation can SIGKILL the current Bash call's own shell mid-script, truncating all remaining output with no distinguishing error beyond a generic nonzero/early-stop. Symptoms seen: a match count of "2" when only 1 real process existed; a multi-line diagnostic script silently stopping after the pkill line with no further output. Mitigation: when checking process state from an interactive Bash call, scan `/proc/[0-9]*/cmdline` directly and eyeball results for the wrapper's own distinctive `bash -c "source .../shell-snapshots/... && eval '...'"` shape rather than trusting a bare match count; never assume a lone pgrep/pkill match is the real target without checking its `ps -o cmd` first. This does NOT affect a target script's OWN internal `pkill -f` calls (e.g. inside a cleanup() function) since that script's own argv is just its invocation line (interpreter + path + args), not its source text. This ALSO bites a from-scratch verification watcher, not just a diagnostic pkill: any background polling loop launched via the interactive Bash tool with the target pattern typed inline in its own command text self-matches too (found live, task 6.1) -- write the pattern into a script FILE and launch that file instead, or check `/proc/<pid>/exe` (the resolved binary path) rather than grepping `/proc/<pid>/cmdline` text at all.
- **xce_test's worker-context injection recipe** (resource-value-verification task 6.1, cross-checked against `xce-test-ha/run-ha-soak.sh`'s extensive prior use of the identical pattern): `printf 'set masters %s\nset myname %s\nworker\njoin\nresupdate -n <name> -v <value> [...]\nquit\n' "$PEERS" "$WORKER_HOST" | xtest`. `set myname <host>` is what tags every subsequent `resupdate`'d value's `.host` field (`test.c`: `config.from = get_setting("myname")`; `set_resval()` sets `vals[idx].host = get_setting("myname")`) -- Requirement 2.2's "tag every resupdate-injected value with the joined worker's own identity" is satisfied entirely by this one setting, no separate flag needed. `resupdate_report_one()`'s output lines are exact whole-line strings (`<name>: accepted` / `<name>: rejected: unknown resource` / `<name>: rejected: value mismatch`) -- parse with `grep -Fxq` (fixed-string, whole-line), not a substring/regex grep, since a short resource name (`it`, `io`, `ls`...) can appear as a literal substring of an unrelated line otherwise.
- **A synthetic worker's host-attrs resources (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks`) get RESET to baseline the moment it disconnects -- this is correct, intentional production behavior, not a bug, but it bites any test tool that queries the controller in a SEPARATE `xce_test` process/session after the injecting session's own `quit`** (resource-value-verification task 6.2, bug-1579, not-a-bug/documented). `add_hostattrs_resvals()` (`src/controller/worker.c:260-287`) is the fixed N_HOST_ATTRS list; `mark_worker_unavail()` -> `apply_worker_dead_mutation()` (`worker.c:704-716`) calls `reset_hostattrs_unknown()` then re-applies `add_hostattrs_resvals()` on every worker disconnect, by design (a dead worker shouldn't keep advertising stale capacity). `run-resource-verify.sh`'s `inject_resources()` (task 6.1) runs `join;resupdate;quit` as ONE `xce_test` process -- the `quit` disconnects the synthetic worker before ANY later per-layer check (a separate `xce_test` process) can observe it. A live controller-check run against the task's own DEFAULT_RESOURCE_ARGS set showed 8/9 MATCH + 1 false MISMATCH (`cpuf`, the only host-attrs-colliding name in that default set) -- re-running with the same 8 non-`cpuf` resources gave a clean 8/8 MATCH, confirming the check logic itself is correct and this is purely a worker-disconnect-timing artifact. Only the CONTROLLER-layer check (a live `resval` query) is exposed to this -- the scheduler/apiserver checks (tasks 6.3/6.4) are trace-log-tail based (read what was logged at apply-time, immune to later disconnect). When choosing/validating a resource name for any live check against this script, cross-check it isn't in `worker.c`'s `N_HOST_ATTRS` list first, or expect a query run in a later, separate `xce_test` session to legitimately read back 0/default. **UPDATE 2026-08-18 (task 6.2 remediation round 2, reviewer-mandated fix): the default-set instance is now FIXED.** `DEFAULT_RESOURCE_ARGS` swapped `cpuf=9.5` for `r1m=9.5` (a periodic/dynamic load-index resource per `src/worker/src/resource_sync.c`, confirmed pre-existing/"already-supported" via `resval_typeresolve.c`'s `resinfo[]` table, and NOT in `worker.c`'s `N_HOST_ATTRS` list) -- live-validated clean 9/9 MATCH on the default (no-args) set post-fix, no regression on `mem=5`-only or mixed accept/reject scenarios. Option (b) (keep the synthetic worker's session connected across all per-layer checks instead of `quit`ting) was investigated live in this same round and confirmed NOT a simple call-reordering fix: `xce_test`'s REPL has no command path from the `worker` context back to the `client` context without first calling `leave_ctx()` (via `..`/`cmd_up`), which performs the exact same disconnect `quit` does -- making option (b) work would require either a C-level `xce_test` change or a materially riskier bash coprocess restructuring, both out of scope. This class of issue (a host-attrs resource name queried after synthetic-worker disconnect) remains a live risk for any FUTURE task or user-supplied custom `resource=value` argument naming `type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` -- see buglog.json bug-1579 for the full analysis.
- **bug-1577 (FIXED, task 6.1): `xce-deploy-ha`'s `deploy-ha.sh` silently starts a REAL worker daemon (`xcecd_virtual`) whenever one happens to be built alongside the controller** (its own pre-existing opt-in-by-presence convention, predates resource-value-verification) -- a plain `/xce-build all` leaves `xcecd_virtual` present, so `xce-test-resources`'s own deployment picked one up unconditionally, contradicting Requirement 2.1 ("no real worker daemon process required") and risking bug-1572 (scheduler crash on a resource's 2nd periodic report) on a long-enough run. Fix, without touching deploy-ha.sh (out of that task's boundary): the generated `start-worker.sh` reads `${XCE_WORKER_BIN:-<real path>}` from its OWN inherited environment at run time -- export `XCE_WORKER_BIN=/bin/true` before invoking the generated `start-all.sh`, so the worker step exits immediately with zero persistent process. General lesson: when a skill's design deliberately avoids starting some component, verify what a shared deploy helper ACTUALLY launches by inspecting real running processes, don't assume your own new code is the only thing determining what ends up running.
- **bug-1578 (FIXED 2026-08-18): controller SIGSEGVs on worker disconnect if a scheduler ever connected AND that worker's resvals were pushed at least once.** Fix applied: `sched_worker_change()`'s dead `#if 0 ... #endif` block replaced with `w->host.nresvals_to_sched = 0; w->host.resvals_to_sched = NULL;` before `xlist_fini(&resvalist);` -- exactly mirrors `apiserver_worker_change()`. Verified two ways: (1) `xce-build.sh controller` clean build, 1102/1104 tests pass (2 failures are pre-existing bug-1561 overlayfs/fdatasync, unrelated); (2) live repro via `run-resource-verify.sh --keep-running` -- deployed full stack, joined synthetic worker, ran 9 `resupdate`s, disconnected (`quit`), confirmed the exact crash-trigger log sequence fired (`handle_worker_leave` -> `sched_worker_change` pushing resvals to scheduler twice) and the controller stayed alive ~2 minutes past disconnect with zero SEGV/ASAN/core-file (previously died within 1-10s every time). **UPDATE 2026-08-18 (follow-up task) -- the "no testable-static-function harness pattern exists" claim below was wrong; corrected so it isn't trusted stale in a future session.** Added `src/controller/testsuite/test_sched_worker_change.c` (new standalone binary `sched_worker_change_unit_test`, wired into meson.build mirroring `sched_push_unit_test`'s real-`sched.c`+`dataex.c`+`egress.c` linking strategy -- the exact same "drive a static hook function through the real installed `hooksys` via `xce_run_hooks()`" pattern `test_job_array_sched_push.c` already proved for the JOB hook works identically for the WORKER hook: `on_scheduler_join()` installs `sched_worker_hook_fun`, then `xce_run_hooks(&c.hooksys, XCE_CONTROLLER_HOOK_WORKER, HOOK_AFTER_EVENT, HOOK_DETAIL_WORKER_DEAD, &c, w)` reaches `sched_worker_change()` exactly as `handle_worker_leave()` does in production). Test 1 asserts `w->host.resvals_to_sched==NULL && nresvals_to_sched==0` after the hook fires. Test 2 is the stronger ASAN-strength proof: it calls the REAL production encoder `encode_Worker()` (`controller.capnp.c`, linked for free via `libxce_dep` -- no need to link `persist.c` at all, since `persist.c`'s own `capnp_encode_worker()` is just a 20-line pass-through to it) directly on the worker right after the hook fires, exactly what `persist_worker_hook_fun` does on disconnect in production. RED/GREEN-verified live: temporarily re-wrapping the fix in the original dead `#if 0 ... #endif` reproduced a genuine ASAN heap-use-after-free (`encode_NameValue_list`, `controller.capnp.c:3408`, freed by `xlist_fini<-sched_worker_change<-sched_worker_dead<-sched_worker_hook_fun`) -- then the approved fix was restored (confirmed byte-identical via `git diff` to the pre-revert state) and both tests went GREEN. Gotcha hit along the way: `makeResource()`/`makeString()` (`resobj.c`/`xstring.c`) need the global xstring intern table initialized first (`xstring_table_init(N)`, once, before first use) or `makeString()` returns an object whose `->name` is NULL, and the very next `key_add()`/`hash_string()` call SIGSEGVs on the NULL string -- easy to miss in a fresh standalone test binary since no other linked test file has already called it process-wide (existing precedent: `test_worker_join.c`'s own `RUN_TEST` call site comments this exact requirement). Original root-cause notes retained below. `sched_worker_change()` (`src/controller/sched.c`) aliases `w->host.resvals_to_sched`/`nresvals_to_sched` directly to a local `xlist_t`'s backing array, pushes it, then `xlist_fini()`s that array WITHOUT resetting the alias first -- the correct reset lives in the SAME function but is wrapped in a dead `#if 0 /*FIXME*/ ... #endif` (commit acfbec875, 2026-03-03 -- predates resource-value-verification by ~5 months; a later commit e5fde7b5 "fixed" this but edited code INSIDE the already-dead block, so its fix silently never ran). Leaves `w->host.resvals_to_sched` dangling for the worker's whole remaining lifetime. On disconnect, `handle_worker_leave -> mark_worker_unavail -> xce_run_hooks -> persist_worker_hook_fun -> capnp_encode_worker -> encode_Worker -> encode_NameValue_list -> encode_NameValue` dereferences the freed array -> SIGSEGV (si_addr=NULL). Only reachable once a scheduler has connected at least once (`on_scheduler_join()` registers `sched_worker_hook_fun` on a one-shot `static hook_installed` gate) -- a solo controller or controller+apiserver-only never triggers it, which is why the original A/B isolation matrix (lone controller alive 90s+, full stack + xce_test never run alive 100s+) didn't catch it: the missing variable wasn't "was xce_test run" but "did a scheduler ever join AND did that worker's resvals get pushed." Sibling `apiserver_worker_change()` (apiserver.c) does the reset correctly (not `#if 0`-wrapped) -- mirror that exact pattern for the fix, do NOT just uncomment the dead `free_name_val_list()` call (it already frees the array AND nulls the field itself; the very next `xlist_fini()` line would then double-free the same pointer -- almost certainly why the original author reached for `#if 0` instead of deleting it). Fully outside resource-value-verification's task boundaries (sched.c, untouched by tasks 2.1/2.2/6.1) -- tasks 2.1/2.2 are innocent, no re-review needed. See buglog.json bug-1578 for the full backtrace and fix diff.

- **`do_update_resvals()` (`src/controller/opupd.c`) fans out the FULL, UNFILTERED submitted resource list to the scheduler AND apiserver hooks BEFORE (and still, after) `update_resval_list()` does its own per-item validity check — the controller's own `w->host.resvals` write is filtered (rejects unregistered names via `key_find(c->resmgr,...)`), but the HOOK_AFTER_EVENT payload `sched_resource_hook_fun`/`apiserver_resource_hook_fun` (`sched.c`/`apiserver.c`) act on is the SAME raw list regardless.** Found live during resource-value-verification task 8.1 (bug-1583, NOT fixed): injecting one deliberately-unregistered resource name showed the scheduler's `core_events.c` independently re-validates against its own resource table and discards+WARNs on an unknown name, but apiserver's `OnResourceLocationUpdated` (`handler.go`) has zero such validation and unconditionally `store.ResourceLocations.Put()`s it — a genuine 3-way inconsistency (controller rejects, scheduler rejects, apiserver silently stores forever). General lesson: when a controller hook fans a raw request payload out to multiple downstream consumers, don't assume upstream validation (even a REJECT logged elsewhere in the same request) actually filtered what those consumers received — check each consumer's own hook function for whether it re-validates or blindly trusts the payload.
- **When a live crash won't reproduce under gdb (even a direct `gdb --args` child launch, which sidesteps this sandbox's `ptrace_scope=1` attach restriction), suspect gdb's ptrace overhead itself is masking a timing-sensitive bug rather than concluding "not reproducible."** bug-1578: a 100%-reproducible (bare launch) controller SIGSEGV did NOT reproduce within 20s under gdb for either the ASAN or debug(-O0) build, for both a plain `run` and a `--batch` launch. Fix: skip gdb entirely -- compile a tiny `LD_PRELOAD` shared library (constructor installs a `sigaction` handler with `SA_ONSTACK`+a pre-allocated altstack for SIGSEGV/SIGABRT/SIGBUS/SIGILL/SIGFPE that calls `backtrace()`+`backtrace_symbols_fd()` to stderr, then `signal(sig, SIG_DFL); raise(sig)` to preserve normal termination semantics) and launch the target with `LD_PRELOAD=./crash_handler.so <normal invocation, no wrapper>`. Near-zero runtime overhead (one constructor + a couple `sigaction()` calls), so it does not perturb thread scheduling/timing the way ptrace does -- reproduced the exact same crash reliably and captured a full backtrace. Resolve the raw addresses with `addr2line -e <binary> -f -C -p <offset...>` using the `(+0xNNN)` MODULE-RELATIVE offsets `backtrace_symbols_fd()` prints (not the raw runtime addresses, which include the PIE ASLR base and won't resolve against the on-disk binary directly).

- **Testing a bash function that both populates caller-scope globals AND prints via a `log()`-style echo (task 8.2, `aggregate_verdict()`): never capture its output with `$(...)` command substitution if you also need the globals it set.** Command substitution forks a subshell in bash — any associative-array/variable writes the function makes are lost the instant the subshell exits, even though the function's exit code and printed text survive. Fix: call the function directly with plain `>` file redirection (`fn ... > "$tmpfile" 2>&1; rc=$?`), which does NOT fork a subshell, then read `$tmpfile` separately for text assertions while the just-set globals remain live in the current shell. This is exactly `.claude/skills/xce-test-resources/verdict-logic-selftest.sh`'s `run_agg()` helper.
- **This codebase's established pattern for a persistent bash-tooling regression test is `.claude/skills/xce-test-ha/soak-verdict-selftest.sh`**: name it `<subject>-selftest.sh`, `source` (not execute) the target script to get its functions in scope, use a `check DESC EXPECTED ACTUAL` string-equality helper tallying a `fails` counter, end with a `SELFTEST PASS`/`SELFTEST FAIL (N failed)` line and exit 0/1. `run-resource-verify.sh` itself is written to support this — it has a `[[ "${BASH_SOURCE[0]}" == "${0}" ]]` guard at the bottom specifically so sourcing it never runs `main()`/starts a live deployment. Reused for `verdict-logic-selftest.sh` (task 8.2).

- **Sandbox capability update (2026-08-19): `sudo` now works passwordless here** (`sudo -n true` succeeds) — this unblocks anything previously gated on "this session/skill cannot sudo" (e.g. real `xcecd`, `worker/src/main.c`'s `check_user()`). Still true, and load-bearing for anything assuming a real production host: this sandbox has **no systemd, no dbus, no `/run` at all** (`ls /run` → No such file or directory; PID 1 is `s6-svscan`; no `dbus-daemon`/`systemctl` even under `nix develop`). Root access and a real init/dbus stack are independent axes — don't assume sudo working means systemd-dependent things (job dispatch via `org.freedesktop.systemd1.Manager.StartTransientUnit`, anything needing `/run/dbus/system_bus_socket`) will also work. `/run/systemd/system` existing is the standard `sd_booted(3)`-style signal for "a real systemd instance is running as this host's manager" — used in `src/tools/deploy-ha.sh` to auto-pick `xcecd` (real) vs `xcecd_virtual` per host (`XCE_REAL_WORKER=0|1` overrides).
- **A root-owned file created by a daemon under `sudo` is not automatically readable by the invoking non-root user** — confirmed live (bug-1592): real `xcecd`'s main log file comes out `root:root` mode `640` even though the SAME binary explicitly `chmod(..., 0666)`s its domain socket and its `xcecd_worker.log` sibling. Don't assume "the daemon already chmod's some of its output files permissive" generalizes to all of them — check each file's actual mode after a live run before writing a verification command that reads it as the invoking user; prefix with `sudo` where it doesn't.

- **xce_resource_t (xce/types.h) has a `flags` bitmask (XCE_RESF_BUILTIN/DYNAMIC/GLOBAL/SHARED/EXTERNAL/RELEASE/DEFAULT_HOST/CONSUMABLE/LIMIT) distinguishing a genuinely cluster-wide resource (GLOBAL/SHARED, stored ONCE in `core->sharedInst`, a zarray_t indexed by the resource's `idx` -- the same pool RESVALS_FULL/RESVALS_DELTA event handling and snapshot save/load already treat as authoritative) from a host-based/per-worker-distributed resource (stored per-worker in `w->resvals`/`w->resInUse`).** Before this session, `repl_info_resource_one()` (scheduler REPL) conflated the two -- aggregated total/available across every worker's resvals for EVERY resource regardless of kind, a fabricated number for a host-based resource (cpus/mem) and redundant with `info worker`'s own correct per-host display. When adding any new resource-value display/log code, check `r->flags & (XCE_RESF_GLOBAL|XCE_RESF_SHARED)` first to decide which pool (`core->sharedInst` vs per-worker `resvals`) is the correct source of truth -- see bug-1593.
- **`repl_info.c` (scheduler REPL state-inspection dumps) was already OVER this project's 800-line cap (883 lines) before this session even started** -- accumulated from three prior sessions' bug fixes (bug-1587/1590/1591) each adding a little without ever triggering a new sibling-file split. A task-12.5 rewrite of the resource-display functions brought it down to 856, still over. Flag this to the user rather than silently ignoring it or unilaterally doing a 4th-generation file split (repl.c -> repl_break.c/repl_info.c -> repl_params.c precedent) without being asked -- a split touches meson.build, test_sim_isolation.c's per-file scan macros, AND every white-box test TU's #include list (real, mechanical work, not worth taking on speculatively).
- **A `repl_split_tokens()`-style whitespace tokenizer with zero quote-awareness silently breaks any downstream parser that requires a whole-string-anchored regex match** (e.g. `parse_jobid()`'s `pmatch[0].rm_so==0 && pmatch[0].rm_eo==len` check) when a user quotes an argument out of habit (common when copy-pasting a value from another command's own output). The numeric fallback path (`strtoull()`) then silently parses the leading quote character as "not a digit" and returns 0 rather than erroring -- producing a confusing wrong-answer, not a clear parse failure. Worth checking for in any REPL/CLI tokenizer this codebase adds. See bug-1596.

- **A pointer returned by `makeString()` (rtti-managed, `lib/xstring.c`) must always be released via `dereference()`, never `FREEUP()`/plain `free()`** — `makeInstance()`/`objAlloc()` returns an offset into a combined header+data block, not the block's own `malloc()`-returned base address, and the string is refcounted (freed only when the refcount hits zero), not directly owned. bug-1600 (2026-08-19, production, not a test fixture — see also bug-1503's earlier test-fixture instance of the identical mismatch): `lib/group.c`'s `makeGroup()` "existing group" update branch (hit every time a config-push re-applies a group name that already has members, e.g. any retried/duplicate config push — no thread concurrency required) called `FREEUP(m->u.name)` on a `makeString()`-allocated field, while the SAME FILE's `group_killer()` destructor released the identical field correctly via `dereference(m->u.name)`. This caused the real, reproducible (8 confirmed core dumps) `xce_scheduler` `free(): invalid pointer` SIGABRT crash during controller-failover config-push retries. **When adding or auditing any free/release path for a field populated by `makeString()`, grep the same file/struct for an existing destructor and match its release call — don't assume `FREEUP()` is safe just because it's used for adjacent plain-`calloc`'d fields in the same struct.**
- **`lib/group.c`'s `makeGroup()` "existing group" update path takes only `RTTI_LOCK_READ` (a genuine shared `pthread_rwlock_rdlock`, confirmed via `kvmgr.c`) while mutating shared group member-list state, and the scheduler's entire config-push handler chain (`sched_init_config()` → `plugin_config_update()` → ... → `makeGroup()`) holds no serializing lock at all — while `scheduler_req_handler()` genuinely runs on whichever of the scheduler's 4 vcpu worker pthreads picked up that vtask (`vtaskmgr_init(..., 4)`).** `xce_ctx_reconnect()` reuses the same `ctx->ep` struct and `endpoint_init()` unconditionally resets `ep->busy=0`, so a reconnect racing with a still-running config-push handler can let a second, concurrently-dispatched config-push run `makeGroup()` on the same group object at the same time as the first. This is a REAL latent concurrency bug distinct from bug-1600's allocator mismatch (which alone was sufficient to explain the observed crash and was the one fixed) — flagged as a follow-up, not yet fixed; would need a write-lock scope redesign since `init_group_by_config()`'s callees (`get_group()`) take their own internal read lock, so naively holding a write lock across the whole function would self-deadlock on the non-recursive rwlock.

- **`apply_committed_entry()`'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON dispatch case (`persist.c`) never stamped `start_time`/`finish_time`/`run_time`** — it only ever called the generic, state-only `apply_job_update_transition(j, state)` (2026-08-19, `xcecli joblist` showed `StartTime`=0 for every RUN/DONE/EXIT job, `FinishTime`=0 for DONE jobs). Fixed additively (user's explicit steering: "don't change the logic, just add extra information") by keying a `time(NULL)` stamp on `wal_type` right after `apply_job_update_transition()` succeeds, still under the same `lock(j, RTTI_LOCK_WRITE)` — `WAL_JOB_START` → `start_time`, `WAL_JOB_FINISH` → `finish_time` + `run_time = finish_time - start_time`. No change to `apply_job_update_transition()`'s signature or the dispatch structure. Mirrors what the richer (currently-unreached-on-this-path) `apply_job_start_transition()`/`apply_job_finish_transition()` cores in `job.c` already do for their own callers.
- **`test_config_repl.c`'s `test_conf_trim_keeps_only_n_versions`/`test_conf_get_saved_tag_no_config` failing with the exact same Expected/Was values plus a burst of `persist: flusher fdatasync FAILED (errno=5 Input/output error)` lines is bug-1561, a confirmed overlayfs-sandbox `fdatasync` artifact — 6 occurrences now across unrelated sessions/diffs, never once traced to a real code change.** If a `/xce-build controller` run shows exactly these 2 failures with that exact error pattern, it's environment noise, not something your change caused — don't spend time re-investigating, just note it and move on (though still bump bug-1561's `occurrences`/`last_seen` per buglog protocol).

## Do-Not-Repeat
- Don't run `clang-format -i` on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via `git checkout HEAD -- <file>` on a file with other uncommitted work destroys that work irrecoverably.
- Before citing a bug as "still open" or a spec as "blocked," cross-check `.wolf/buglog.json`'s actual fix status and the spec's own tasks.md/commit history — blocked_on notes and prior narrative claims (including this file's own past entries) can be stale. **Self-caught recurrence (2026-08-19)**: while reconciling OTHER stale doc references, wrote a new tasks.md status line asserting "tasks 2.1-2.6 still carry no `_Verified:` annotations" by trusting a 2026-08-07 validation finding's own wording verbatim, without re-checking the file — they'd actually been backfilled the very next day (commit `d7b49690`, 2026-08-08). A dispatched verification agent caught it via `git log`/direct read. Lesson holds even for a claim you are ABOUT to write, not just one you're citing from memory: grep/read the actual current state of the specific lines before asserting their status, every time, no exceptions for "this one's probably still true."
- Before trusting an "all tests pass, GO" claim on a stale/uncommitted tree, re-run the suite yourself — a buglog "FIXED" note can describe the *intended* fix while the code on disk is mid-revert from RED-phase construction.
- A generic retry/failover classifier that judges retriable-vs-terminal by a raw return value's magnitude (comparing against known errno values) is unsafe unless every wrapped call site shares the same success convention — a call that legitimately returns a variable count on success can numerically collide with an error code.
- When two independent "is it ready" signals exist server-side, a wait/convergence helper gating on only one does not imply the other is satisfied — check what the actual raced precondition depends on.
- Nix fixed-output-derivation hash mismatches from an upstream input pinning a *mutable* branch: re-locking flake.lock is a dead end once already at the latest commit; fix via a local `overrideAttrs` pinning `src` to an explicit immutable commit in the consuming repo.
- `pkill`/`pgrep` by bare process name (unscoped) can kill or match a *different* concurrent invocation's processes, including a sibling session's build/soak — always scope process matching to something unique to the current run (e.g. a workdir path).
- **Stacked/masking bugs in never-before-reached code paths: when a bug prevents a code path from EVER executing, fixing it can immediately expose 1+ MORE bugs that were always there but never triggered.** bug-1569 thread B was "no job ever gets scheduled" -- fixing cause #1 (scheduler.c's sched_init_queues() never calling picker_iface->on_queue_update() for builtin_fcfs, so its candidate registry stayed empty forever) let a job reach the allocator for the first time ever, which immediately SIGSEGV'd on cause #2 (core_events.c's core_build_job() never constructing job->schedobj, a documented-but-wrong assumption from the scheduler-core rewrite). Fixing #2 let a job reach genuine dispatch+resource-charge accounting for the first time ever, which immediately SIGSEGV'd on cause #3 (resource.c's put_resvals() had an off-by-one, `idxs[i] <= 0` treating resource idx 0 -- "slots" -- as an invalid sentinel). Lesson: after fixing a "nothing ever happens" bug, don't declare victory on the first crash/error that surfaces -- it may be the SECOND (or third) bug the first one was masking, not a regression from your own fix. Verify via gdb `bt full` on the actual live crash, not guesswork from reading code alone.
- **`xce_scheduler -i`/`--sim` (the interactive REPL) starts PAUSED at the initial snapshot** and never advances the simulated clock past a recorded `ReplayEvent`/`ReplaySession` until you send an explicit `step [N]` or `continue`. Piping `"info worker <name>\nquit"` with no `step`/`continue` first only ever shows the pre-replay baseline state — this cost a large chunk of bug-1569 investigation, wrongly appearing to show a second "periodic resources still 0.00 after the host-mismatch fix" bug when the fix was already fully correct and sufficient; `step N` (N large enough to pass the relevant event, confirm via `xceadmin scheduler dump <file>` first if unsure of event ordering) revealed the true, correct values. Before concluding a replay-derived state is wrong, always `step`/`continue` first.
- **A dead `#if 0 /*FIXME*/ ... #endif` block is a signal the disabled code is unsafe as written, not merely "not yet wired in" — don't just uncomment it.** bug-1578: `sched_worker_change()`'s disabled block called `free_name_val_list(&w->host.resvals_to_sched, &w->host.nresvals_to_sched)` right before `xlist_fini(&resvalist)` — but `resvalist.data` and `w->host.resvals_to_sched` are the SAME pointer (the earlier line aliased one to the other's backing array), and `free_name_val_list()` already frees that array itself; uncommenting as-is would double-free. The safe fix null-and-reset the aliased fields directly (`w->host.resvals_to_sched = NULL; w->host.nresvals_to_sched = 0;`) and let `xlist_fini()` be the sole owner of the free — the pattern already proven correct in the sibling `apiserver_worker_change()` (apiserver.c). General lesson: when re-enabling dead code, first work out WHY it was disabled (git blame + read what it actually does to shared/aliased memory), don't assume `#if 0` means "harmless, just needs a flip." Other call sites still doing `free_name_val_list(&(w->resvals_to_sched), ...)` on this same field (dataex.c:435, opget.c:128, reply.c:66, apiserver.c:51) were NOT audited for the identical aliasing hazard as part of this fix — flagged, not yet checked.
- **`xce_test`'s `!<shell-command>` escape (`preprocess_and_run()`, single call site in `main()`'s top-level REPL loop, `test.c` ~line 950-998, backed by a plain blocking `system()`) works IDENTICALLY in every REPL context, including `worker` — no `leave_ctx()`/disconnect happens, unlike moving between contexts via `..`/`cmd_up`.** This is the general pattern for "run an out-of-band check while a stateful REPL session must stay connected": bake a `!`-prefixed line into the script fed to the session (before its `quit`) that shells out to ANOTHER invocation of your own driver script in a hidden internal mode (e.g. `! bash "$0" --internal-<mode> <args>`), intercepted via a dispatch check placed BEFORE the normal arg-parsing entry point at the very bottom of the file. Used to fix bug-1579 (resource-value-verification): the controller-side resval check now runs nested inside the injecting worker session's own `!` escape instead of as a separate xce_test process launched after `quit` already disconnected the worker (which reset host-attrs resources to baseline first, an always-false MISMATCH). Gotchas: (1) the nested process must generally return exit 0 even on partial/expected failure conditions — a nonzero exit becomes the outer session's `g_last_status`/`$?`, and in non-interactive/batch mode `preprocess_and_run()` ABORTS the REPL loop before any later command (including `quit`) runs; surface real failures via a sentinel/marker file the OUTER script checks AFTER the whole session completes, not via the nested process's own exit code. (2) `system()`'s child inherits the invoking process's environment but NOT its parent bash's local (non-exported) shell variables — bake every value the nested invocation needs (paths, host lists, names) as literal, `printf '%q'`-quoted arguments directly into the `!` line's command string, don't rely on env-var passthrough through the `nix develop`/ASAN-wrapped intermediate hops. (3) since accept/reject-style bookkeeping computed by the REPL session itself is only visible as TEXT PRINTED to that session's own stdout — not accessible to a freshly-spawned nested process — have the nested step collect RAW, unfiltered data for every candidate (not just ones later known to be "accepted") and defer accept/reject filtering to the outer script, after it has parsed the whole session's captured output post-`quit`. (4) live-verified no orphan/zombie processes result even when a SIGTERM interrupts the outer driver script WHILE the nested `!`-escape's own sub-session is actively running — bash defers a pending trap until the current foreground command (the whole piped xce_test invocation, nested check included) returns, so the child process tree either completes and exits cleanly on its own or the trap fires right after; confirmed via a full `/proc/[0-9]*/cmdline` scan, not just `pgrep`.
- **Piping scripted commands into `xce_test` (the controller REPL/testsuite binary) with default `ASAN_OPTIONS` loses ALL stdout output when redirected to a file/pipe.** `xce_test` is an ASAN build; `detect_leaks=1` is the default, and this binary has small pre-existing intentional leaks (e.g. `set_resval`'s `calloc`, `cmd_resupdate`'s `strdup` for `resname` — never freed by design, REPL-lifetime scratch data). LeakSanitizer's exit-time report appears to bypass/precede the normal stdio flush when stdout is non-interactive (fully block-buffered), so a redirected-output smoke test showed ONLY the ASAN leak report and none of the actual REPL command output (join/resupdate/resval results), even though the commands genuinely executed (confirmed via the leak stack trace itself). Fix: always run `xce_test` with `ASAN_OPTIONS=detect_leaks=0` (matching the meson test harness's own invocation, which already sets this) whenever piping/redirecting its stdout for a live smoke test — e.g. `nix develop ./src --command env ASAN_OPTIONS=detect_leaks=0 xce_test < cmds.txt`.
- **A fork/subagent dispatched in a prior session that claimed "build/tests in progress, will report back" is NOT reachable in a new session — `ListAgents` will not show it, and its self-report (if logged to memory before the session ended) describes an *intent*, not a completed result.** 2026-08-19: a fork agent from the previous session was recorded as "actively writing tests... then building via xce-build skill and reporting final metrics," but the session ended before it reported. On resuming ("continue"), the scheduler test binaries in `src/xce.build` were still dated Aug 18 14:27 while the source edits were Aug 19 09:18-09:27 — proof the claimed build never actually happened in this environment. Always verify via binary/source `stat` mtime comparison (or just re-run the build) before trusting a cross-session "in progress"/"dispatched" claim, same discipline as the existing "stale/uncommitted tree GO claim" entry above but for cross-session fork continuity specifically, not just tree staleness.
- **A REPL/replay test fixture that reuses the SAME jobid for both a "real" event (job push) and a "cheap padding" event (job state change, added purely to exercise a second event type) accidentally mutates the real job's state, not just decorates the log.** bug-1595 test regression (task 12.5, 2026-08-19): `test_step_echoes_events_before_session_decisions` pushed job 6161 then sent a JOBSTATECHANGE event for the SAME jobid 6161 (PEND->RUN) purely to get a second event type echoed — this genuinely transitioned the real job to RUNNING before the session ran, so the picker's `next_job()` never offered it and the session's own PEND decision (which the test also asserted on) was never produced. The sibling test `test_event_breakpoint_pauses_repl_at_matching_event` already documents the correct pattern in its own comment (`append_jobstatechange_event() establishes: core_events.c WARN+skips an unknown jobid rather than fail, so ... below needs no real job to exist`) — it deliberately used unrelated jobids (111/222) precisely to keep the padding event inert. When adding a "just need any event of type X to be logged" filler event to a fixture that also has a real job in flight, always target an unrelated/nonexistent jobid unless the state transition itself is the thing under test.
- **`[ -n "${arr[key]+x}" ]` (associative-array key-EXISTENCE) is NOT a safe proxy for "the value is meaningful/reachable" when the producer unconditionally sets that key even on failure.** bug-1582 (resource-value-verification task 7.1, round-1 REJECTED, round-2 fixed): `main()`'s record-building loop derived `sched_ok`/`api_ok` from whether `SCHEDULER_OBSERVED[$vname]`/`APISERVER_OBSERVED[$vname]` existed — but `query_scheduler_resvals()`/`query_apiserver_resvals()` (already-approved tasks 6.3/6.4) always set that key, setting it to `""` on a genuine UNREACHABLE/timeout rather than leaving it unset. Key-existence was therefore always true, silently masking every real UNREACHABLE as a MISMATCH (comparing the injected value against an empty string). Fix: derive the "is this layer reachable" signal from the layer's own pre-classified status string (`SCHEDULER_CHECK_RESULTS[$vname]`/`APISERVER_CHECK_RESULTS[$vname]`, which DOES distinguish `UNREACHABLE:*` from `MATCH`/`MISMATCH:*`) instead of from a sibling data array's key presence. General lesson: before using `${arr[k]+x}`/`-v arr[k]` as a "was this checked / did this succeed" signal, verify the producer actually leaves the key UNSET on failure — if it sets it to an empty/placeholder value instead (a common pattern for "always populate every requested key so downstream code doesn't need existence-checks"), key-existence collapses to "was this key in the requested set" a wholly different, weaker signal than "did this succeed". Also: a synthetic test that only calls the pure aggregation function directly with hand-built records (task 8.2's original approach) cannot catch a bug living in the data-marshaling code that BUILDS those records — this class of bug needs the marshaling code itself extracted into a directly-callable function (`build_verdict_records_from_tsv()`) and exercised by its own test.

## Decision Log
- HA/consistency model follows Raft + CP strictly: majority ACK is the hard commit barrier for all state-changing writes (reject on timeout, never local-apply-and-degrade); the own-term commit rule (not a 4th "COMMITTING" state) is what gives a new leader safety.
- A controller process must NEVER self-exit/self-abort as a designed response to an operational condition (only a genuine crash or admin/supervisor kill) — an unrecoverable/fenced member logs loudly, stays permanently non-promotable, stays alive, and keeps retrying.
- Commit-first-then-reply: job/queue/host control ops validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply; no success reply before commit. Worker-reported fact entries instead follow pure Raft semantics — retained/retried indefinitely, no commit timeout.
- No client reply-wait timeout as a fix for a hung request — fail loud at the real transport/source instead of masking with a deadline backstop.
- Product is pre-release: no backward/mixed-version wire compatibility required; wire format changes may be made freely.
- `resource-value-verification` task 7.1 (2026-08-18): `aggregate_verdict()`/`build_verdict_record()` (`run-resource-verify.sh`) re-derive MATCH/MISMATCH/UNREACHABLE from raw `*_OBSERVED` values via `controller_values_match()` at aggregation time, rather than trusting each layer's already-rendered `CONTROLLER_CHECK_RESULTS`/`SCHEDULER_CHECK_RESULTS`/`APISERVER_CHECK_RESULTS` strings from tasks 6.2-6.4 -- deliberate, so the SAME classification code path is exercised by both the live-run wiring and task 8.2's synthetic-record tests (a synthetic test that only fed pre-classified strings would just be testing string formatting, not the comparison logic). A rejected resource name is modeled as its own failure category (`REJECTED`, layer=`injection`) distinct from a per-layer `MISMATCH`/`UNREACHABLE`, and always forces overall FAIL, per design.md's Error Handling ("unrecognized resource name is a run-level failure") -- this is NOT implied by Requirement 6.2/6.3 alone (those only speak to "checked layers"), it's a deliberate design-level extension. The script's bottom dispatch block is now guarded (`[[ "${BASH_SOURCE[0]}" == "${0}" ]]`) so it can be `source`d to get `aggregate_verdict()` et al. in scope with zero side effects (no `main()`/deployment triggered) -- this is the intended integration point for task 8.2's dedicated test harness.
- `resource-value-verification` task 7.1 round 2 (2026-08-18, bug-1582, round-1 REJECTED -- see Do-Not-Repeat entry for the mechanism): `main()`'s OWN record-building loop (feeding `aggregate_verdict()`, distinct from `aggregate_verdict()`/`build_verdict_record()` themselves, which were and remain correct) had the actual bug -- `sched_ok`/`api_ok` derived from `*_OBSERVED` array-key existence instead of the `*_CHECK_RESULTS` classification strings, masking every genuine scheduler/apiserver UNREACHABLE as MISMATCH. Fixed by switching to a leading-`UNREACHABLE:` check on `SCHEDULER_CHECK_RESULTS`/`APISERVER_CHECK_RESULTS` (the controller layer's `ctrl_ok` derivation from `CONTROLLER_OBSERVED` array-key existence was independently confirmed fine and left untouched -- that layer has no separate UNREACHABLE concept). Also extracted the loop into a standalone `build_verdict_records_from_tsv()` function (populates global `VERDICT_RECORDS`) specifically so a synthetic test can exercise `main()`'s actual data-marshaling code, not just `aggregate_verdict()` called with hand-built records -- the round-1 gap was that no test exercised the code that BUILDS records from the `*_OBSERVED`/`*_CHECK_RESULTS` globals, only the pure aggregation function itself. Live-reproduced by killing the scheduler process right after `confirm_deployment_up()` returns (before `inject_resources()`/`query_scheduler_resvals()` ever run) -- deterministic, no race window, and confirmed the controller/apiserver layers are unaffected by an isolated scheduler kill (each layer's check is independent).
- `resource-value-verification` spec (2026-08-17): verify scheduler/apiserver resource-value propagation via a one-line trace/debug log at each layer's point of application (`core_events.c`'s RESVALS_DELTA case; apiserver's `OnResourceLocationUpdated`), grepped live, instead of record+replay+REPL or a REST/CLI query — matches how bug-1568/1569 were actually root-caused, and is a stronger read of "the scheduler's own resource state" than a replayed reconstruction. Trade-off accepted: doesn't exercise the real query/REST path, only storage correctness. Load-bearing gotcha: `TRACE`=`tlog_debug3` needs `-vvv`; apiserver's `logger.Debug` needs `logger.yaml`'s `logging.level: debug` — both silently no-op otherwise, producing false timeouts. Also: extend `xce-deploy-ha`'s `deploy-ha.sh` (opt-in, conditional-on-binary-present, mirroring its existing worker/scheduler pattern) plus a generated `start-all.sh`, rather than building a parallel standalone deploy script — avoids re-implementing the same deploy conventions twice.