myxce/.wolf/cerebrum.md

Cerebrum

OpenWolf's learning memory. General, reusable knowledge only — bug-specific history lives in .wolf/buglog.json, not here. Do not edit manually unless correcting an error. Last updated: 2026-08-16 (condensed from a ~40k-token file; see .wolf/buglog.json for full bug history)

User Preferences

• Run soak tests with specific round counts and deploy directories for precise regression tracking.
• Prefer parallel agent runs for independent analysis work; sequential for dependent tasks.
• Keep test logs for analysis, don't skip intermediate output even if a test hangs.

Key Learnings

• xcecli Cobra rootCmd location: src/frontend/xcecli/cmd/main.go imports cmd "xcecli/router" (the import alias is cmd, but the actual package lives at xcecli/router/router.go, package-named cmd). Cobra's rootCmd is declared there as an unexported package-level var (var rootCmd = &cobra.Command{...}, router.go:21). The only exported symbol is Execute(cm *config.ConfigManager) error, which takes no version param... boundary conflict — SUPERSEDED, resolved by build-time-versioning task 4.5 (commit fb09afe) and confirmed still true as of dirty-content-fingerprint task 4.5 (2026-08-21): Execute was extended to Execute(cm *config.ConfigManager, version string) error, which itself sets rootCmd.Version = version (router.go:29-34) before rootCmd.Execute(). main.go now composes the full version string (GitCommit/BuildTime, plus task 4.5's composeVersionDisplay(GitCommit, DirtyFingerprint) strip-and-append) and passes it into cmd.Execute(configManager, version) — no further router.go changes needed for version-string composition; only main.go is the boundary now.

• Build entry point: flake.nix lives in src/, not the repo root. Always nix develop /abs/path/to/src --command bash -c '...' with an explicit path — Bash cwd isn't stable across calls, and a bare cd src && nix develop can silently short-circuit.

• Test coverage gaps: documented full-suite commands (xce-build.sh <component>, make test) each cover only a subset — e.g. a component's default meson run misses newly-added standalone test binaries unless wired into its RUN_TESTS case, and the CGO/apiserver backend suite needs make test-backend with CGO_CFLAGS/LDFLAGS reconstructed via pkg-config (package is c-capnproto, not capnp_c). A green default build is not proof of full coverage — verify every new test binary actually runs.

• Unity harness: use UnitySetTestFile(__FILE__) (not nested UNITY_BEGIN()/UNITY_END()) when multiple test-group files share one binary — nested BEGIN/END resets global counters and can mask most of the real suite exit code. Authoritative green = grep raw testlog for :FAIL: / word-boundary \bTIMEOUT\b, never trust an aggregate Fail:0 alone. Also rule out a concurrent competing build/soak against the same build dir before treating a timeout/crash as a regression.

• apiserver zap encoding: console still renders structured fields as trailing JSON: logger.yaml's console encoder is only for the message/level/time/caller prefix — any zap.Fields passed to logger.Debug/Info/... still render as a single trailing {"key": val, ...} JSON object at end of line (e.g. ... resource-location stored {"resource": "r15m", "host": "resverify-worker", "value": 1.5}), in the exact field-append order from the call site. Don't assume encoding: console means key=value text — grep/parse trailing zap log fields as a JSON tail block (index()/substr() in bash is fine, no JSON lib needed for a fixed, known field order) regardless of the configured encoding name.

• capnp/typemap conventions: every schema struct gets $C.mapname(...); for a wire-only concept with no live C struct, hand-define a small matching struct in typemap.h. A Data-typed field's generated composite codec is a silent no-op in this backend AND the generated _ptr wrapper doesn't zero-init — an unset field is written as literal uninitialized stack garbage (real crash risk). Prefer List(UInt8) + a $C.maplistcount field, which this backend does generate real code for. A nested-struct getter can return an unresolved far pointer after a segment round-trip — call capn_resolve() before reading its scalar fields (Text/Data getters resolve internally and mask this).

• C comments: never write a bare type*/ (pointer type immediately followed by /) inside a /* */ block comment — it silently closes the comment early, producing a cascade of unrelated syntax errors elsewhere. Recurred repeatedly; grep new/edited comments for [A-Za-z0-9_]\*/ before building. Same failure mode, different trigger (caught before building this time, writing test_sched_worker_change.c's file header): quoting a piece of source code that itself contains /* ... */ (e.g. describing the literal dead #if 0 /*FIXME*/ ... #endif block from sched.c) inside your own outer /* */ doc comment closes the outer comment at that inner */ too — C block comments never nest. When a doc comment needs to reference literal /* */-bearing source text, paraphrase it (e.g. "a dead #if 0 (FIXME-commented) block") instead of pasting the real comment markers.

• Thread/lock hazards on async-ified single-threaded code: a library/struct built for serialized calling (Raft state, SSL* objects, an epoll batch-dispatch loop) that later gets a "drive it on a thread pool" feature needs its own lock/refcount around the shared state — per-endpoint or per-callback serialization does not imply serialization across different endpoints. A same-thread same-batch use-after-free (one event's handler freeing something a later event in the identical batch still references) can look identical to a genuine cross-thread race.

• makeString() (lib/xstring.c) is rtti-backed, not malloc/strdup — only free it via its own module's rtti-aware teardown, or don't free it at all; a generic free()/capnp free_X() on it is heap corruption. Use strdup() for fields a generated free_X() will free directly.

• Threaded test fixtures: teardown for a fixture with real background threads must run unconditionally before any TEST_ASSERT_* — a failed assertion longjmps past later code, orphaning the thread and corrupting the next test's fixture memory.

• Live-soak validation discipline: a unit-test-green fix can still be unreachable/dead code in production — always re-run the exact live scenario end-to-end, not just the mechanism fixed. A later, differently-shaped soak failure after a fix isn't automatically a regression from that fix — check whether that failure point was ever exercised before.

• Sandbox quirks: /tmp can be overlayfs with intermittent fdatasync EIO — durability-sensitive tests need a real fs (e.g. ~/tmp on ext4). This sandbox's date may resolve to BusyBox (no %N, silently passes it through instead of erroring) — use bash's $EPOCHREALTIME/SECONDS builtins for timing instead.

• HA election timer discipline (bug-1562): every early-return branch in a function driven by a one-shot timer (ha_on_primary_ctrl_fd_rdhup's liveness-watchdog callback) must either re-arm the next timer itself or verify some OTHER live timer is actually still scheduled before skipping the re-arm — checking only elapsed-time-since-last-event (the 5.10b debounce branch) is not the same as checking an election is still in flight (the 5.6 guard's actual role/pre_vote_in_progress check). A silently-dropped re-arm freezes the node with zero CPU/log activity forever (looks identical to a clean idle state, not a crash) — diagnose via /proc/<pid>/task/*/wchan (all threads in futex_do_wait/do_epoll_wait with no fds beyond listen+epoll = dropped timer, not deadlock) before reaching for gdb (ptrace is blocked in this sandbox, yama ptrace_scope=1 + no ancestor relationship to the target).

• kiro-spec-tasks' independent task-graph sanity review is worth actually dispatching, not skipping as formality: writing scheduler-sim-repl-verbosity's design, the round-boundary "collapse active output range" logic was placed entirely in repl.c (where the round-boundary hook lives) — but a fresh review subagent (no drafting context) caught that the OTHER two triggers for ending that range (a breakpoint hit, a structural event arriving) are detected in repl_break.c, which had no interface to reach into repl.c's state. This is exactly the class of bug a same-context self-review reliably misses (the drafting mind already "knows" the round-boundary path handles it and doesn't re-derive from the other call sites) — worth the two review rounds it took (RETURN_TO_DESIGN, then NEEDS_FIXES catching a second smaller gap in the same fix) rather than rubber-stamping the first draft.

• Reproducing HA bootstrap bugs: match the user's actual inter-node start stagger exactly — a tight ~0.5s stagger (the /xce-deploy-ha skill's default) can mask a bug that only triggers when a node sits solo for tens of seconds (real peers answering fast enough prevents the trap window from ever closing). Always ask/check the real timestamps before concluding "could not reproduce."

• The sim/replay driver (sim_loop.c) does not fully replicate the normal-mode driver's (driver.c) per-session obligations, and this has now caused two separate real bugs: xce_core_begin_session()/xce_core_signal_go() (core_session.c) were deliberately split from the older monolithic xce_core_go() (scheduler-core spec task 5.2) specifically so a driver can interleave plugin_begin_session()/sched_account_jobs() between the resval reset and the picker signal — driver.c:739-749 does this correctly; sim_loop.c:680 still calls the old monolithic xce_core_go() with zero interleaving slot and never calls sched_account_jobs() anywhere (bug-1613: every worker's/shared resource's "used" accounting stays permanently zero in --sim/--interactive, since the per-session resval reset is never followed by re-accumulation). Before trusting any sim-driver behavior that has a normal-mode counterpart, grep the normal-mode driver (driver.c) for the equivalent call and confirm sim_loop.c actually has one too — don't assume parity. UPDATE 2026-08-24 (xce-audit-invariants scheduler run): bug-1613 is now CONFIRMED FIXED (scheduler-sim-decision-parity implementation-complete 2026-08-22) — sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709) reconstructs job->accounts from the recorded decision stream and sched_account_jobs() is called at sim_loop.c:948, mirroring driver.c. bug-1611 is only PARTIALLY fixed — the same mechanism only reconciles when a job's DISPATCH decision and its RUN-state confirmation land in the same replay round (trigger gate at sim_loop.c:592 requires JOB_IS_RUN before it will even search that round's decoded session), so the original real-world scenario (dispatch round 32, confirmation through round 107) still silently drops — a dedicated follow-up spec scheduler-sim-eager-reconciliation is already scoped (requirements-generated, not yet designed) to decouple allocation capture from the RUN-state gate. The audit also surfaced a third, currently-inert instance of this exact shape: plugin_begin_session()/plugin_end_session() (plugin.c, driver.c:750,802) are never called by sim_loop.c at all — harmless today only because sim mode's sim_bootstrap_picker() refuses every picker except builtin_fcfs, which leaves those hooks NULL (bug-1681). FIXED same day: both relocated to xce_sched_core_t* (mirroring sched_account_jobs()'s own relocation -- their bodies never touched anything beyond core->plugins, so the driver.c comment claiming plugin_begin_session() "needs the full xce_scheduler_t*" was itself stale/wrong, not just sim_loop.c's missing call) and mirrored into sim_loop.c's round boundary. Blast radius was wider than the one file: relocating a function's signature that's void*-typed in its header can hide callers from a simple grep of the two files you're editing -- a full-tree grep for the function name turned up a THIRD live call site (scheduler.c's legacy sched_begin_session()/sched_end_session()) plus four testsuite files with their own hand-written stub definitions of the same functions (matching the old signature), none of which surfaced until the first build attempt failed on scheduler.c (a file nobody had touched). Lesson: before relocating/changing any function's signature, grep the FULL src+testsuite tree for its name first, not just the files your task description mentions -- a void*-cast internal signature is exactly the shape that lets a stale caller compile silently right up until the declaration changes. RED/GREEN independently verified by temporarily commenting out the two new call sites, confirming both new spy assertions fail (Expected 2 Was 0 / Expected 1 Was 0), then restoring (git-diff-confirmed byte-identical) and reconfirming green. General lesson: this project's own buglog fix field can go stale relative to fast-moving specs — a spec can reach implementation-complete without anyone updating the buglog entry that motivated it, so always re-verify current code/spec.json state before trusting a buglog "not yet fixed" status as still accurate. Also: any per-session driver obligation found once in this file (sched_account_jobs, decoded_session->decisions application) is worth checking for siblings — the invariant is general ("any core-derived state driver.c produces that sim_loop.c doesn't mirror"), not just the two specific calls first found.

• Scheduler sim REPL has TWO independent, un-gated stdout print paths, not one: xce_repl_on_event() (repl_break.c, sim->interactive-only) unconditionally echoes every decoded event regardless of step vs continue (bug-1595's fix over-applied — it was requested for step only), and sim_print_decision() (sim_loop.c, fires independent of the REPL/breakpoint machinery) reprints an identical PEND/DISPATCH/PREEMPT line every round a decision recurs, with no dedup and no session/round context on the line. Neither path consults xce_repl_on_decision()'s existing per-round tally (last_n_dispatch/last_n_pend/last_n_preempt, surfaced only via info session). Logged as bug-1609/bug-1610 (not yet fixed) — before touching REPL output volume, check both files, not just repl_break.c.

• A soak's own "excluded"/PASS verdict is not a license to stop reading (bug-1563): /xce-test-ha --chaotic's verdict deliberately excludes bug924_fdatasync (overlayfs EIO) from pass/fail — but a huge excluded count (300K+ in one run) or a different errno than the documented one (EINVAL, not EIO) is worth a manual look anyway. Found: fsync_worker's retry loop only backs off when NOT pending, and a sustained fdatasync failure makes "pending" permanently true (durable_lsn never advances on failure) — a real busy-spin/livelock, invisible to the soak's own job-loss/divergence checks since the other 2-of-3 standbys still satisfy majority. Lesson: skim excluded/noise counts for magnitude and errno before trusting a documented exclusion covers them.

• Fixing a busy-spin/livelock can unmask a latent notification gap (bug-1564): throttling an unbounded retry loop (bug-1563's fix) removed an accidental fast-repoll that had been quietly covering for persist_flush() never firing on_durable_advance (only the async fsync_worker did). A deterministic (3/3) new test failure appeared immediately after — always suspect "the old code accidentally covered for a real gap via sheer retry speed" when a correctness fix (removing a spin/race) makes a previously-green test start failing 100% of the time; that's a signal to look for a missing notification/wakeup, not to revert the fix.

• nix devShell Python packages are PYTHONPATH entries, not merged into python3's own site-packages (found building a standalone-installable xceadmin wheel): python3 -m venv --system-site-packages only inherits the base interpreter's OWN site-packages dir, which is empty in this project's nix shell (python3Packages.click/pycapnp/etc. are each a separate /nix/store/... path added via $PYTHONPATH, confirmed via nix develop … --command sh -c 'echo $PYTHONPATH'). A --system-site-packages venv therefore silently CANNOT see them (ModuleNotFoundError at runtime, discovered live). Fix: skip venv/pip entirely — extract the wheel directly (python3 -m zipfile -e) and bake the resolved $PYTHONPATH string into a wrapper script's export PYTHONPATH=... at generation time (expand now, not ${PYTHONPATH:-} at runtime) — mirrors src/xceadmin/meson-install.sh's existing nix-packaging wrapper, which already solved this correctly; I should have grepped for that pattern before reaching for venv the first time.

• src/tools/xce-build.sh's setup_build() does rm -rf "$BUILD_DIR" for every single-component invocation (bug-1565, FIXED): /xce-build xceadmin after /xce-build controller in the same dir used to silently delete xce_controller/xce_test/etc. Fixed by an opt-in --incremental flag (meson setup --reconfigure in place instead of wiping) — deliberately NOT the default, since this project's capnp-generation custom_target has known missing-dependency-edge races that a full wipe papers over as a loud, retry-fixable failure (skipping the wipe could turn the same gap into silent stale-generated-code instead). /xce-build all (build everything together) remains the simplest option when you don't specifically need incremental speed.

• A never-joined background thread reading a stack-local test fixture is a latent ASAN stack-use-after-return, exposed only by test ORDERING, not by the thread's own logic (2026-08-19, scheduler-core design-review follow-up). sched_sim_driver_run() (sim_loop.c) spawns sim_picker_thread_fn on a real pthread that is, by this codebase's own documented design, "never joined or cancelled... no sim-driver teardown/fini function exists yet" — it loops forever on cbuf_get(&sim->core.wbuf, ...). Every existing white-box test in test_sim_event_loop.c already called this with a stack-local xce_sched_sim_driver_t sim;, which was ALWAYS a latent stack-use-after-return once the test function returned — it just hadn't been caught, because no test's stack frame had yet been reused by a later frame in a way ASAN's shadow-memory detector caught. Simply INSERTING one new test between two existing ones (no logic change to the existing tests) shifted the stack-reuse pattern enough to make it fire deterministically (reproduced twice). Fix: change the fixture from stack-local to static storage — safe here specifically because (a) sched_sim_driver_start() unconditionally memset()s the whole struct as its first action so no stale-data leak across runs, and (b) each affected test function is called exactly once per process (verified via grep, no re-entrancy) so a second call's memset can never stomp a still-live zombie thread's cbuf mutex/condvar state from a first call — static would be UNSAFE to apply to a helper called more than once while its first zombie thread might still be alive. General lesson: when a background thread spawned by code-under-test is documented as "never joined," any test giving that code a stack-local fixture is a ticking ASAN time bomb regardless of whether it has fired yet — check for exactly this "never joined/cancelled" phrasing in the target function's own comments before adding a new test that changes stack layout/ordering in the same file.

• A plain (non---system-site-packages) venv's pip install DOES work for xceadmin's deps, including pycapnp, as long as venv creation + pip install + running the installed entry point all happen inside nix develop … --command (2026-08-20, build-time-versioning task 6.3, extends the PYTHONPATH entry above rather than contradicting it): unlike the --system-site-packages case, a plain venv's pip install pycapnp resolves the prebuilt PyPI wheel fresh over the network (confirmed reachable: pypi.org/files.pythonhosted.org both resolve in this sandbox) — no PYTHONPATH-visibility problem, since pip is installing its own separate copy into the venv rather than trying to see the nix-injected one. The catch: that PyPI pycapnp wheel dynamically links libstdc++.so.6, which the bare host has NO system-wide copy of at all (import capnp → ImportError: libstdc++.so.6: cannot open shared object file outside nix) — confirmed the SAME venv/pip-install/import capnp sequence works cleanly when the whole thing (venv creation, pip install, AND the later python/entry-point invocation) runs inside nix develop $SRC_DIR --command bash -c '...', which resolves it via the devshell's own environment. Net: use a plain venv + real pip install (matching .claude/skills/xce-build/SKILL.md's own documented "Installing the xceadmin wheel elsewhere" section) for throwaway/test installs where network is available and everything runs under nix develop; reach for the PYTHONPATH-entry's zipfile-extraction wrapper trick instead only for a permanent, no-nix, no-network deployment target (e.g. /xce-deploy-ha's scripts/xceadmin).

• A pipeline into head -N can SIGPIPE-kill the upstream script early under set -o pipefail (bit twice this session: once with timeout N cmd | head -N leaving a stale background process running for hours after the harness reported the task "complete", once with xce-build.sh ... | head -3 silently truncating a build before it finished writing its wheel). If you need to sanity-check just the first few lines of a long-running command's output, redirect to a file and head the file afterward instead of piping directly into head.

• sim_loop.c (and its *_isolation-audited siblings) genuinely has zero line-budget headroom — its own file header says so explicitly. When a fix needs new logic there, split a new sibling file (bug-1567's sim_picker.c, following the exact precedent chain sim_loop.c→sim_record.c→repl.c→repl_break.c/repl_info.c→repl_params.c) rather than growing it even slightly. This requires wiring the new file into: scheduler_srcs AND sched_sim_isolation_lib (both src/scheduler/meson.build), a new XCE_SIM_*_PATH macro (testsuite/meson.build) consumed by test_sim_isolation.c's 3 per-file text-scan functions, AND every white-box test TU that #includes sim_loop.c directly (9 files this session — grep #include "../src/sim_loop.c" in testsuite/ to find them all) needs the new file #included too or the link fails on the new extern. Mechanical but real work — budget for it.

• controller_unit_test's intermittent SIGSEGV (bug-1553, ~1-in-3 full runs) is a known pre-existing flake, not a new regression — reproduce standalone (./src/build.dbg/controller/controller_unit_test, no meson test wrapper) a few times before suspecting your own change; meson test's MALLOC_PERTURB_ randomization per invocation is what makes it flaky, not deterministic per-build.

• bug-1568 (FIXED) had TWO independent, unrelated root causes stacked on the same symptom ("info worker" showing 0.00 for real resource values) — worth remembering the general pattern: (1) xce_core_push_events()'s WORKERS_PUSH case called sched_add_worker() (a plain key_add(), add-only) unconditionally on every push; a worker connecting sends multiple pushes in quick succession, so any push after the first for an already-registered name silently -EEXIST-discarded its resource values — fixed via core_upsert_worker(), mirroring the existing core_upsert_queue() add-or-update precedent (core_events.c). (2) Separately, repl_info.c's display code compared xce_val_t.which against XCE_VAL_NUMERIC (xce/types.h #define, value 1 — a resource-definition value-type constant, xce_resource_t.valtype's own enum) instead of Value_u_fvalue (controller.capnp.h-generated, value 5 — the real tag every producer in this codebase actually sets on xce_val_t.which) — always false, so it displayed 0.00 regardless of whether (1) was fixed. Lesson: when value-level TRACE at every pipeline hop (push → apply → snapshot save → snapshot load/decode) all show the correct number but the final displayed/returned value is still wrong, the bug is one hop further than "the pipeline" — trace the DISPLAY/formatting code too, don't stop at the data layer. And don't assume two enum families with parallel-looking names (XCE_VAL_* vs Value_u_*) are interchangeable just because both appear near xce_val_t/.which/.valtype — check which one a given producer actually sets before trusting a comparison against it.

• IFS=$'\t' read (or any single-tab IFS) silently COLLAPSES adjacent/empty fields, just like the default whitespace IFS does -- bash classifies tab (along with space/newline) as "IFS whitespace" for splitting purposes regardless of what IFS is actually set to, so a\tb\t\tc\td (an empty 3rd field) reads back as 3 fields (a,b,c\td collapsed) not 4, silently shifting every field after the gap. Found live building resource-value-verification task 7.1's aggregate_verdict() (run-resource-verify.sh) -- its own required synthetic self-test caught it immediately (an UNREACHABLE-layer record's empty *_observed field shifted every later field by one, corrupting the apiserver classification). Every OTHER tab-delimited TSV already in that file happens to never carry a genuinely-empty field, which is why this never surfaced before. Fix: when a delimited record format can legitimately have an empty field, use a non-whitespace-class separator (e.g. ASCII Unit Separator $'\x1f') instead of tab -- IFS=$'\x1f' read does NOT collapse/strip, tab-like or space-like IFS always does.

• This Bash tool aborts a multi-line script at the first command returning nonzero, even with ; true/|| true on that exact line — observed repeatedly with pkill matching zero processes (a completely normal, expected outcome) silently killing every following line in the same call. Don't chain a possibly-zero-match pkill/similar with follow-on commands in one Bash call — verify via pgrep first (it's just as prone to this, so treat ITS nonzero "no match" the same way) or issue the pkill as its own standalone call and check the result before proceeding.

• nix build in this sandbox dispatches to a remote builder (ssh://rshen@172.31.109.18) that is NOT reachable from inside the sandbox (2026-08-21, dirty-content-fingerprint task 6.3 review). nix show-config//etc/nix/nix.conf inside the sandbox show builders = (empty) — the remote dispatch is configured on the nix-daemon side (invisible to a show-config run inside the sandbox's mount/network namespace), not discoverable by inspecting client-side config. A plain nix build therefore either hangs/times out on SSH connect or falls back to the known bcctool/goproxy.cn failure without ever completing locally. Fix: pass --builders '' to force a fully local build (this sandbox has max-jobs = 22, plenty of local capacity) whenever running a real nix build/nix develop-triggered build for verification in this sandbox — don't assume remote-dispatch failure/unreachability is unavoidable or treat it as an automatic excuse to fall back to eval-only/log-only verification; try the local-builder override first, since it can produce a genuinely complete, runnable binary instead of just a build log.

• apiserver's main.go only calls RegisterToBackend when BackendYamlConfig.Address != "" — checked BEFORE Peers is ever consulted (cmd/main.go:58). Setting app.yaml's backend.peers to a fully-populated HA candidate list while leaving backend.address/backend.port blank (reasoning "Peers takes priority inside RegisterToBackend anyway, per backend/config.go's own doc comment") is NOT sufficient — the apiserver silently boots in permanent "stub mode" (logger.Info("Backend not configured")) and never even attempts a connection, peers or not. Caught live while building xce-deploy-ha's new apiserver step (resource-value-verification task 3.1): always populate address/port too (e.g. the first --peers candidate's host:port) even when also setting peers — belt-and-suspenders, not redundant, because the two fields gate two different things (whether to connect at all, vs. which candidate(s) to try).

• resource-value-verification task 4.1's scheduler trace-log exact format (xce_core_push_events()'s XCE_CORE_EVENT_RESVALS_DELTA case, src/scheduler/src/core_events.c, right after zarray_put(&(w->resvals), res->idx, rv->val)): TRACE("resval applied host=%s resource=%s value=%f", ...) for numeric (Value_u_fvalue) resources, or value=%s for Value_u_svalue ones — live-confirmed via xce-deploy-ha + xce_test worker-context join/resupdate -n r15m -v 12345.5 producing exactly resval applied host=task41-worker resource=r15m value=12345.500000 in scheduler.log at -vvv. Task 6.3 (scheduler-side log-tail check) MUST grep for resval applied host=<H> resource=<R> value=<V> — labeled key=value fields, no other fixed text to depend on. %f on a normal resource value produces trailing zeros (e.g. 12345.500000), not the input's original decimal precision — task 6.3's comparison should parse-and-compare as float, not do a literal string match against the injected value string.

• bug-1572 (NOT FIXED, pre-existing, discovered live task 4.1 round-2): xce_scheduler double-frees on a worker's SECOND periodic resource report for the same resource(s), silently dying with no ASAN output (backgrounded via setsid nohup ... >/dev/null 2>&1, so the crash report is lost -- only visible as scheduler.log stopping mid-stream + handle_scheduler_leave in the controller log ~1s later). Root cause: core_events.c's XCE_CORE_EVENT_RESVALS_DELTA case does zarray_put(&(w->resvals), res->idx, rv->val) with NO MOVE_PTR (unlike its sibling core_apply_config_resvals(), which does MOVE_PTR(val, nv->val) first) -- the same xce_val_t* stays aliased in the decoded request. driver.c's release_translated_payload() then unconditionally frees that request's resvals list even when applied=1 (its XCE_SCHED_OP_UPDATE_RESVALS case has no if (!applied) guard, unlike the sibling XCE_SCHED_OP_UPDATE_CONFIG case which explicitly skips the free when applied). Net effect: w->resvals[idx] is left dangling immediately after every successful RESVALS_DELTA batch; the SECOND update to the same index triggers zarray's killer to free the already-freed pointer. Confirms identically on round-1's code (unrelated to the enum-branch bug fixed in that round) -- a live smoke test that only runs one resupdate quickly after deploy will NOT hit this; anything that waits past one real worker's report interval (xcecd_virtual reports ~every 15s) will. When live-testing anything touching RESVALS_DELTA, either run fast (single update, no wait) or expect this crash and don't misattribute it to whatever fix is actually under test.

• src/frontend/Makefile's build-server target is missing CGO_CFLAGS/CGO_LDFLAGS (bug-1597, NOT FIXED): unlike test-backend (bug-1558) and apiserver/meson.build, build-server's recipe is a bare go build with no -I/-L for client.h/controller.capnp.h/libxce.a, so nix develop ../ --command make build-server fails with fatal error: client.h: No such file or directory regardless of source changes. Confirmed via git-stash A/B that this reproduces identically with or without an unrelated diff, so don't misattribute it to whatever change is under test. Workaround to actually verify compilation: reconstruct CGO_CFLAGS/CGO_LDFLAGS the same way test-backend does (pkg-config for librtti/c-capnproto/openssl/tinylog + -I$BUILD_DIR/lib/-L$BUILD_DIR/lib -lxce, BUILD_DIR defaulting to src/xce.build) and run CGO_ENABLED=1 CGO_CFLAGS=... CGO_LDFLAGS=... go build -o /tmp/apiserver ./cmd/main.go directly inside nix develop ../ — see bug-1597 for the exact one-liner.

• test.c's set_resval() leaves xce_resval_t.name (and .val->which) uninitialized for the Value_u_svalue fallback path (-n type/-n model/explicit -t string on an unknown name) — its switch statement has no case Value_u_svalue: arm, and vals[] is a non-zeroed stack array reused across REPL invocations, so .name can be garbage, not NULL. Pre-existing (predates resource-value-verification's task 2.1/2.2), explicitly flagged as out-of-scope-to-fix in both tasks' code comments (Requirement 1.4's "preserve byte-for-byte" constraint). When building task 2.2's per-name verification loop on top of this, the safe pattern was to track the submitted name in a separate parallel array (subnames[]) captured directly from the getopt loop's own resname variable (always valid — that pointer is deliberately leaked/never freed, so it stays alive for the whole REPL call) rather than trusting vals[i].name.

• Trap-based cleanup that must survive an async signal (SIGTERM/SIGINT) should combine multiple pkill -9 -f calls into ONE alternation pattern, not several sequential invocations, plus a retry-until-verified (pgrep) loop (bug-1575, resource-value-verification task 5.1's run-resource-verify.sh). Four separate sequential pkill -9 -f -- "pattern" calls inside cleanup() worked reliably every time when reached via normal control flow (a direct fail()/exit call), but non-deterministically left 1-2 of the four target processes running when cleanup() instead ran from the EXIT trap fired by an externally-delivered SIGTERM translated via a separate trap 'exit N' TERM handler -- the exact failure point (which of the 4 pkill calls was "last to complete") varied run to run, consistent with a timing race in the trap-handler code path rather than a pattern-matching bug (each pattern was independently verified correct). Fix: one combined pkill -9 -f -- "pat1|pat2|pat3|pat4" call (fewer fork/exec round trips through the trap handler) wrapped in a loop that re-checks via pgrep -f and retries (up to ~2s total) until genuinely clean. Verified reliable across 5+ repeated live SIGTERM-mid-deployment interrupts after the fix (zero leftover processes every time), vs. non-deterministic leftovers before it. General lesson: don't trust a cleanup routine's reliability from its NORMAL-completion path alone -- explicitly live-test the signal-interrupted path too, since bash's async-signal-trap execution context can behave differently from the same code reached via ordinary top-to-bottom flow or a plain function call/exit.

• pgrep -f/pkill -f run interactively via this Bash tool can self-match (and pkill can self-KILL) the invoking Bash-tool call's own wrapper process (bug-1576, discovered repeatedly while diagnosing bug-1575 above -- cost significant back-and-forth before recognized). This harness wraps every Bash call as roughly bash -c "... && eval '<submitted script text>' ...", so the wrapper's own /proc/<pid>/cmdline contains the FULL LITERAL TEXT of whatever was submitted -- including any search pattern typed directly into that same script (unavoidable, since testing a pgrep/pkill pattern means typing it). A pgrep -f -- "sometext" that also appears as literal text elsewhere in the SAME submitted command block will match the wrapper too; pkill -9 -f in the same situation can SIGKILL the current Bash call's own shell mid-script, truncating all remaining output with no distinguishing error beyond a generic nonzero/early-stop. Symptoms seen: a match count of "2" when only 1 real process existed; a multi-line diagnostic script silently stopping after the pkill line with no further output. Mitigation: when checking process state from an interactive Bash call, scan /proc/[0-9]*/cmdline directly and eyeball results for the wrapper's own distinctive bash -c "source .../shell-snapshots/... && eval '...'" shape rather than trusting a bare match count; never assume a lone pgrep/pkill match is the real target without checking its ps -o cmd first. This does NOT affect a target script's OWN internal pkill -f calls (e.g. inside a cleanup() function) since that script's own argv is just its invocation line (interpreter + path + args), not its source text. This ALSO bites a from-scratch verification watcher, not just a diagnostic pkill: any background polling loop launched via the interactive Bash tool with the target pattern typed inline in its own command text self-matches too (found live, task 6.1) -- write the pattern into a script FILE and launch that file instead, or check /proc/<pid>/exe (the resolved binary path) rather than grepping /proc/<pid>/cmdline text at all.

• xce_test's worker-context injection recipe (resource-value-verification task 6.1, cross-checked against xce-test-ha/run-ha-soak.sh's extensive prior use of the identical pattern): printf 'set masters %s\nset myname %s\nworker\njoin\nresupdate -n <name> -v <value> [...]\nquit\n' "$PEERS" "$WORKER_HOST" | xtest. set myname <host> is what tags every subsequent resupdate'd value's .host field (test.c: config.from = get_setting("myname"); set_resval() sets vals[idx].host = get_setting("myname")) -- Requirement 2.2's "tag every resupdate-injected value with the joined worker's own identity" is satisfied entirely by this one setting, no separate flag needed. resupdate_report_one()'s output lines are exact whole-line strings (<name>: accepted / <name>: rejected: unknown resource / <name>: rejected: value mismatch) -- parse with grep -Fxq (fixed-string, whole-line), not a substring/regex grep, since a short resource name (it, io, ls...) can appear as a literal substring of an unrelated line otherwise.

• A synthetic worker's host-attrs resources (type/model/cpuf/maxcpus/maxmem/maxswap/maxtmp/ndisks) get RESET to baseline the moment it disconnects -- this is correct, intentional production behavior, not a bug, but it bites any test tool that queries the controller in a SEPARATE xce_test process/session after the injecting session's own quit (resource-value-verification task 6.2, bug-1579, not-a-bug/documented). add_hostattrs_resvals() (src/controller/worker.c:260-287) is the fixed N_HOST_ATTRS list; mark_worker_unavail() -> apply_worker_dead_mutation() (worker.c:704-716) calls reset_hostattrs_unknown() then re-applies add_hostattrs_resvals() on every worker disconnect, by design (a dead worker shouldn't keep advertising stale capacity). run-resource-verify.sh's inject_resources() (task 6.1) runs join;resupdate;quit as ONE xce_test process -- the quit disconnects the synthetic worker before ANY later per-layer check (a separate xce_test process) can observe it. A live controller-check run against the task's own DEFAULT_RESOURCE_ARGS set showed 8/9 MATCH + 1 false MISMATCH (cpuf, the only host-attrs-colliding name in that default set) -- re-running with the same 8 non-cpuf resources gave a clean 8/8 MATCH, confirming the check logic itself is correct and this is purely a worker-disconnect-timing artifact. Only the CONTROLLER-layer check (a live resval query) is exposed to this -- the scheduler/apiserver checks (tasks 6.3/6.4) are trace-log-tail based (read what was logged at apply-time, immune to later disconnect). When choosing/validating a resource name for any live check against this script, cross-check it isn't in worker.c's N_HOST_ATTRS list first, or expect a query run in a later, separate xce_test session to legitimately read back 0/default. UPDATE 2026-08-18 (task 6.2 remediation round 2, reviewer-mandated fix): the default-set instance is now FIXED. DEFAULT_RESOURCE_ARGS swapped cpuf=9.5 for r1m=9.5 (a periodic/dynamic load-index resource per src/worker/src/resource_sync.c, confirmed pre-existing/"already-supported" via resval_typeresolve.c's resinfo[] table, and NOT in worker.c's N_HOST_ATTRS list) -- live-validated clean 9/9 MATCH on the default (no-args) set post-fix, no regression on mem=5-only or mixed accept/reject scenarios. Option (b) (keep the synthetic worker's session connected across all per-layer checks instead of quitting) was investigated live in this same round and confirmed NOT a simple call-reordering fix: xce_test's REPL has no command path from the worker context back to the client context without first calling leave_ctx() (via ../cmd_up), which performs the exact same disconnect quit does -- making option (b) work would require either a C-level xce_test change or a materially riskier bash coprocess restructuring, both out of scope. This class of issue (a host-attrs resource name queried after synthetic-worker disconnect) remains a live risk for any FUTURE task or user-supplied custom resource=value argument naming type/model/cpuf/maxcpus/maxmem/maxswap/maxtmp/ndisks -- see buglog.json bug-1579 for the full analysis.

• bug-1577 (FIXED, task 6.1): xce-deploy-ha's deploy-ha.sh silently starts a REAL worker daemon (xcecd_virtual) whenever one happens to be built alongside the controller (its own pre-existing opt-in-by-presence convention, predates resource-value-verification) -- a plain /xce-build all leaves xcecd_virtual present, so xce-test-resources's own deployment picked one up unconditionally, contradicting Requirement 2.1 ("no real worker daemon process required") and risking bug-1572 (scheduler crash on a resource's 2nd periodic report) on a long-enough run. Fix, without touching deploy-ha.sh (out of that task's boundary): the generated start-worker.sh reads ${XCE_WORKER_BIN:-<real path>} from its OWN inherited environment at run time -- export XCE_WORKER_BIN=/bin/true before invoking the generated start-all.sh, so the worker step exits immediately with zero persistent process. General lesson: when a skill's design deliberately avoids starting some component, verify what a shared deploy helper ACTUALLY launches by inspecting real running processes, don't assume your own new code is the only thing determining what ends up running.

• bug-1578 (FIXED 2026-08-18): controller SIGSEGVs on worker disconnect if a scheduler ever connected AND that worker's resvals were pushed at least once. Fix applied: sched_worker_change()'s dead #if 0 ... #endif block replaced with w->host.nresvals_to_sched = 0; w->host.resvals_to_sched = NULL; before xlist_fini(&resvalist); -- exactly mirrors apiserver_worker_change(). Verified two ways: (1) xce-build.sh controller clean build, 1102/1104 tests pass (2 failures are pre-existing bug-1561 overlayfs/fdatasync, unrelated); (2) live repro via run-resource-verify.sh --keep-running -- deployed full stack, joined synthetic worker, ran 9 resupdates, disconnected (quit), confirmed the exact crash-trigger log sequence fired (handle_worker_leave -> sched_worker_change pushing resvals to scheduler twice) and the controller stayed alive ~2 minutes past disconnect with zero SEGV/ASAN/core-file (previously died within 1-10s every time). UPDATE 2026-08-18 (follow-up task) -- the "no testable-static-function harness pattern exists" claim below was wrong; corrected so it isn't trusted stale in a future session. Added src/controller/testsuite/test_sched_worker_change.c (new standalone binary sched_worker_change_unit_test, wired into meson.build mirroring sched_push_unit_test's real-sched.c+dataex.c+egress.c linking strategy -- the exact same "drive a static hook function through the real installed hooksys via xce_run_hooks()" pattern test_job_array_sched_push.c already proved for the JOB hook works identically for the WORKER hook: on_scheduler_join() installs sched_worker_hook_fun, then xce_run_hooks(&c.hooksys, XCE_CONTROLLER_HOOK_WORKER, HOOK_AFTER_EVENT, HOOK_DETAIL_WORKER_DEAD, &c, w) reaches sched_worker_change() exactly as handle_worker_leave() does in production). Test 1 asserts w->host.resvals_to_sched==NULL && nresvals_to_sched==0 after the hook fires. Test 2 is the stronger ASAN-strength proof: it calls the REAL production encoder encode_Worker() (controller.capnp.c, linked for free via libxce_dep -- no need to link persist.c at all, since persist.c's own capnp_encode_worker() is just a 20-line pass-through to it) directly on the worker right after the hook fires, exactly what persist_worker_hook_fun does on disconnect in production. RED/GREEN-verified live: temporarily re-wrapping the fix in the original dead #if 0 ... #endif reproduced a genuine ASAN heap-use-after-free (encode_NameValue_list, controller.capnp.c:3408, freed by xlist_fini<-sched_worker_change<-sched_worker_dead<-sched_worker_hook_fun) -- then the approved fix was restored (confirmed byte-identical via git diff to the pre-revert state) and both tests went GREEN. Gotcha hit along the way: makeResource()/makeString() (resobj.c/xstring.c) need the global xstring intern table initialized first (xstring_table_init(N), once, before first use) or makeString() returns an object whose ->name is NULL, and the very next key_add()/hash_string() call SIGSEGVs on the NULL string -- easy to miss in a fresh standalone test binary since no other linked test file has already called it process-wide (existing precedent: test_worker_join.c's own RUN_TEST call site comments this exact requirement). Original root-cause notes retained below. sched_worker_change() (src/controller/sched.c) aliases w->host.resvals_to_sched/nresvals_to_sched directly to a local xlist_t's backing array, pushes it, then xlist_fini()s that array WITHOUT resetting the alias first -- the correct reset lives in the SAME function but is wrapped in a dead #if 0 /*FIXME*/ ... #endif (commit acfbec875, 2026-03-03 -- predates resource-value-verification by ~5 months; a later commit e5fde7b5 "fixed" this but edited code INSIDE the already-dead block, so its fix silently never ran). Leaves w->host.resvals_to_sched dangling for the worker's whole remaining lifetime. On disconnect, handle_worker_leave -> mark_worker_unavail -> xce_run_hooks -> persist_worker_hook_fun -> capnp_encode_worker -> encode_Worker -> encode_NameValue_list -> encode_NameValue dereferences the freed array -> SIGSEGV (si_addr=NULL). Only reachable once a scheduler has connected at least once (on_scheduler_join() registers sched_worker_hook_fun on a one-shot static hook_installed gate) -- a solo controller or controller+apiserver-only never triggers it, which is why the original A/B isolation matrix (lone controller alive 90s+, full stack + xce_test never run alive 100s+) didn't catch it: the missing variable wasn't "was xce_test run" but "did a scheduler ever join AND did that worker's resvals get pushed." Sibling apiserver_worker_change() (apiserver.c) does the reset correctly (not #if 0-wrapped) -- mirror that exact pattern for the fix, do NOT just uncomment the dead free_name_val_list() call (it already frees the array AND nulls the field itself; the very next xlist_fini() line would then double-free the same pointer -- almost certainly why the original author reached for #if 0 instead of deleting it). Fully outside resource-value-verification's task boundaries (sched.c, untouched by tasks 2.1/2.2/6.1) -- tasks 2.1/2.2 are innocent, no re-review needed. See buglog.json bug-1578 for the full backtrace and fix diff.

• do_update_resvals() (src/controller/opupd.c) fans out the FULL, UNFILTERED submitted resource list to the scheduler AND apiserver hooks BEFORE (and still, after) update_resval_list() does its own per-item validity check — the controller's own w->host.resvals write is filtered (rejects unregistered names via key_find(c->resmgr,...)), but the HOOK_AFTER_EVENT payload sched_resource_hook_fun/apiserver_resource_hook_fun (sched.c/apiserver.c) act on is the SAME raw list regardless. Found live during resource-value-verification task 8.1 (bug-1583, NOT fixed): injecting one deliberately-unregistered resource name showed the scheduler's core_events.c independently re-validates against its own resource table and discards+WARNs on an unknown name, but apiserver's OnResourceLocationUpdated (handler.go) has zero such validation and unconditionally store.ResourceLocations.Put()s it — a genuine 3-way inconsistency (controller rejects, scheduler rejects, apiserver silently stores forever). General lesson: when a controller hook fans a raw request payload out to multiple downstream consumers, don't assume upstream validation (even a REJECT logged elsewhere in the same request) actually filtered what those consumers received — check each consumer's own hook function for whether it re-validates or blindly trusts the payload.

• When a live crash won't reproduce under gdb (even a direct gdb --args child launch, which sidesteps this sandbox's ptrace_scope=1 attach restriction), suspect gdb's ptrace overhead itself is masking a timing-sensitive bug rather than concluding "not reproducible." bug-1578: a 100%-reproducible (bare launch) controller SIGSEGV did NOT reproduce within 20s under gdb for either the ASAN or debug(-O0) build, for both a plain run and a --batch launch. Fix: skip gdb entirely -- compile a tiny LD_PRELOAD shared library (constructor installs a sigaction handler with SA_ONSTACK+a pre-allocated altstack for SIGSEGV/SIGABRT/SIGBUS/SIGILL/SIGFPE that calls backtrace()+backtrace_symbols_fd() to stderr, then signal(sig, SIG_DFL); raise(sig) to preserve normal termination semantics) and launch the target with LD_PRELOAD=./crash_handler.so <normal invocation, no wrapper>. Near-zero runtime overhead (one constructor + a couple sigaction() calls), so it does not perturb thread scheduling/timing the way ptrace does -- reproduced the exact same crash reliably and captured a full backtrace. Resolve the raw addresses with addr2line -e <binary> -f -C -p <offset...> using the (+0xNNN) MODULE-RELATIVE offsets backtrace_symbols_fd() prints (not the raw runtime addresses, which include the PIE ASLR base and won't resolve against the on-disk binary directly).

• Testing a bash function that both populates caller-scope globals AND prints via a log()-style echo (task 8.2, aggregate_verdict()): never capture its output with $(...) command substitution if you also need the globals it set. Command substitution forks a subshell in bash — any associative-array/variable writes the function makes are lost the instant the subshell exits, even though the function's exit code and printed text survive. Fix: call the function directly with plain > file redirection (fn ... > "$tmpfile" 2>&1; rc=$?), which does NOT fork a subshell, then read $tmpfile separately for text assertions while the just-set globals remain live in the current shell. This is exactly .claude/skills/xce-test-resources/verdict-logic-selftest.sh's run_agg() helper.

• This codebase's established pattern for a persistent bash-tooling regression test is .claude/skills/xce-test-ha/soak-verdict-selftest.sh: name it <subject>-selftest.sh, source (not execute) the target script to get its functions in scope, use a check DESC EXPECTED ACTUAL string-equality helper tallying a fails counter, end with a SELFTEST PASS/SELFTEST FAIL (N failed) line and exit 0/1. run-resource-verify.sh itself is written to support this — it has a [[ "${BASH_SOURCE[0]}" == "${0}" ]] guard at the bottom specifically so sourcing it never runs main()/starts a live deployment. Reused for verdict-logic-selftest.sh (task 8.2).

• Sandbox capability update (2026-08-19): sudo now works passwordless here (sudo -n true succeeds) — this unblocks anything previously gated on "this session/skill cannot sudo" (e.g. real xcecd, worker/src/main.c's check_user()). Still true, and load-bearing for anything assuming a real production host: this sandbox has no systemd, no dbus, no /run at all (ls /run → No such file or directory; PID 1 is s6-svscan; no dbus-daemon/systemctl even under nix develop). Root access and a real init/dbus stack are independent axes — don't assume sudo working means systemd-dependent things (job dispatch via org.freedesktop.systemd1.Manager.StartTransientUnit, anything needing /run/dbus/system_bus_socket) will also work. /run/systemd/system existing is the standard sd_booted(3)-style signal for "a real systemd instance is running as this host's manager" — used in src/tools/deploy-ha.sh to auto-pick xcecd (real) vs xcecd_virtual per host (XCE_REAL_WORKER=0|1 overrides).

• A root-owned file created by a daemon under sudo is not automatically readable by the invoking non-root user — confirmed live (bug-1592): real xcecd's main log file comes out root:root mode 640 even though the SAME binary explicitly chmod(..., 0666)s its domain socket and its xcecd_worker.log sibling. Don't assume "the daemon already chmod's some of its output files permissive" generalizes to all of them — check each file's actual mode after a live run before writing a verification command that reads it as the invoking user; prefix with sudo where it doesn't.

• The "rule out a concurrent competing build/soak" caution (line 16 above) includes SIBLING AGENTS in your own Claude Code session, not just unrelated external processes (2026-08-23, scheduler-dispatch-commit task 3.1 GREEN verification). A peer subagent within the same session independently launched its own xce-build.sh controller asan xce.build.green31 --incremental + meson test against the exact build dir this session was told to use exclusively, overwriting the live meson-logs/testlog.txt in place mid-investigation. Identify this via ps -o pid,ppid,cmd to find the process tree, then tr '\0' '\n' < /proc/<pid>/environ | grep CLAUDE_CODE_SESSION_ID — a match against your own session's ID (visible in your own scratchpad path) confirms it's a sibling agent, not an unrelated build. Mitigation: copy/preserve any log file you need as evidence (e.g. sed/python3-clean it into your scratchpad) the MOMENT you capture it, before doing further slow diagnostic work that gives a peer time to overwrite the live file in place.

• controller_unit_test TIMEOUT (300s SIGTERM, Ok:4 Fail:0 Timeout:1) is bug-1429, a pre-existing, NOT-yet-root-caused, systemic hang — now reproduced 3 times across 3 completely unrelated diffs (delaymgr/persist.c changes; a pure version-string change; scheduler-dispatch-commit's job.c change) and 3 different hang locations each time (test_demotion_replay_yields_committed_only; the test_configack_* group; test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs). Before suspecting your own diff caused a hang here: (1) check whether the hanging test is even in a file your diff touched — if not, that's strong evidence against causation; (2) /proc/<pid>/task/*/wchan on the hung process (gdb attach is blocked by sandbox ptrace restrictions) — both/all threads in futex_do_wait confirms deadlock-consistent, not CPU-bound; (3) re-run the already-built binary directly (bypassing meson's 300s cap) with a generous timeout — if it reproduces in total isolation with zero concurrent activity, that rules out resource contention as the cause too. Update bug-1429's occurrences/last_seen/fix narrative rather than filing a new bug — do not conclude "regression in my diff" from this signature alone.

• xce_resource_t (xce/types.h) has a flags bitmask (XCE_RESF_BUILTIN/DYNAMIC/GLOBAL/SHARED/EXTERNAL/RELEASE/DEFAULT_HOST/CONSUMABLE/LIMIT) distinguishing a genuinely cluster-wide resource (GLOBAL/SHARED, stored ONCE in core->sharedInst, a zarray_t indexed by the resource's idx -- the same pool RESVALS_FULL/RESVALS_DELTA event handling and snapshot save/load already treat as authoritative) from a host-based/per-worker-distributed resource (stored per-worker in w->resvals/w->resInUse). Before this session, repl_info_resource_one() (scheduler REPL) conflated the two -- aggregated total/available across every worker's resvals for EVERY resource regardless of kind, a fabricated number for a host-based resource (cpus/mem) and redundant with info worker's own correct per-host display. When adding any new resource-value display/log code, check r->flags & (XCE_RESF_GLOBAL|XCE_RESF_SHARED) first to decide which pool (core->sharedInst vs per-worker resvals) is the correct source of truth -- see bug-1593.

• repl_info.c (scheduler REPL state-inspection dumps) was already OVER this project's 800-line cap (883 lines) before this session even started -- accumulated from three prior sessions' bug fixes (bug-1587/1590/1591) each adding a little without ever triggering a new sibling-file split. A task-12.5 rewrite of the resource-display functions brought it down to 856, still over. Flag this to the user rather than silently ignoring it or unilaterally doing a 4th-generation file split (repl.c -> repl_break.c/repl_info.c -> repl_params.c precedent) without being asked -- a split touches meson.build, test_sim_isolation.c's per-file scan macros, AND every white-box test TU's #include list (real, mechanical work, not worth taking on speculatively).

• A repl_split_tokens()-style whitespace tokenizer with zero quote-awareness silently breaks any downstream parser that requires a whole-string-anchored regex match (e.g. parse_jobid()'s pmatch[0].rm_so==0 && pmatch[0].rm_eo==len check) when a user quotes an argument out of habit (common when copy-pasting a value from another command's own output). The numeric fallback path (strtoull()) then silently parses the leading quote character as "not a digit" and returns 0 rather than erroring -- producing a confusing wrong-answer, not a clear parse failure. Worth checking for in any REPL/CLI tokenizer this codebase adds. See bug-1596.

• A pointer returned by makeString() (rtti-managed, lib/xstring.c) must always be released via dereference(), never FREEUP()/plain free() — makeInstance()/objAlloc() returns an offset into a combined header+data block, not the block's own malloc()-returned base address, and the string is refcounted (freed only when the refcount hits zero), not directly owned. bug-1600 (2026-08-19, production, not a test fixture — see also bug-1503's earlier test-fixture instance of the identical mismatch): lib/group.c's makeGroup() "existing group" update branch (hit every time a config-push re-applies a group name that already has members, e.g. any retried/duplicate config push — no thread concurrency required) called FREEUP(m->u.name) on a makeString()-allocated field, while the SAME FILE's group_killer() destructor released the identical field correctly via dereference(m->u.name). This caused the real, reproducible (8 confirmed core dumps) xce_scheduler free(): invalid pointer SIGABRT crash during controller-failover config-push retries. When adding or auditing any free/release path for a field populated by makeString(), grep the same file/struct for an existing destructor and match its release call — don't assume FREEUP() is safe just because it's used for adjacent plain-calloc'd fields in the same struct.

• lib/group.c's makeGroup() "existing group" update path takes only RTTI_LOCK_READ (a genuine shared pthread_rwlock_rdlock, confirmed via kvmgr.c) while mutating shared group member-list state, and the scheduler's entire config-push handler chain (sched_init_config() → plugin_config_update() → ... → makeGroup()) holds no serializing lock at all — while scheduler_req_handler() genuinely runs on whichever of the scheduler's 4 vcpu worker pthreads picked up that vtask (vtaskmgr_init(..., 4)). xce_ctx_reconnect() reuses the same ctx->ep struct and endpoint_init() unconditionally resets ep->busy=0, so a reconnect racing with a still-running config-push handler can let a second, concurrently-dispatched config-push run makeGroup() on the same group object at the same time as the first. This is a REAL latent concurrency bug distinct from bug-1600's allocator mismatch (which alone was sufficient to explain the observed crash and was the one fixed) — flagged as a follow-up, not yet fixed; would need a write-lock scope redesign since init_group_by_config()'s callees (get_group()) take their own internal read lock, so naively holding a write lock across the whole function would self-deadlock on the non-recursive rwlock.

• apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/CKPT/PREASON dispatch case (persist.c) never stamped start_time/finish_time/run_time — it only ever called the generic, state-only apply_job_update_transition(j, state) (2026-08-19, xcecli joblist showed StartTime=0 for every RUN/DONE/EXIT job, FinishTime=0 for DONE jobs). Fixed additively (user's explicit steering: "don't change the logic, just add extra information") by keying a time(NULL) stamp on wal_type right after apply_job_update_transition() succeeds, still under the same lock(j, RTTI_LOCK_WRITE) — WAL_JOB_START → start_time, WAL_JOB_FINISH → finish_time + run_time = finish_time - start_time. No change to apply_job_update_transition()'s signature or the dispatch structure. Mirrors what the richer (currently-unreached-on-this-path) apply_job_start_transition()/apply_job_finish_transition() cores in job.c already do for their own callers.

• test_config_repl.c's test_conf_trim_keeps_only_n_versions/test_conf_get_saved_tag_no_config failing with the exact same Expected/Was values plus a burst of persist: flusher fdatasync FAILED (errno=5 Input/output error) lines is bug-1561, a confirmed overlayfs-sandbox fdatasync artifact — 6 occurrences now across unrelated sessions/diffs, never once traced to a real code change. If a /xce-build controller run shows exactly these 2 failures with that exact error pattern, it's environment noise, not something your change caused — don't spend time re-investigating, just note it and move on (though still bump bug-1561's occurrences/last_seen per buglog protocol).

• Testing an "on a clean tree" scenario against a shared repo that's under continuous concurrent background editing (other agent sessions, OpenWolf hooks, etc.) cannot rely on the ambient working tree's git status ever actually being clean. dirty-content-fingerprint task 6.1 (src/tools/test-dirty-fingerprint-distinguishability.sh, 2026-08-21) solved this by creating its own throwaway, fully-isolated git worktree add --detach <tmp-dir> HEAD before doing anything else — a fresh linked worktree checkout is clean by construction (zero uncommitted changes) and is a completely separate directory tree, so concurrent activity in the original shared worktree can never touch it. Bonus: src/tools/xce-build.sh resolves its own SRC_DIR from $0's location and cds there itself before running any git command — so simply invoking the ISOLATED worktree's own copy of xce-build.sh (not the shared tree's copy, and regardless of the caller's own cwd) automatically scopes every git command inside it (rev-parse/status/diff/ls-files) to the isolated worktree, with zero extra -C/cd bookkeeping needed by the caller. Clean up with git worktree remove --force <dir> (plain rm -rf alone leaves stale worktree metadata — needs a git worktree prune fallback too). General pattern: reach for an isolated git worktree (not git stash, not hoping the ambient tree happens to be clean) any time a test genuinely needs to observe a controlled clean/dirty state transition in a repo that isn't exclusively yours for the run.

• A local a="$1" b="$a/something" — two assignments in ONE local statement where the second references the first — is a real bug under set -u, not just bad style (bug-1615, found live writing the script above). Bash expands every word of a single local command before ANY of that command's assignments take effect, so $a in the second word's value expands against the OLD (not-yet-declared) a, producing iso_src: unbound variable (or silent empty-string under plain set -e without -u). Fix: split into two separate local statements on their own lines. Same family as the earlier-logged fp="$(cmd)" under set -e gotcha (cerebrum, xce-build.sh dirty_fingerprint note) — another case where a single-line-looking assignment has a non-obvious evaluation-order trap under bash's strict-mode flags.

• builtins.currentTime is entirely unavailable ("attribute 'currentTime' missing") during a flake's default pure evaluation — not merely fixed to a dummy value, the attribute is removed from builtins outright. Confirmed via nix eval both ways: a bare nix eval .#pkg.someAttr (or nix build) referencing builtins.currentTime anywhere it gets forced fails at eval time; nix eval --impure .#pkg.someAttr succeeds and returns the real epoch. This is unrelated to the derivation build sandbox (sandbox = false in this machine's nix.conf) — it's Nix's flake-level pure-eval restriction on impure builtins, and it applies regardless of where in the file the reference sits (laziness only delays the error until the value is actually forced, e.g. by string-concatenating it into mesonFlags). self.shortRev/self.dirtyShortRev (the existing commit_id mechanism) are NOT affected — those come from flake input/git introspection that Nix computes specially even in pure mode, not from an impure builtin. RESOLVED (bug-1603, 2026-08-20): task 2.2's first attempt used builtins.currentTime in src/flake.nix per research.md's original decision, and was REJECTED by review because nix build .#xcecore_debug (no flag, the documented canonical command) failed at evaluation exactly as predicted above. Fixed by moving the computation entirely out of Nix evaluation and into each derivation's own build phase instead — see the preConfigure/mesonFlagsArray entry below. Nix-eval-time impure builtins should be treated as generally unusable for anything that must survive a plain nix build/nix eval (no --impure), full stop — don't reach for builtins.currentTime (or similar) in a flake output again; use a build-phase shell hook.

• nixpkgs' meson.hook/setup-hook.sh (mesonConfigurePhase, confirmed by reading the actual hook source in the locked nixpkgs tree at pkgs/by-name/me/meson/setup-hook.sh) runs runHook preConfigure FIRST, then builds its meson setup invocation via concatTo flagsArray mesonFlags mesonFlagsArray — so a Nix derivation's preConfigure shell-hook attribute can compute a value at BUILD time (inside the sandbox, via stdenv's own bundled coreutils — no extra nativeBuildInputs needed) and mesonFlagsArray+=("-Dfoo=$val") it, and that flag reaches the real meson setup command line, all within the same shell process (stdenv's generic builder runs every phase/hook in one continuous bash script, no subshells between them) — confirmed foo+=(...) on a variable never declared as a Nix attribute (so genuinely unset, not just empty) still works fine even under stdenv's set -eu (bash -c 'set -eu; foo+=("x"); echo "${foo[@]}"' succeeds, auto-vivifies as an indexed array). This is the general pattern for "a nix package needs a value only knowable at build time, not eval time" (e.g. build-time-versioning task 2.2's build_time, fixed via this exact mechanism in xce.nix/xcecli.nix after builtins.currentTime broke pure-eval — see the entry above). Live-verified: nix build .#xcecore_debug's build log (nix log <drv>) showed mesonConfigurePhase flags: ... -Dbuild_time=2026-08-20T09:13:29Z and meson's own summary printing build_time : 2026-08-20T09:13:29Z.

• nixpkgs.lib has no epoch-seconds→ISO8601 formatter (checked lib/*.nix in the locked nixpkgs source tree directly — no strings.nix/trivial.nix/etc. function does this). This is now moot for build_time specifically (see above — it's computed via shell date -u in a build-phase hook, not in Nix expression code at all), but still true in general for any future nix-side ISO8601 need.

• xce-build.sh defaults BUILD_DIR to the same path (xce.build) regardless of component, and its non---incremental setup_build() does rm -rf "$BUILD_DIR" + fresh meson setup on every invocation — two concurrent xce-build.sh runs for DIFFERENT components in the SAME working directory race on that shared path. (bug-1604, 2026-08-20). Live-confirmed twice in one session: (1) a scheduler asan run's initial rm -rf hit rm: cannot remove '.../controller_unit_test.p': Directory not empty because a second, independently-running xce-build.sh controller asan process (same cwd, different session) was actively writing into xce.build/ at that instant — simply retrying once the other process's own clean pass finished resolved it. (2) minutes later, the freshly-verified xce_scheduler binary (checked present + fresh mtime, --version run successfully) had VANISHED from src/xce.build/scheduler/ — the same concurrent controller build's own rm -rf had wiped and was regenerating xce.build/ for its component, silently invalidating the just-verified scheduler binary out from under the caller. Before trusting any non-incremental xce-build.sh binary/mtime to still be valid, or before starting a clean build in a worktree another session might also be using, check ps aux | grep xce-build.sh for a concurrent run — different components sharing one worktree's default xce.build is a real, reproducible race, not environment flakiness (this was initially misdiagnosed as an overlayfs artifact before the concurrent process was found).

• Live smoke-testing apiserver's -version/startup-log output (build-time-versioning task 4.4) needs a real topdir, but a full xce-deploy-ha deployment is overkill. Recipe: mkdir -p $TOPDIR/config $TOPDIR/logs then copy the four shipped templates src/frontend/apiserver/config/{app,logger,auth,rbac}.yaml into $TOPDIR/config/ verbatim, then timeout 10 xce_apiserver -d $TOPDIR. The shipped app.yaml template has a live backend: {address: localhost, port: 9901} block with no controller actually running there, so the process logs Attempting backend connection → WARN All configured backend candidates failed → ERROR Failed to connect to backend and exits 1 within ~10ms — this is fine and expected for verifying only the startup log's FIRST lines (logger init, before backend connect is even attempted); no live controller needed. Log lands at $TOPDIR/logs/apiserver_out.log (from logger.yaml's outputPath: apiserver_out.log, joined with $TOPDIR/logs/ by InitLogger()). Separately, xce_apiserver -version (no -d at all) exits 0 with zero filesystem side effects (no logs/ dir, no config read) — confirms the -version early-exit genuinely runs before PromptForTopDir()/InitLogger(), not just before the visible log line.

• This codebase's apiserver Go code has ONE idiomatic zap-logging style: structured fields via zap.String(...)/zap.Error(...) appended to the logger.Info/Error/Debug/Warn(msg, fields...) call, never Sprintf-style interpolated messages — confirmed by grepping every logger.Info/Error/Debug/Warn call site across controllers/, middleware/, router/ (files.go, auth.go, rbac_check.go, validatehandler.go, routers.go, permissions.go all exclusively use this pattern). Even though encoding: console is configured in logger.yaml, the fields still render as trailing JSON on the log line (already documented above) — that's fine, it's still the right convention to follow for any new log call site in this package, don't reach for fmt.Sprintf into the message string instead.

• gofmt -l/gofmt -d on apiserver/cmd/main.go reports a PRE-EXISTING import-ordering violation (_ "apiserver/docs" sorted before "apiserver/backend" instead of after, alphabetically) that predates build-time-versioning task 4.4 and is unrelated to it — confirmed via git show HEAD:.../main.go | gofmt -l - on the pre-edit blob, same violation. Don't "fix" it as a drive-by inside a task whose boundary is a single file with no mandate to touch unrelated pre-existing formatting — flag it, don't silently reformat, since an unrelated reformat in the same diff makes the real change harder to review (mirrors the existing Do-Not-Repeat entry about not running clang-format -i on whole existing C files).

• scheduler-sim-decision-parity task 1.2 (2026-08-22): white-box-including resource.c alongside the FULL driver+sim+recording stack (not just core.c, unlike test_resource_accounting.c's own deliberately-narrower pairing) works and needs exactly one NEW stub beyond what the driver/sim stack already required: sched_ctx_get_uncomit_host() (resource.c's sched_try_alloc_on_host_()'s own link-time dependency, never actually called when the test only drives the public sched_check_host_satisfy()/sched_charge_account()/sched_commit_job_alloc() entry points directly). Two stubs test_resource_accounting.c's OWN narrower resource.c-only pairing needs (plugin_prepare_allocation, sched_fill_limit_requests) must NOT be redefined once plugin.c is also in the TU (test_sim_equivalence.c's set already includes it) — plugin.c provides a REAL plugin_prepare_allocation, and resource.c itself defines the REAL sched_fill_limit_requests (not test_resource_accounting.c's stub version) — redefining either is a duplicate-symbol link error, not a harmless no-op. Also: sched_worker_release_host_ctx must NOT be stubbed once resource.c is included (every OTHER file in testsuite/ stubs it because none of them pull in resource.c, which defines it for real).

• A genuine, non-fabricated "capacity denied" fixture is achievable without driving scheduler.c's real sched_allocator()/resreq-DSL/plugin machinery at all: a test-owned "stand-in allocator" thread (the SAME established pattern test_sim_equivalence.c already uses to replace sched_allocator(), there for the different reason of avoiding real allocation entirely) can instead call the REAL sched_check_host_satisfy()/sched_charge_account()/sched_commit_job_alloc() (resource.c) directly against a real xce_sched_worker_t with a hand-set finite resvals capacity (added via sched_add_worker(), put_resvals() — same residx-direct-bypass-of-resmgr convention test_resource_accounting.c's make_test_account() already established) and a fixed per-job "wanted" lookup table standing in for resreq-DSL compilation. This produces genuinely-arithmetic-derived DISPATCH/PEND decisions (not fabricated) while staying at the same "don't pull in the untested resreq-DSL/plugin layer" scope every other white-box test in this directory already commits to. sched_check_host_satisfy(xce_scheduler_t*, ...) itself was NOT relocated by task 1.1 (only the accounting/charge functions were) — it still needs a real xce_scheduler_t*, not just a core, because unsatisfy_()'s TRACE line reads sched->core.resmgr (harmless when resmgr has no matching entry: the output pointer is pre-NULL'd and the TRACE format string degrades to "?", no crash).

• **scheduler-sim-decision-parity task 2.2 (2026-08-22): a new static function added to sim_loop.c but not yet wired into any call site (by design -- the wiring is a LATER task) triggers -Wunused-function at production-object compile time (ninja scheduler/xce_scheduler.p/src_sim_loop.c.o), but this project has no -Werror/warning_level override in src/meson.build (default meson warning_level=1, no werror), so it's a harmless warning, not a build failure -- confirmed by building that exact object directly. Don't mistake this for a real problem needing an early premature call site or a (void) no-op reference; just note it as an expected, temporary CONCERNS item until the wiring task lands. Also confirmed the standard "RED phase for a brand-new function with no prior runtime behavior" technique: cp the modified source aside, git checkout -- <file> to revert to HEAD (pre-implementation), rebuild+run the new test to capture a genuine implicit declaration of function compile-time RED failure, then restore the saved copy -- cleaner than hand-editing the diff back out and back in, and git-safe (never touches the actual uncommitted work beyond a temporary revert/restore pair on one file).

• scheduler-sim-decision-parity task 2.3 (2026-08-22): to prove an ASAN "no leak" claim for a white-box test in src/scheduler/testsuite/, do NOT trust the meson-registered test run alone — every test target in this directory is registered with env: ['ASAN_OPTIONS=detect_leaks=0'] (meson.build) because this whole test family deliberately never tears down its xce_sched_core_t/fixture allocations (documented file-header convention). Re-running the standalone binary with default ASAN_OPTIONS (leak detection ON) is the only way to actually see leak provenance, but doing so SWALLOWS Unity's own PASS/FAIL summary from stdout entirely (LeakSanitizer's report prints, then the process exits via a path that never flushes the earlier buffered Unity output) — so run it TWICE: once as ASAN_OPTIONS=detect_leaks=0 ./binary for Unity's real pass/fail summary (matches the project's own convention/exit code), once as a bare ./binary (or ASAN_OPTIONS=detect_leaks=1) purely to inspect leak stack traces. When triaging those traces, filter for ones whose stack passes through the code you actually changed (grep -B3 '<your_function>' ) — the other 100+ leak reports are the SAME pre-existing "fixture not torn down" noise on every test in the file (kvmgr/groupmgr/select-parser allocations from xce_sched_core_init()), not signal.

• free_host_account() (src/lib/misc.c) never frees acct->host (the strdup(halloc->host) make_host_account() sets, xcealloc.c:119) — confirmed via the above ASAN leak-provenance technique while verifying task 2.3's reconcile-then-release round trip. Pre-existing, NOT scheduler-sim-decision-parity-specific: equally leaks on the LIVE scheduler.c dispatch path (sched_commit_decision() → make_job_account() → same free_job_account()/free_host_account() release chain), previously masked only by this test suite's detect_leaks=0 convention. Logged as bug-1621, not fixed (out of boundary for task 2.3 — misc.c wasn't in that task's file set). Correct fix if picked up later: add FREEUP(acct->host); to free_host_account(), mirroring free_res_account()'s adjacent FREEUP(acct->resname);.

• scheduler-sim-decision-parity task 3 (2026-08-22): resource.c's get_resvals(zarray_t*, int nres, int*, float*) returns nres on success (not 0) — confirmed by reading its body (return nres; at the very end). Bit a new end-to-end test (test_sim_round_boundary_reconciliation.c) that wrongly asserted TEST_ASSERT_EQUAL_INT(0, rc) after a 1-resource query; briefly looked like the task-3 wiring hadn't worked (GREEN rebuild still failed) before the real cause (a test-authoring mistake, not a reconciliation bug) was found. Logged as bug-1622. Lesson: don't assume this codebase's near-universal "0/negative" return convention applies uniformly -- get_resvals() is a documented exception, check the actual body.

• scheduler-sim-decision-parity task 3 (2026-08-22): to exercise Recorded-Decision Reconciliation's REAL make_job_account()/make_res_account() (xcealloc.c) path end-to-end — as opposed to every existing resource-contention-style test in this directory, which hand-builds xce_job_account_t with residx already resolved, deliberately bypassing this — you must handle that make_res_account() (xcealloc.c) always sets residx = -1, forcing sched_update_res_accounts() (resource.c) to LAZILY resolve resname -> residx via key_find(core->resmgr, resname, ...) at charge time (falling back to auto-registering the name via add_vres() if not found). Don't gamble on which index that lazy auto-registration would land on (e.g. assuming a fresh resmgr's first-ever entry gets index 0) — instead, pre-register the resource explicitly via the real makeResource(core->resmgr, &xce_resource_t{...}) (resobj.c) BEFORE adding worker capacity, and read back the returned object's own ->idx to use consistently for both the worker's resvals/resInUse setup (put_resvals()) and every later sched_check_host_satisfy() wanted/unmeet vector. This is the only test file in src/scheduler/testsuite/ (as of this task) that exercises this lazy-resolution path at all — every sibling test that touches xce_res_account_t.residx sets it directly, sidestepping the question.

• In any script running under set -euo pipefail (e.g. xce-build.sh), a bare assignment whose RHS is solely a command substitution (fp="$(cmd)") is NOT exempt from -e — if cmd fails, that assignment's own exit status equals cmd's, and since a standalone assignment statement isn't a context -e treats as "tested" (unlike being the condition of if/while/&&/||/!), a failure there aborts the WHOLE script, not just that value's computation. Only if var="$(cmd)"; then ... fi (or cmd || true) is actually exempt. This bit dirty-content-fingerprint task 2.1's dirty_fingerprint computation (sha256 over git diff HEAD + sorted untracked file content) in xce-build.sh: every intermediate command (mktemp, git diff, the git ls-files | sort pipe, each cat in the untracked-file loop, and the final sha256sum | cut) had to be wrapped in its own if/elif ... else fp_ok=0 check rather than trusted to propagate failure naturally, specifically so a failure (e.g. sha256sum missing) degrades to an empty fingerprint (Requirement 5.1's "never fabricate") instead of crashing the entire build invocation. Verified live by temporarily hiding sha256sum from PATH in an isolated scratch repo — script printed dirty_fingerprint= and exited 0, not aborted.

• scheduler-sim-decision-parity task 4.2 (2026-08-22): a job that stays genuinely PEND (never dispatched, never state-changed) is re-offered to the live picker and re-PENDed EVERY subsequent round it remains unresolved, not just the round it first arrives in — builtin_fcfs.c's pendJobs discipline only removes a job on dispatch/state-change (its own comment: "A non-pending job is left removed and not reinserted"), so a still-PEND job simply stays in its queue's pendJobs and next_job() keeps offering it every round. Confirmed independently by test_repl_equivalence.c's own existing (untouched) fixture output: job 501 (pushed before session 1, never dispatched) recurs as a fresh PEND decision in sessions 2 AND 3 too. Bit a new multi-round --sim-verify regression test (test_sim_verify.c's test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatch): the round-2 fixture initially recorded only the round's one FRESH job's PEND, producing a real MISMATCH (expected vs actual position mismatch) because the still-PEND job from round 1 also reappeared, unrecorded, in the live-drained stream. Fix: any multi-round sim fixture with a job that stays PEND across round boundaries must include that job's OWN recurring PEND in every subsequent round's recorded decisions list, ordered by queue_time ascending (same-priority ties break by arrival order) to match next_job()'s real dequeue order. General lesson: don't assume a job "handled" in round N (PEND, not DISPATCH) is done being decided — only DISPATCH/state-change removes picker candidacy, so a genuinely-unresolved job is a recurring cost across every fixture round.

• scheduler-sim-decision-parity task 4.3 (2026-08-22, FINAL task): Recorded-Decision Reconciliation's SUCCESSFUL-MATCH path is structurally unreachable from ANY continuously, organically live-driven recording (real normal-mode OR simulated) — this is a hard mathematical fact about the design, not a limitation of test-authoring creativity, and it is worth knowing before attempting a "real, long, driven recording" fixture for this feature again. sim_reconcile_unaccounted_jobs() (sim_loop.c) matches a RUN-state, job->accounts==NULL job against a DISPATCH decision recorded in THAT SAME replayed round's own decoded ReplaySession only (each round's decoded_session is decoded fresh and freed immediately after that round's own processing — no historical carry-forward across rounds exists anywhere in the loop). But a job cannot be BOTH "freshly PEND when this round's live picker session starts" (the ONLY way any live picker/allocator, real or simulated, can ever produce a DISPATCH decision for it) AND "already RUN before this round's live picker session starts" (reconciliation's own trigger precondition) in the SAME round — builtin_fcfs_on_dispatch()/on_job_state_change() make these two states mutually exclusive by construction (dispatch permanently removes a job from pendJobs; a state-change to RUN removes it too, via the SAME on_job_state_change guard, whichever fires first in a batch). Confirmed BOTH by direct source read (sim_loop.c/driver.c/builtin_fcfs.c, before writing any test code) AND empirically, live: a test built with 11 "organic" jobs (arrive PEND, get denied/recur, get genuinely dispatched by a real capacity-aware allocator thread in their OWN arrival round, get confirmed RUN via a JOB_STATE_CHANGE event in a LATER round) showed EVERY ONE of those jobs' real dispatches permanently invisible during replay (job->accounts stays NULL forever, per Requirement 4's own documented graceful-degradation path) — only ONE job, deliberately engineered so its arrival+RUN-confirmation events AND its own "originating decision" all land in the exact same round (built via real constructors — make_charge_account/sched_commit_job_alloc/make_dispatch_decision — fed through the real core->decisions cbuf, NOT hand-crafted capnp bytes), ever showed nonzero resInUse during replay. Every EXISTING test in this feature that exercises reconciliation's success path (test_sim_round_boundary_reconciliation.c task 3, test_sim_contention_parity.c's "Part B" task 4.1) independently arrives at the SAME resolution (hand-construct the one round where the trigger and its matching decision coincide) for exactly this reason — it is the ONLY way to reach that code path at all. Lesson: if a future task needs "a job whose historical dispatch becomes visible via reconciliation," do NOT assume dispatch-now-confirm-a-few-rounds-later will work for ANY organically-arriving job — it structurally cannot, regardless of round count or job count; the trigger and its match must be engineered into the identical round, unavoidably requiring at least a minimally test-authored decision object for that one job/round. See buglog.json bug-1623 for the live test-authoring mistake this caused (initial checkpoint values assumed churn jobs' dispatches WOULD be reconciled; they never were).

• (2026-08-22) Normal-mode resource accounting does NOT survive a round boundary for a dispatched-but-not-yet-worker-confirmed job — a real over-commit/double-booking risk, distinct from and NOT fixed by scheduler-sim-eager-reconciliation. sched_commit_job_alloc() (scheduler.c:1446, called from sched_commit_decision()'s DISPATCH branch) synchronously charges job->accounts into core->sharedInUse/worker->resInUse at dispatch-decision time, regardless of job->state (still PEND — RUN only sets later via the separate schedStartJob notify path). But EVERY round, xce_core_begin_session() (core_session.c:66-108) unconditionally zeroes those same totals, and sched_account_jobs() (scheduler.c:1258-1292) only re-derives them from jobs where JOB_IS_RUN(job->state) is true (scheduler.c:1276) — silently skipping a still-PEND-but-dispatched job. Normal-mode rounds run on a 1-second cadence (main.c:453-460, while(1){ ...round...; sleep(1); }), so any dispatch-to-confirmation latency over ~1s (ordinary for real worker RPC+fork/exec) wipes that job's charge at the very next round and doesn't restore it until confirmation lands. The allocator's admission check (sched_check_host_satisfy_/sched_check_shared_satisfy_, resource.c:252-370, called from seqjob_allocator.c:130) reads exactly this same resInUse/sharedInUse — no separate reserved/committed pool exists (xce_sched_job_t only has state/allocs/accounts, sched/types.h:96-116) — so the resource looks free again and can be double-granted to a different job. Logged as bug-1624, NOT YET FIXED. scheduler-sim-eager-reconciliation's boundary explicitly excludes normal-mode behavior changes, so this needs its own spec.

• (2026-08-23) scheduler.c's sched_update()/sched_job_state_change() (the notify-driven job-state-apply path) is DEAD CODE in the live production driver — do not reuse or trace through it when designing scheduler changes. main.c's production loop calls only xce_sched_driver_run_normal_mode_round(), which drains controller-originated events via xce_sched_driver_translate_pending_events() (driver.c) → xce_core_push_events() → core_events.c's typed-event switch (e.g. XCE_CORE_EVENT_JOB_STATE_CHANGE) — a separate, independently-maintained mirror of scheduler.c's logic. handler.c's notify handlers still enqueue onto the same underlying core->pend_updates dataq both drain paths read from, so the ENQUEUE side is shared/live, but only core_events.c's DRAIN/APPLY side is ever actually invoked. Confirmed via direct trace of main.c's while(1) loop body (found during scheduler-dispatch-commit design). Any future scheduler design work touching state-change application must target core_events.c, not scheduler.c's sched_update().

• (2026-08-23) Iterative /kiro-validate-design loops (fix → re-review → repeat) keep finding NEW, escalating-severity issues well past round 1 — don't stop at the first clean-looking pass. For scheduler-dispatch-commit's design, round 1 found 3 issues (a locking claim that didn't match the code, a dead field, a requirements-wording mismatch); fixing those and re-reading surfaced a genuine double-notify race in round 2 (one function unconditionally sent the same notify type on both success and failure); fixing THAT surfaced an rc-value classification ambiguity + a missed 6th failure site in round 3-4; fixing THAT surfaced a 4-site reference-count leak (bug-1627) and an unconditional pre-failure-check commit in the same function; a final full re-read caught the WAL apply logic only covering one of two apply functions (apply_wal_entry() recovery path vs apply_committed_entry() live path). Each fix changed the code enough to expose a DIFFERENT class of bug the previous read hadn't looked for. Lesson: budget for 4-5+ rounds on a design touching genuinely concurrent/durable code, not 1-2 — and re-read the FULL file after each fix, not just the section just edited, since fixes ripple (count/terminology drift across sections was itself a repeat source of new-looking "issues" that were really just staleness from earlier edits).

• (2026-08-23) Weekly status reports are an established, continuous series in reports/, and their windows are defined by CONTINUITY, not by the calendar. Filenames are mmmDD-mmmDD-YYYY.md (lowercase month, zero-padded day, end date's year) but the ranges deliberately drift — jun29-jul04 (Mon–Sat), jul19-jul26 (8 days), aug08-aug15 (Sat–Sat) — because each report starts the day after the previous one ended and runs to the last completed day. Always ls reports/ and read the latest report's end date BEFORE picking a window; the naive "last complete ISO week" is usually wrong. Concrete case: on 2026-08-23 the ISO default resolved to Aug 10–15, but aug08-aug15-2026.md already covered it — the genuine gap was Aug 16–22, so the default would have written a duplicate while leaving a whole week unreported. The current stable format (last 3 reports) is: # XCE Project Status — <range> → ## TL;DR → ## Commits by day (per-day ### subsections each with a theme, not a commit dump) → ## Bugs opened/fixed this window → ## Spec status changes → ## Stats → ## Open threads / to-do heading into next session (numbered, carried forward and explicitly closed out in the next report). Two metric traps: quote non-.wolf line counts as the headline (.wolf/token-ledger.json churn made one week read +288,983 raw vs +28,657 real), and split chore(wolf) checkpoint commits out of the commit count (89 of 192 in one window).

• (2026-08-23) .wolf/buglog.json currently has 39 DUPLICATE IDs from a counter restart on Aug 15–17, 2026 — do not trust bug-NNN as a unique key, and do not infer "old bug re-touched" from a low ID with a recent timestamp. New entries were appended reusing the already-taken bug-875–bug-949 range; e.g. bug-908 is both a 2026-06-21 raft_sim.c guard-clause bug and a 2026-08-16 config-upgrade-snapshot/spec.json refactor note. When counting bugs for a report or analysis, split IDs at the prior window's max (genuinely-new vs collision) rather than trusting min/max. The Stop hook flags duplicates every turn until this is repaired, and repair needs a git-history-informed merge (identify the newcomer of each pair, renumber above the current max). This is the exact failure the OpenWolf bug-logging rule calls "unrecoverable without a git-history merge" — when appending, always compute the next ID as max(existing numeric IDs) + 1, never from a local or remembered counter.

• (2026-08-23, scheduler-dispatch-commit task 5.2) A design.md state-model claim of "no reversal needed on failure" can be silently scoped to only the FIRST occurrence of an event, and a later requirement in the SAME design can anticipate a repeat occurrence the claim doesn't cover — don't trust such a claim without checking it against every scenario the design's OWN other requirements construct. dispatch_a_job_()'s (job.c) four retryable-failure branches never write job.state/job.allocs, justified by design.md's "Controller Dispatch Commit" text: "on RPC failure, state is untouched (still PEND — never entered RUN, so no reversal is needed, per the Research Log finding)" — true ONLY for a job's first-ever dispatch attempt. Requirement 4.3's own text ("consecutive... failures with no intervening successful dispatch") and task 3.1's own observable-completion criterion ("a job that fails, succeeds, then fails again is evaluated against the bound as if it had only failed once") explicitly anticipate a job that has ALREADY succeeded once (RUN, allocs committed) failing again later — a scenario where "never entered RUN" is false, so "no reversal is needed" doesn't hold. Building a real, end-to-end fail→succeed→fail test (not a manual-field-poke unit test — those all vacuously start-and-stay-PEND, which is exactly why 4 EXISTING passing tests never caught this) exposed that job.state/job.allocs genuinely never revert on this path — logged as bug-1631 (unfixed; fix requires a job.c change, out of a test-only task's boundary). The counter-reset itself (Req 4.3, task 1.3/3.1) IS correctly implemented and IS proven end-to-end — this is a narrower, separate gap (Req 4.1/4.4/4.5) than the one the counter test was written to prove. General lesson: when a design document asserts an invariant with a specific justifying scenario ("never entered RUN"), check whether ANY other requirement in the same design describes a scenario that would falsify the justifying premise — don't assume the invariant holds universally just because the design states it as if it does.

• (2026-08-23) Technique: driving a real fail→succeed→fail sequence through a black-box test of a static dispatch function that never gates on the target object's own state — dispatch_a_job_() re-resolves the worker via a fresh key_find() on every call and never checks job.state itself, so toggling the SAME worker_t*'s ->ep field between a "broken" endpoint (task 3.3's dfr_setup_broken_ep() — real socket, unregistered in iomgr's epoll, xce_startjob_req() fails deterministically and repeatably) and a "working" one (task 3.1's zeroed/no-iomgr endpoint_t — enqueue always succeeds) between consecutive calls against the identical job/worker pair is sufficient to construct a real fail→succeed→fail cycle with no second fixture and no manual field pokes. Reusable pattern for any test needing multiple, alternating real outcomes from the same call site.

• (2026-08-23, scheduler-dispatch-commit task 5.4) controller_test.c's -t <group> filter operates at the GROUP-function level only (TEST(test_commit_barrier_reply) etc.), never at the individual RUN_TEST() case level inside a group's own entry function — if a hang (e.g. bug-1429) sits at an earlier RUN_TEST() call than the specific test you need to isolate, WITHIN THE SAME group function, -t <that-group> still hits the hang first and never reaches your test; there is no finer-grained filter in this harness. Confirmed live: test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs (bug-1429's usual hang site) is registered at test_commit_barrier_reply.c:3639, BEFORE test_jobdispatch_receipt_ack_no_wal_append at line 3649, inside the same test_commit_barrier_reply() function — so -t test_commit_barrier_reply cannot isolate the latter if the former hangs. In practice this session bug-1429 did NOT reproduce in 2 consecutive full unfiltered meson test --suite controller runs (both completed clean in ~38s, 1133 tests, only the 2 known bug-1561 failures) — reinforces that it's genuinely non-deterministic/timing-dependent, not something to route around by default; try a full unfiltered run FIRST (fast when it doesn't hang) before reaching for -t group-exclusion workarounds.

• (2026-08-23, scheduler-dispatch-commit task 5.4) bug-1598's "never-joined background thread + stack-local test fixture" ASAN stack-use-after-return pattern (sim_picker_thread_fn in sim_loop.c, reading a stack-local xce_sched_sim_driver_t after its owning test frame returns/is reused) has an UNFIXED sibling instance in test_sim_verify.c's own separately-defined run_sim_and_capture_stdout() helper — bug-1598's fix only converted the 3 stack-local fixtures in test_sim_event_loop.c; any OTHER file with its own stack-local xce_sched_sim_driver_t sim; calling sched_sim_driver_run()/sched_sim_driver_start() still carries the identical latent bug, non-deterministically triggered by whatever stack-layout/MALLOC_PERTURB_ randomization a given test invocation happens to produce (fired once in a /xce-build scheduler run this session, then didn't reproduce across 3 standalone re-runs + 1 more full-suite re-run). Logged as bug-1632. Before trusting "the scheduler suite is at its known N/52 baseline," grep for other stack-local xce_sched_sim_driver_t fixtures beyond the ones bug-1598 already fixed — this class of bug is per-file, not fixed globally by one file's remediation.

• (2026-08-25, bug-1685) key_add() on a kvmgr (src/lib/kvmgr.c's xkvmgr_add()) that was created with a non-NULL .datakiller in its kvmgr_params_t ALREADY frees a rejected value for you on failure — the caller must NOT also free it. xkvmgr_add() calls vstore(kvm->data, value) first (stores into the backing zarray), and only THEN key_add(kvm->kvtab, key, idx); if that second call fails (e.g. -EEXIST for a duplicate key), it rolls back via vclear(kvm->data, idx) -> xzarray_clear() (src/lib/zarray.c), which — since the slot is in-use — fires a->killer(a->data[idx].data) on the just-stored value automatically. A caller whose kvmgr has .datakiller set (e.g. limit.c's privdata->limits, .datakiller = kill_limit) that ALSO calls its own free function on key_add() failure double-frees — this was bug-1685's exact live SIGABRT (glibc malloc_printerr). Conversely, a kvmgr created with .datakiller = NULL (e.g. job.c's c->jobmgr, per restore_job_array()'s own doc comment) has a no-op rollback, so THAT caller correctly must free the rejected value itself. Rule: before writing/reviewing any key_add() error-handling path, check the target kvmgr's OWN kvmgr_params_t.datakiller at its construction site — that single field, not intuition about "who owns this pointer," determines whether the caller must free on failure. This is the second confirmed real bug in this exact shape (job.c's restore_job_array() comment documents the mirror case) — worth grepping for every other key_add(...) call in the codebase whose target kvmgr has a non-NULL datakiller and confirming its own error path doesn't also free.

• (2026-08-25, bug-1686) A lock that correctly protects a data structure DURING a single critical section does NOT protect a raw pointer CACHED across SEPARATE lock acquisitions from going stale if freed by another properly-locked call in between — mutual exclusion prevents concurrent corruption, not cross-call pointer lifetime bugs. builtin_fcfs.c's fcfs_plugin_data_t.lock (the bug-1611-era concurrency fix) makes every individual hook call — on_queue_update(), next_job(), etc — atomic with respect to each other; that part works. But priv->cursor/priv->job_cursor are raw ordq_item_t* values that PERSIST in fcfs_plugin_data_t across separate next_job() calls, each taking and releasing the lock independently. A later, individually well-locked on_queue_update() call (removing-then-reinserting a queue's node to re-sort it — ordq_get + FREEUP(it) + ordq_put) can legitimately free the exact node a cursor from an EARLIER, already-completed next_job() call is still resting on; the very next next_job() call then dereferences that freed node — a genuine heap-use-after-free, no concurrency/threading required to trigger it (single-threaded, sequential calls reproduce it deterministically, confirmed live under ASAN). Fixed by NULLing whichever cursor points at a node right before that node is freed, at every site that frees/removes a node the cursor might reference (on_queue_update's reuse branch, on_job_state_change's !JOB_IS_PEND removal, fcfs_remove_job_everywhere) — safe because next_job() already treats a NULL cursor as "exhausted" and begin_session() resets both fresh every round, so invalidation costs at most a one-round delay, never a lost job or a crash. Rule: any struct field that is a raw pointer INTO a lock-protected structure, and that OUTLIVES the single critical section that produced it (i.e., read under one lock acquisition, dereferenced under a LATER, separate one), is a latent stale-pointer bug regardless of how correctly the lock itself is used — audit for "cache a pointer, release the lock, reacquire the lock later, dereference the cached pointer" as its own category, distinct from and not fixed by the mutex being correctly held everywhere. Same file as bug-1611's original concurrency fix (builtin_fcfs.c) — worth re-checking any other "cursor"/"cached node"-shaped field in this codebase for the identical cross-call pattern.

• (2026-08-25, bug-1687) This worktree can have MULTIPLE concurrent agent sessions actively editing it at once — always re-check .wolf/buglog.json's max id and git status for files you never touched immediately before writing, not just once at session start. Mid-session, .wolf/buglog.json's max id moved from 1685 to 1686 (a different session's builtin_fcfs.c SIGSEGV fix, bug-1686) between my first check and my write — required a bulk sed -i 's/bug-1686/bug-1687/g' across 6 already-edited files (job.c, persist.c, persist.h, wal_types.h, the new test file, and a test-fixture fix) to renumber every in-code comment reference before writing the buglog entry. Also: git status mid-session showed src/scheduler/src/plugins/builtin_fcfs.c/.wolf/cerebrum.md/.wolf/buglog.json as modified by that OTHER session, and a full top-level ninja/meson build can fail on THEIR uncommitted, unrelated changes even when your own target component builds fine — that other session's own bug-1686 entry documents hitting the mirror image of this same collision (my job.c/persist.c changes broke ITS full-tree ninja build). Never assume .wolf/* files or the working tree are exclusively yours in this environment; re-verify immediately before any write that depends on "current state" (next bug id, clean baseline, etc), and scope builds/tests to your own component when a full-tree build fails on files you didn't touch.

• **(2026-08-25, bug-1687) When a WAL/wire fact type's payload grows past a previously-fixed size (e.g. adding trailing bytes to what used to be a bare fixed-size struct — the bug-1678/bug-1687 pattern), grep the WHOLE testsuite for any test helper that manually parses the raw WAL segment file and matches entries by an EXACT payload-length equality (hdr.payload_len == sizeof(fixed_struct)) — such a helper will silently start UNDER-COUNTING (returning 0 instead of the real count) the instant the payload legitimately grows, and if any OTHER test polls/waits on that count reaching a value, the undercounting can manifest as a genuine, reproducible DEADLOCK far downstream (all threads futex-blocked, near-zero CPU accrual) rather than a clean assertion failure at the actual call site. Found via cbrp_count_jobstate_entries_ex() in test_commit_barrier_reply.c (hdr.payload_len == sizeof(wal_jobstate_t) — fixed to >=, since jobid/state always live at the same fixed leading offset regardless of trailing bytes). This is a DIFFERENT, more dangerous failure mode than bug-1678 ever hit (that fix's own tests never counted WAL_JOB_DISPATCH via this exact helper, so it went unnoticed) — always grep for sizeof(wal_jobstate_t)-style exact-match parsers project-wide whenever changing a WAL fact's payload shape, don't just trust the direct callers/apply seams to be the whole blast radius.

• (2026-08-25, bug-1689) The bug-1687 apply-seam pattern (split a shared-payload WAL fact type out of apply_wal_entry()'s shared case group into its own case, extend apply_committed_entry()'s existing inline else if (wal_type == X) branch, add a restore_job_<type>_state() sibling in job.c, add a persist_submit_job_<type>_fact() sibling in persist.c/persist.h) generalizes cleanly to a THIRD job-fact type (WAL_JOB_FINISH) with one twist: not every such fact needs Cap'n Proto. job_exitinfo_t (xce/types.h) is a single {int exitcode;} with no existing capnp mapping (confirmed via grep — no JobExitInfo in controller.capnp), so the trailing payload was a small NATIVE (raw memcpy) fixed-size trailer instead of a capnp-encoded blob — no malloc needed either, since the whole payload is fixed-size (unlike WAL_JOB_START's variable-length JobPIDInfo or WAL_JOB_DISPATCH's JobAlloc). General rule going forward: before reaching for capnp on a new WAL trailing payload, check whether the struct being carried already has a capnp mapping AND has any variable-length/nested-pointer fields — if it's a small fixed POD with neither, a raw memcpy'd native trailer (mirroring wal_jobstate_t's own style) is simpler and correct, matching this codebase's own "small native trailing struct" precedent for wal_jobstate_t itself. Also discovered mid-fix: apply_job_finish_transition() (the "richer" core that DOES set exit_info/finish_time/run_time correctly) is effectively DEAD CODE on the real production apply path today — raft-apply-decoupling's redesign routes all real WAL_JOB_FINISH mutation through the leaner apply_job_update_transition() (state-only) instead, so exit_info was never populated through the live commit-apply pipeline at all, restart or not, prior to this fix (worse than "only breaks on restart"). When auditing a "richer transition function exists but the apply seam uses a leaner one" pattern (already documented for WAL_JOB_START in bug-1687's persist.h comment), always check whether the richer function has ANY live caller at all, not just whether it's correct.

• (2026-08-25, bug-1689) Reconfirms the bug-1687 "multiple concurrent sessions write .wolf/buglog.json" lesson with a fresh, independent collision — re-check the max id immediately before EVERY write, every time, not just once per session. Checked max id at session start (1687, free), implemented the full fix using "bug-1688" throughout code comments/test file, then re-checked right before writing the buglog entry (per this project's own mandated protocol) and found bug-1688 had been claimed by a sibling session's unrelated scheduler REPL bug in the interim — required a sed -i 's/bug-1688/bug-1689/g' across all 5 touched files before writing the entry as bug-1689. Zero cost when caught by re-checking as instructed; would have been a duplicate-id buglog corruption if skipped.

• (2026-08-25, bug-1687) ptrace is fully unavailable in this sandbox — gdb -p <pid> fails with "Operation not permitted" even with dangerouslyDisableSandbox: true, and /proc/<pid>/task/*/stack is also unreadable (no permission). The only process-introspection signal available without ptrace is /proc/<pid>/task/*/{wchan,status,comm} (shows which kernel function a thread is blocked in, e.g. futex_do_wait, and its state, e.g. S (sleeping)) plus ps -o pid,etime,time (wall-clock ELAPSED vs. CPU TIME — a process with large ELAPSED but near-zero TIME is genuinely blocked/deadlocked, not slow-computing; this distinction is what first proved the bug-1687 hang was a real deadlock and not just a slow test run). To actually isolate WHICH test in a monolithic Unity binary is hanging when -t <group> only filters at group-function granularity (see the existing entry on this): temporarily insert an early return; inside the group's own driver function (e.g. test_commit_barrier_reply()) right after a candidate RUN_TEST() call, rebuild+run with a bounded timeout N, and binary-search by moving the return; — each step is a full, clean process exit (no ptrace needed) so the log's last line IS trustworthy for that step (unlike a genuinely-hung run, where stdio is fully-buffered when redirected to a file/pipe and the last visible log line can be UP TO ~4KB of buffered-but-unflushed output stale relative to where the hang actually occurred — do not trust "last line before a real hang" as the hanging test's identity without this bisection). After finishing this kind of diagnostic bisection, grep the group's own driver function for accidentally-DUPLICATED RUN_TEST() calls before considering the file clean — inserting new RUN_TEST() lines ahead of a temporary return; marker and then later deleting only the marker (not the inserted lines) leaves the original, further-down calls to the same tests still present, silently double-running them (caught here via git diff showing a nonzero +RUN_TEST/-RUN_TEST net and a suspiciously-inflated total test count).

• (2026-08-25, bug-1691) /tmp in this shared sandbox can silently fill to 0 bytes free across many unrelated sessions, because mkdtemp()-based test fixtures across this whole codebase only rm -rf their scratch dir on the NORMAL end-of-test path — any run aborted mid-way by ENOSPC (or a crash, or a killed process) leaves its 64-128MB scratch dir behind forever, compounding: fuller disk -> more runs abort early -> more orphans -> fuller disk still. Found live: a build+test run failed with OSError: [Errno 28] No space left on device, both from meson's own JSON/XML report writer AND, worse, from genuine mid-test mkdtemp failed/spurious Expected Non-NULL failures in UNRELATED test files once disk hit 0 during actual test execution — that second failure mode produces a CORRUPTED, untrustworthy pass/fail signal, not just a build-tool crash, so a run showing many unrelated failures right after an ENOSPC condition should be re-run after fixing disk, never trusted as real regression evidence. Root cause found via ls -1 /tmp | grep -c '^xce_': 15,437 orphaned /tmp/xce_* scratch directories (~658GB) had accumulated from prior sessions. Verified none were held open by any live process first (for p in /proc/[0-9]*; do ls -la "$p/fd" | grep <pattern>; done — empty output = safe to delete; a live xce-deploy-* HA cluster's OWN workdir, e.g. /tmp/xce-deploy-1679, DOES show up in a live process's cmdline/cwd and must NOT be deleted). Fix: rm -rf /tmp/xce_* recovered 658GB instantly (359MB free -> 659GB free). Whenever a build/test run in this sandbox fails with ENOSPC or df -h / shows near-0 available, check for this exact pattern before assuming it's a real disk-capacity problem or someone else's live process — it is very likely safe, accumulated test-fixture debris, not a resource genuinely in use.

• (2026-08-25, bug-1691) A shared-fixture footgun found copy-pasting a sibling WAL-fact test file's "drive the real production entry point through a real restart cycle" test: persist_init(&c.persist, &cfg, NULL) (the exact line every prior sibling test — bug-1687/1689's START/FINISH suites — uses) is WRONG for a hook-driven fact type, even though it is byte-identical to what those working sibling tests do. persist_start() only registers persist_job_hook_fun/persist_queue_hook_fun/persist_worker_hook_fun on p->cluster->hooksys when p->cluster != NULL (persist.c). DISPATCH/START/FINISH's production functions call persist_submit_job_*_fact() DIRECTLY, bypassing the hook system entirely, so those sibling tests' NULL 3rd-arg is harmless. But CKPT (like bug-1690's RUSAGE) is HOOK-DRIVEN: on_job_ckpt() only ever calls xce_run_hooks(..., HOOK_DETAIL_JOB_CKPT, ...) and relies ENTIRELY on persist_job_hook_fun being a registered subscriber to reach the WAL at all. With NULL passed, the real-call test silently appends NOTHING (no crash — xce_run_hooks on an unregistered detail is simply a no-op), so the restart-replay assertion fails with "ckptinfo NULL" even though the fix code itself was already fully correct — a genuine RED-signal red herring that looks exactly like a fix defect. Diagnosed by noticing the 6 hand-built-payload tests all passed (proving the apply-seam decode/adopt logic itself was right) while ONLY the real-production-call test failed, then re-reading persist_start()'s actual gating condition rather than assuming the copied fixture was correct because it was copied from a passing sibling. General rule: before copying a sibling WAL-fact test file's "drive the real production function" fixture, check whether YOUR fact type's producer calls persist_submit_job_*_fact() directly (safe to pass NULL) or is purely hook-driven via xce_run_hooks() (MUST pass a real cluster_t* to persist_init()'s 3rd arg so persist_start() registers the hook) — grep the producer function's own body, don't assume parity from a structurally-similar sibling test.

• (2026-08-25, bug-1691) controller_unit_test (not just xce_test, extending the existing entry on this) also loses ALL buffered stdout (both PASS and FAIL lines) when run directly with output redirected to a file/pipe AND ASAN_OPTIONS's default detect_leaks=1 finds leaks — LeakSanitizer's atexit-registered check appears to bypass libc's normal exit()-time stdio flush, so a process that ran 7/7 tests successfully to completion produced a log with ONE clean LeakSanitizer summary and ZERO printed PASS/FAIL text anywhere, first reading as "harness produced no output, something is badly wrong" before realizing it was a buffering artifact. stdbuf -oL does NOT fix it (worse: it breaks ASAN's LD_PRELOAD-ordering requirement, producing ASan runtime does not come first in initial library list) — the correct fix is ASAN_OPTIONS=detect_leaks=0 ./controller_unit_test -t <group> when reading stdout directly (matches meson's own test-harness invocation, which already sets this — the SAME binary run through meson test shows real PASS/FAIL text fine). The leaks themselves (spy-fixture jobs created via makeJob() into a jobmgr with no .datakiller, never individually torn down) are PRE-EXISTING and IDENTICAL across every sibling test_wal_job_*_fact.c suite (confirmed: test_wal_job_rusage_fact in isolation shows 78 leaked allocations too) — not a new leak introduced by any one fix, not worth chasing.

• WAL field-parity fix family (bug-1678/1687/1689/1690/1691/1692, controller persist.c/job.c): when a hook-driven WAL fact (payload built inside persist_job_hook_fun, not a direct persist_submit_job_*_fact() call site) needs a new field added to survive replay: (1) check whether the field's own release function uses FREEUP()/free() (plain malloc/strdup/calloc — safe to free-and-replace, e.g. JobAlloc/JobPIDInfo/JobPendReason) vs dereference() (xstring/RTTI-backed, e.g. ckptinfo) vs "can be the SAME live object the apply pump re-applies moments later" (needs a MERGE, not free-and-replace — only WAL_JOB_RUSAGE's run_usage needed this, because update_host_runusage() mutates it in place before the WAL fact is even appended); (2) check whether adoption must be gated on JOB_TRANSITION_APPLIED or unconditional — if the setter never changes job.state (true for CKPT/RUSAGE/PREASON, false for START/FINISH/DISPATCH), gating makes the fix dead code since every real replay hits JOB_TRANSITION_NOOP_DUPLICATE; (3) check whether the field has a genuine CLEAR-to-NULL operation (only pend_reasons/do_clr_preason in this family so far) — if so, "trailing bytes present or absent" (bug-1690/1691's convention) cannot distinguish a legitimate clear from an old-format payload; needs an explicit presence byte instead. Always re-verify (2) and (3) from the real setter code, never assume the newest precedent's shape applies unchanged.

• capnp-c's generated decode path does not validate content, only bounds via capn_getp/capn_resolve: passing well-formed-length-but-semantically-garbage bytes to a decode_X()/read_X() pair (e.g. decode_JobPendReason) does NOT return NULL or error — it silently produces a zero-valued struct. A "decode failure returns NULL" defensive branch (decode_X() == NULL) is only reachable via allocation failure or an upstream length/bounds guard (e.g. payload_len too short to even attempt the read) — never via corrupt-but-present bytes. Don't write a regression test assuming garbadge bytes will fail decode; it will silently pass through and the test either false-passes or fails for a confusing, misleading reason (Expected 4 Was 0 — the "corrupted" value decoded as an all-zero struct, not a NULL).

Do-Not-Repeat

• When translating a fp_ok-tracked, per-step-gated failure-detection pattern (like xce-build.sh's dirty-fingerprint computation) into a Make $(shell sh -c '...') one-liner, do NOT collapse two commands into { cmd1; cmd2; } >"$$fp_tmp" and gate the whole group with one if — a POSIX sequential group's exit status reflects only its LAST command, so cmd1's failure is silently discarded whenever cmd2 still succeeds (2026-08-21, bug-1612: git diff HEAD failing was swallowed because the trailing git ls-files | sort | xargs cat pipe still exited 0, producing a fabricated fingerprint instead of degrading to empty). Give each step its own explicit if ! step; then fp_ok=0; fi (or &&-chain), exactly mirroring the reference script's discipline, even inside a cramped one-liner. Caught by adversarial code review with a targeted failure-injection test (wrapper git on PATH that fails only the targeted subcommand) — that test pattern is the reusable way to verify this class of fix.

• Don't run clang-format -i on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via git checkout HEAD -- <file> on a file with other uncommitted work destroys that work irrecoverably.

• (2026-08-22, scheduler-sim-decision-parity task 2.3) Actually ran clang-format -i on two files mid-task despite this rule already being logged above — caught it via git diff before finishing, not before running it. If this happens again: do NOT git checkout the file (destroys real work). Instead: git diff -- <file> and read every hunk; any hunk whose ONLY change is whitespace/line-wrapping on code you did NOT intentionally write/edit this turn is collateral reformatting — manually Edit just that hunk back to its pre-format text (find each such hunk, revert by hand). Verify success with git diff -- <file> | grep -c '^-[^-]' on regions you didn't touch dropping to 0, then rebuild+retest to confirm nothing broke. Cheap enough to be a non-event once caught, but the real fix is: check this Do-Not-Repeat list (or just never run clang-format -i without --dry-run/-output-replacements-xml first to preview scope) BEFORE running it, not after.

• Before citing a bug as "still open" or a spec as "blocked," cross-check .wolf/buglog.json's actual fix status and the spec's own tasks.md/commit history — blocked_on notes and prior narrative claims (including this file's own past entries) can be stale. Self-caught recurrence (2026-08-19): while reconciling OTHER stale doc references, wrote a new tasks.md status line asserting "tasks 2.1-2.6 still carry no _Verified: annotations" by trusting a 2026-08-07 validation finding's own wording verbatim, without re-checking the file — they'd actually been backfilled the very next day (commit d7b49690, 2026-08-08). A dispatched verification agent caught it via git log/direct read. Lesson holds even for a claim you are ABOUT to write, not just one you're citing from memory: grep/read the actual current state of the specific lines before asserting their status, every time, no exceptions for "this one's probably still true."

• Before trusting an "all tests pass, GO" claim on a stale/uncommitted tree, re-run the suite yourself — a buglog "FIXED" note can describe the intended fix while the code on disk is mid-revert from RED-phase construction.

• A generic retry/failover classifier that judges retriable-vs-terminal by a raw return value's magnitude (comparing against known errno values) is unsafe unless every wrapped call site shares the same success convention — a call that legitimately returns a variable count on success can numerically collide with an error code.

• When two independent "is it ready" signals exist server-side, a wait/convergence helper gating on only one does not imply the other is satisfied — check what the actual raced precondition depends on.

• Nix fixed-output-derivation hash mismatches from an upstream input pinning a mutable branch: re-locking flake.lock is a dead end once already at the latest commit; fix via a local overrideAttrs pinning src to an explicit immutable commit in the consuming repo.

• pkill/pgrep by bare process name (unscoped) can kill or match a different concurrent invocation's processes, including a sibling session's build/soak — always scope process matching to something unique to the current run (e.g. a workdir path).

• Stacked/masking bugs in never-before-reached code paths: when a bug prevents a code path from EVER executing, fixing it can immediately expose 1+ MORE bugs that were always there but never triggered. bug-1569 thread B was "no job ever gets scheduled" -- fixing cause #1 (scheduler.c's sched_init_queues() never calling picker_iface->on_queue_update() for builtin_fcfs, so its candidate registry stayed empty forever) let a job reach the allocator for the first time ever, which immediately SIGSEGV'd on cause #2 (core_events.c's core_build_job() never constructing job->schedobj, a documented-but-wrong assumption from the scheduler-core rewrite). Fixing #2 let a job reach genuine dispatch+resource-charge accounting for the first time ever, which immediately SIGSEGV'd on cause #3 (resource.c's put_resvals() had an off-by-one, idxs[i] <= 0 treating resource idx 0 -- "slots" -- as an invalid sentinel). Lesson: after fixing a "nothing ever happens" bug, don't declare victory on the first crash/error that surfaces -- it may be the SECOND (or third) bug the first one was masking, not a regression from your own fix. Verify via gdb bt full on the actual live crash, not guesswork from reading code alone.

• xce_scheduler -i/--sim (the interactive REPL) starts PAUSED at the initial snapshot and never advances the simulated clock past a recorded ReplayEvent/ReplaySession until you send an explicit step [N] or continue. Piping "info worker <name>\nquit" with no step/continue first only ever shows the pre-replay baseline state — this cost a large chunk of bug-1569 investigation, wrongly appearing to show a second "periodic resources still 0.00 after the host-mismatch fix" bug when the fix was already fully correct and sufficient; step N (N large enough to pass the relevant event, confirm via xceadmin scheduler dump <file> first if unsure of event ordering) revealed the true, correct values. Before concluding a replay-derived state is wrong, always step/continue first.

• A dead #if 0 /*FIXME*/ ... #endif block is a signal the disabled code is unsafe as written, not merely "not yet wired in" — don't just uncomment it. bug-1578: sched_worker_change()'s disabled block called free_name_val_list(&w->host.resvals_to_sched, &w->host.nresvals_to_sched) right before xlist_fini(&resvalist) — but resvalist.data and w->host.resvals_to_sched are the SAME pointer (the earlier line aliased one to the other's backing array), and free_name_val_list() already frees that array itself; uncommenting as-is would double-free. The safe fix null-and-reset the aliased fields directly (w->host.resvals_to_sched = NULL; w->host.nresvals_to_sched = 0;) and let xlist_fini() be the sole owner of the free — the pattern already proven correct in the sibling apiserver_worker_change() (apiserver.c). General lesson: when re-enabling dead code, first work out WHY it was disabled (git blame + read what it actually does to shared/aliased memory), don't assume #if 0 means "harmless, just needs a flip." Other call sites still doing free_name_val_list(&(w->resvals_to_sched), ...) on this same field (dataex.c:435, opget.c:128, reply.c:66, apiserver.c:51) were NOT audited for the identical aliasing hazard as part of this fix — flagged, not yet checked.

• xce_test's !<shell-command> escape (preprocess_and_run(), single call site in main()'s top-level REPL loop, test.c ~line 950-998, backed by a plain blocking system()) works IDENTICALLY in every REPL context, including worker — no leave_ctx()/disconnect happens, unlike moving between contexts via ../cmd_up. This is the general pattern for "run an out-of-band check while a stateful REPL session must stay connected": bake a !-prefixed line into the script fed to the session (before its quit) that shells out to ANOTHER invocation of your own driver script in a hidden internal mode (e.g. ! bash "$0" --internal-<mode> <args>), intercepted via a dispatch check placed BEFORE the normal arg-parsing entry point at the very bottom of the file. Used to fix bug-1579 (resource-value-verification): the controller-side resval check now runs nested inside the injecting worker session's own ! escape instead of as a separate xce_test process launched after quit already disconnected the worker (which reset host-attrs resources to baseline first, an always-false MISMATCH). Gotchas: (1) the nested process must generally return exit 0 even on partial/expected failure conditions — a nonzero exit becomes the outer session's g_last_status/$?, and in non-interactive/batch mode preprocess_and_run() ABORTS the REPL loop before any later command (including quit) runs; surface real failures via a sentinel/marker file the OUTER script checks AFTER the whole session completes, not via the nested process's own exit code. (2) system()'s child inherits the invoking process's environment but NOT its parent bash's local (non-exported) shell variables — bake every value the nested invocation needs (paths, host lists, names) as literal, printf '%q'-quoted arguments directly into the ! line's command string, don't rely on env-var passthrough through the nix develop/ASAN-wrapped intermediate hops. (3) since accept/reject-style bookkeeping computed by the REPL session itself is only visible as TEXT PRINTED to that session's own stdout — not accessible to a freshly-spawned nested process — have the nested step collect RAW, unfiltered data for every candidate (not just ones later known to be "accepted") and defer accept/reject filtering to the outer script, after it has parsed the whole session's captured output post-quit. (4) live-verified no orphan/zombie processes result even when a SIGTERM interrupts the outer driver script WHILE the nested !-escape's own sub-session is actively running — bash defers a pending trap until the current foreground command (the whole piped xce_test invocation, nested check included) returns, so the child process tree either completes and exits cleanly on its own or the trap fires right after; confirmed via a full /proc/[0-9]*/cmdline scan, not just pgrep.

• Piping scripted commands into xce_test (the controller REPL/testsuite binary) with default ASAN_OPTIONS loses ALL stdout output when redirected to a file/pipe. xce_test is an ASAN build; detect_leaks=1 is the default, and this binary has small pre-existing intentional leaks (e.g. set_resval's calloc, cmd_resupdate's strdup for resname — never freed by design, REPL-lifetime scratch data). LeakSanitizer's exit-time report appears to bypass/precede the normal stdio flush when stdout is non-interactive (fully block-buffered), so a redirected-output smoke test showed ONLY the ASAN leak report and none of the actual REPL command output (join/resupdate/resval results), even though the commands genuinely executed (confirmed via the leak stack trace itself). Fix: always run xce_test with ASAN_OPTIONS=detect_leaks=0 (matching the meson test harness's own invocation, which already sets this) whenever piping/redirecting its stdout for a live smoke test — e.g. nix develop ./src --command env ASAN_OPTIONS=detect_leaks=0 xce_test < cmds.txt.

• A fork/subagent dispatched in a prior session that claimed "build/tests in progress, will report back" is NOT reachable in a new session — ListAgents will not show it, and its self-report (if logged to memory before the session ended) describes an intent, not a completed result. 2026-08-19: a fork agent from the previous session was recorded as "actively writing tests... then building via xce-build skill and reporting final metrics," but the session ended before it reported. On resuming ("continue"), the scheduler test binaries in src/xce.build were still dated Aug 18 14:27 while the source edits were Aug 19 09:18-09:27 — proof the claimed build never actually happened in this environment. Always verify via binary/source stat mtime comparison (or just re-run the build) before trusting a cross-session "in progress"/"dispatched" claim, same discipline as the existing "stale/uncommitted tree GO claim" entry above but for cross-session fork continuity specifically, not just tree staleness.

• Go's -ldflags -X importpath.name=value is silently a no-op (not a build error, not even a printed warning) when the target main package has no such package-level string var declared — confirmed live (build-time-versioning task 4.1, 2026-08-20): built xcecli/cmd/main.go (which does NOT yet declare GitCommit/BuildTime, those land in task 4.4/4.5) with -ldflags "-X main.GitCommit=abc123 -X main.BuildTime=..." via both a direct go_build invocation and the full xce-build.sh xcecli meson path — exit 0 both times, working ELF binary produced, grep -i "warning\|no such" on the full captured output found nothing beyond an unrelated nix git-dirty-tree line. This means task 4.1 (the -ldflags injection in go_build) has NO real ordering dependency on tasks 4.4/4.5 (the consuming vars) — safe to land and validate standalone, contrary to what a cautious reading of Go linker docs might suggest.

• A REPL/replay test fixture that reuses the SAME jobid for both a "real" event (job push) and a "cheap padding" event (job state change, added purely to exercise a second event type) accidentally mutates the real job's state, not just decorates the log. bug-1595 test regression (task 12.5, 2026-08-19): test_step_echoes_events_before_session_decisions pushed job 6161 then sent a JOBSTATECHANGE event for the SAME jobid 6161 (PEND->RUN) purely to get a second event type echoed — this genuinely transitioned the real job to RUNNING before the session ran, so the picker's next_job() never offered it and the session's own PEND decision (which the test also asserted on) was never produced. The sibling test test_event_breakpoint_pauses_repl_at_matching_event already documents the correct pattern in its own comment (append_jobstatechange_event() establishes: core_events.c WARN+skips an unknown jobid rather than fail, so ... below needs no real job to exist) — it deliberately used unrelated jobids (111/222) precisely to keep the padding event inert. When adding a "just need any event of type X to be logged" filler event to a fixture that also has a real job in flight, always target an unrelated/nonexistent jobid unless the state transition itself is the thing under test.

• [ -n "${arr[key]+x}" ] (associative-array key-EXISTENCE) is NOT a safe proxy for "the value is meaningful/reachable" when the producer unconditionally sets that key even on failure. bug-1582 (resource-value-verification task 7.1, round-1 REJECTED, round-2 fixed): main()'s record-building loop derived sched_ok/api_ok from whether SCHEDULER_OBSERVED[$vname]/APISERVER_OBSERVED[$vname] existed — but query_scheduler_resvals()/query_apiserver_resvals() (already-approved tasks 6.3/6.4) always set that key, setting it to "" on a genuine UNREACHABLE/timeout rather than leaving it unset. Key-existence was therefore always true, silently masking every real UNREACHABLE as a MISMATCH (comparing the injected value against an empty string). Fix: derive the "is this layer reachable" signal from the layer's own pre-classified status string (SCHEDULER_CHECK_RESULTS[$vname]/APISERVER_CHECK_RESULTS[$vname], which DOES distinguish UNREACHABLE:* from MATCH/MISMATCH:*) instead of from a sibling data array's key presence. General lesson: before using ${arr[k]+x}/-v arr[k] as a "was this checked / did this succeed" signal, verify the producer actually leaves the key UNSET on failure — if it sets it to an empty/placeholder value instead (a common pattern for "always populate every requested key so downstream code doesn't need existence-checks"), key-existence collapses to "was this key in the requested set" a wholly different, weaker signal than "did this succeed". Also: a synthetic test that only calls the pure aggregation function directly with hand-built records (task 8.2's original approach) cannot catch a bug living in the data-marshaling code that BUILDS those records — this class of bug needs the marshaling code itself extracted into a directly-callable function (build_verdict_records_from_tsv()) and exercised by its own test.

• This coding-agent harness's interactive shell aliases plain grep to a PCRE-capable (ugrep-backed) wrapper via a .bashrc-sourced shell FUNCTION — but that function is interactive-shell-only and is NOT inherited by a script invoked as bash script.sh, which falls back to whatever the real grep binary on PATH is (in this environment: BusyBox, no -P support at all). (2026-08-20, build-time-versioning task 6.2): echo x | grep -oP '(?<=x)y' succeeded fine when typed directly as a Bash-tool command, but the identical grep -oP call inside a standalone .sh file, run via bash thefile.sh, failed with grep: invalid option -- 'P' (BusyBox usage dump). Confirmed via type grep in an interactive call (shows a _cc_a-prefixed shell function shelling out to the Claude Code binary itself acting as ugrep) vs. which -a grep//sbin/grep --version showing BusyBox as the real underlying binary. Any shell script file meant to be run standalone (not just typed inline as a Bash-tool command) must assume POSIX-only grep/sed (BRE, -E, -F, -o) — no -P/PCRE, no lookbehind/\K. Use sed -n 's/.../\1/p' (POSIX BRE, works under BusyBox) instead of grep -oP '...\K...' for field extraction inside any committed script.

• A direct go build (even a real, working one) without -mod=readonly can silently rewrite a COMMITTED go.sum, adding unrelated indirect-dependency checksum lines as a side effect of resolving the full module graph — even though go env GOFLAGS was empty and go.mod itself was never touched. (2026-08-20, build-time-versioning task 6.2): building apiserver/cmd/main.go directly (bypassing the network-hung go_build wrapper, see below) added ~65 new /go.mod-hash-only lines to src/frontend/apiserver/go.sum on the first attempt — caught only via a post-hoc git status --porcelain sweep, not by the build itself (exit 0, no warning). Go 1.16+'s -mod=readonly default blocks go.mod EDITS but does NOT by itself block go.sum EXTENSION for modules already implied by go.mod's requirements. Fix: pass -mod=readonly EXPLICITLY on every ad-hoc go build invocation that isn't supposed to touch repo state — confirmed live that the same build still succeeds under explicit -mod=readonly with zero go.sum diff, so nothing was genuinely missing; the extra entries were pure side effect. Any script/agent that runs go build directly against a checked-out repo (not via the project's own wrapped build path) must always pass -mod=readonly and verify git status --porcelain on go.sum/go.mod afterward, every time, no exceptions.

• src/tools/go_build's hardcoded GOPROXY=https://goproxy.cn + a fresh-per-invocation GOCACHE/GOPATH under meson's own $TMPDIR sandbox can hang indefinitely in this environment — not fail fast, HANG: an established (not refused/reset) TCP connection to goproxy.cn:443 sits in futex_do_wait with near-zero CPU time for 8-10+ minutes across repeated attempts. (2026-08-20, build-time-versioning task 6.2): xce-build.sh apiserver <profile> <dir> (which builds apiserver AND xcecli together via this wrapper) reliably hung on xcecli's fresh module downloads; confirmed via ss -tnp showing ESTAB to 8.45.176.179:443 (=goproxy.cn, independently reachable fine via plain curl//dev/tcp from the SAME shell) with the go process's CPU time frozen — a genuinely stuck data path, not slowness, since it never progressed no matter how long left running. Root cause isolated to go_build's own env overrides (export GOCACHE=$TMPDIR/go-cache; export GOPATH=$TMPDIR/go; export GOPROXY=https://goproxy.cn), which bypass the ALREADY-POPULATED default $GOPATH/pkg/mod cache (go list -m all from an ordinary nix develop shell, no wrapper, resolved apiserver's full module graph in under a second, zero downloads). Workaround (do NOT modify go_build/Makefile — this is this specific sandbox's networking quirk, not a real bug in those files): invoke go build directly with the default module cache (GOPROXY=off is fine since everything needed is already cached) plus the same CGO_CFLAGS/CGO_LDFLAGS pkg-config derivation test-backend/apiserver's own meson.build already uses (see bug-1597's documented fix) whenever a real apiserver/xcecli build is needed standalone in this sandbox and xce-build.sh apiserver/make build-server hang or fail. Do not blindly increase the timeout and retry — a truly stuck TCP data path does not resolve itself no matter how long you wait; diagnose via ss -tnp//proc/<pid>/wchan first.

• Refinement to the above (dirty-content-fingerprint task 4.4, 2026-08-21): when the Go-side change under test lives ONLY in apiserver/cmd/main.go (or another cmd-level file) and doesn't need the real cgo backend, skip bug-1597's full CGO_CFLAGS/CGO_LDFLAGS pkg-config reconstruction entirely — apiserver/backend/ already ships a //go:build !cgo stub (stub.go) alongside every //go:build cgo file, so CGO_ENABLED=0 GOPATH=/home/claude/go GOPROXY=off go build ./cmd/main.go (matching the Makefile's own test target convention, GOFLAGS := CGO_ENABLED=0) compiles cleanly with zero C dependencies and produces a real, runnable binary — confirmed live across go vet, go test, and 5 distinct -ldflags-injected -version scenarios plus a real (if CGO-stub) daemon-start log capture. Only reach for the full CGO reconstruction when the task actually needs the real backend/libxce integration exercised (e.g. bug-1597's own build-server scenario). -mod=readonly was NOT needed this time (checked git status --porcelain on go.sum/go.mod after every build — zero diff both times), likely because the module graph was already fully resolved in /home/claude/go/pkg/mod from prior sessions; still worth checking every time per the entry above, since whether it triggers depends on cache state, not code.

• This sandbox's nix build (not nix develop) dispatches to a REMOTE builder over SSH (ssh://rshen@172.31.109.18), which is NOT reachable directly via SSH from inside this sandbox (connection times out — same network restriction family as the goproxy.cn DNS failures). (2026-08-21, dirty-content-fingerprint task 6.3): confirmed via nix build's own stderr ("The failed build directory was kept on the remote builder due to --keep-failed... re-run with --builders '' to disable remote building"). Practical consequence: --keep-failed's leftover build directory (containing real, successfully-linked binaries from a partially-failed multi-target derivation) is NOT locally inspectable/runnable without either --builders '' (forces a full local rebuild from scratch — expensive, full ASAN tree) or SSH access to that remote host (unavailable here). When a task needs evidence from a nix build that can't fully succeed (e.g. the known bcctool/goproxy.cn limitation), prefer nix eval (fast, eval-time, no builder involved) cross-checked against the real build LOG's mesonConfigurePhase flags: line (via -L) rather than trying to extract-and-run an actual compiled artifact — this is also literally what task 2.2 already did.

• This sandbox's mktemp is BusyBox's implementation, not GNU coreutils — it requires the TEMPLATE to literally END in XXXXXX with nothing after it (no --suffix flag, no trailing literal suffix like .XXXXXX.log). (2026-08-21, dirty-content-fingerprint task 6.3): mktemp "path/name.XXXXXX.log" fails with a bare mktemp: Invalid argument (no other explanation) under BusyBox; mktemp --help confirms ("TEMPLATE must end with XXXXXX"). Always end mktemp templates in this repo's scripts with XXXXXX and nothing else — append any desired suffix to the returned path afterward if truly needed, don't rely on GNU's suffix support.

• A self-check that greps a script's OWN SOURCE for a literal flag string (e.g. confirming --impure is never used) will false-positive on the script's own comments/log messages if those comments discuss the flag by name (which good documentation often does). (2026-08-21, dirty-content-fingerprint task 6.3): a blind grep -q -- '--impure' "$0" matched this script's own header prose explaining why --impure is deliberately avoided. Narrow such self-checks to lines that begin an actual invocation of the relevant command (e.g. grep -A2 -E '^[[:space:]]*nix (eval|build|develop|log)\b' "$0" | grep -q -- '--impure'), not a whole-file grep — this generalizes to any "assert we never do X" self-check in a script whose own documentation legitimately mentions X.

• Reinforcement of the bug-1615 same-statement local a=... b="$a..." gotcha: recurred independently in a brand-new file (extract_flag_from_json(), dirty-content-fingerprint task 6.3, logged as bug-1616) less than a day after being first documented — this pattern is easy to reach for out of habit when writing a "compute one derived value from a parameter" helper. When writing ANY new bash function with local x="$1" y="...$x...", default to splitting the dependent assignment onto its own local statement, don't rely on remembering the gotcha in the moment.

• scheduler-sim-decision-parity task 1.1 (2026-08-22): relocating an xce_scheduler_t*-signature function to xce_sched_core_t* (the sched_resolve_job_picker() precedent, now repeated for sched_account_jobs()/sched_charge_account()/sched_uncharge_account()/sched_update_account()/sched_update_res_accounts()/sched_commit_job_alloc()) has a hidden fan-out cost: EVERY testsuite file that white-box #includes the driver/scheduler source and provides its OWN link-satisfying spy/stub for that function (grep the whole testsuite/ dir for the function name, not just the file(s) named in the task boundary) must have its stub's signature updated too, or the build fails with a type mismatch/duplicate-declaration error — 6 separate test files needed this for sched_account_jobs() alone (test_driver_decision_cb.c, test_resval_ownership.c, test_admin_request.c, test_sim_equivalence.c, test_record_lifecycle.c, test_driver_translate.c), none of them named in the task's own boundary list but all mechanically required by it. This project's codebase convention on the relocated parameter's name is to rename it from sched to core (confirmed via core.c's own sched_get_job/sched_add_worker/etc., not just keeping the old name with a retyped pointer) — match that, don't just retype in place.

• Testing "does function X behave identically given a core from container A vs container B" (Req 1.4-style cross-container proof) does NOT require white-box #include-ing the heavyweight driver file that defines the thin iterate-and-call wrapper (here, scheduler.c's ~20-line sched_account_jobs()) if the real accounting logic lives one level down in a lighter, already-provenincludable file (resource.c, per test.c's own existing #include "../src/resource.c") — no test in this project's testsuite/ white-box-includes scheduler.c at all (confirmed via grep before writing), because its link-time dependency surface (controller-join, capnp publish, HA candidates) is much larger than resource.c's. Test the real lower-level functions directly instead (sched_charge_account()/sched_uncharge_account()/sched_commit_job_alloc() here) and document in the new test's header comment exactly why the thin wrapper itself isn't independently exercised (already covered elsewhere, e.g. by an existing ordering-proof spy test) — this is honest, still exercises 100% real relocated code, and avoids a huge stub-writing burden.

• xce_sched_core_init() is safe to call twice in the same test process/function to build two independent, simultaneously-alive xce_sched_core_t instances for a side-by-side comparison — established precedent test_snapshot_save_load.c (core_a/core_b) already relies on this; it does re-init a couple of process-global singletons internally (xstring_table_init()) but that's a harmless re-init, not a crash/corruption risk, confirmed by that pre-existing test passing. White-box-including ../src/core.c + ../src/resource.c (NOT core_events.c/core_session.c/decision.c/plugin.c, unneeded if the test only calls xce_sched_core_init()/sched_add_worker()/sched_charge_account()-family functions directly, bypassing the event-push/session-begin/decision-drain APIs entirely) needs exactly 7 link-satisfying no-op stubs: the kill_sched_job/queue/worker/user/project family + kill_sched_resval (core.c's kvmgr/zarray datakillers) + plugin_unregister (core.c's plugins-kvmgr datakiller) + plugin_prepare_allocation/sched_ctx_get_uncomit_host (resource.c's sched_try_alloc_on_host_()'s own callees, needed only because that whole file is pulled in, even though this test path never calls it) — the last two are the exact same stub pair test.c's own resource.c-only white-box test already establishes needing.

• Never use date -d with a RELATIVE expression in this environment (-7 days, +5 days, last monday) — date first on PATH is BusyBox v1.36.1, and its failure is silent-by-default rather than loud. (2026-08-23, /weekly_report skill authoring, bug-1628): BusyBox date does parse absolute dates (date -d 2026-06-29 +%A → Monday), which masks the gap during casual probing, but rejects all relative arithmetic with invalid date on stderr — so START=$(date -d "-7 days" +%F) captures only stdout and yields an EMPTY string. The empty value then flows into git log --since="" --until="", which git treats as no time filter at all (unbounded history) instead of erroring, so a broken date computation silently produces a plausible-looking-but-wrong result. Note /bin/date IS GNU coreutils 9.8 — the nix profile bin dir simply precedes /bin on PATH. Do date arithmetic in python3 (/bin/python3, always present) via datetime/timedelta, emitting VAR=value lines consumed by eval "$(python3 - <<'PY' ... PY\n)"; call /bin/date by absolute path only if GNU date is genuinely required. Corollary that generalizes beyond dates: whenever a computed shell variable feeds a git log --since/--until-style range filter, assert it is non-empty first — git's "empty arg means no filter" behavior turns an upstream failure into silent wrong output rather than a crash. Same BusyBox-shadows-GNU family as the grep -P and mktemp entries above.

• A scheduler resource-accounting test fixture that lets a job's xce_res_account_t.residx resolve LAZILY (left at -1, relying on sched_update_res_accounts()'s own first-use key_find()/add_vres() fallback) can SEGV inside zarray_get_ex() instead of failing loudly. (2026-08-23, scheduler-dispatch-commit task 2.1, bug-1629): a minimal test fixture that never pre-registers a resource name in core->resmgr (no real resource-config load, unlike production) hit a silent failure path — sched_update_account() discards sched_update_res_accounts()'s return value entirely, so if that inner call's key_find()/add_vres() resolution doesn't complete, residx silently stays -1 with no error surfaced anywhere up the call chain (sched_commit_decision() still returns success). zarray_get_ex() (lib/zarray.c:332) only guards idx >= array->caps, NOT idx < 0, so passing that leftover -1 to get_resvals()/put_resvals() reads array->data[-1] — an underflow read ASAN reports as a high-address SEGV, not a clean bounds error. Fix: in any test fixture exercising real charge/accounting code without loading real resource config, pre-register the resource explicitly first via add_vres(&vres, core.resmgr, name, 0) and use its returned (non-negative) index for get_resvals()/put_resvals(), rather than trusting the production lazy-resolution path to run to completion inside a minimal fixture.

Decision Log

• HA/consistency model follows Raft + CP strictly: majority ACK is the hard commit barrier for all state-changing writes (reject on timeout, never local-apply-and-degrade); the own-term commit rule (not a 4th "COMMITTING" state) is what gives a new leader safety.
• A controller process must NEVER self-exit/self-abort as a designed response to an operational condition (only a genuine crash or admin/supervisor kill) — an unrecoverable/fenced member logs loudly, stays permanently non-promotable, stays alive, and keeps retrying.
• Commit-first-then-reply: job/queue/host control ops validate → mutate(provisional) → WAL append → commit_submit(WAIT) → reply; no success reply before commit. Worker-reported fact entries instead follow pure Raft semantics — retained/retried indefinitely, no commit timeout.
• No client reply-wait timeout as a fix for a hung request — fail loud at the real transport/source instead of masking with a deadline backstop.
• Product is pre-release: no backward/mixed-version wire compatibility required; wire format changes may be made freely.
• resource-value-verification task 7.1 (2026-08-18): aggregate_verdict()/build_verdict_record() (run-resource-verify.sh) re-derive MATCH/MISMATCH/UNREACHABLE from raw *_OBSERVED values via controller_values_match() at aggregation time, rather than trusting each layer's already-rendered CONTROLLER_CHECK_RESULTS/SCHEDULER_CHECK_RESULTS/APISERVER_CHECK_RESULTS strings from tasks 6.2-6.4 -- deliberate, so the SAME classification code path is exercised by both the live-run wiring and task 8.2's synthetic-record tests (a synthetic test that only fed pre-classified strings would just be testing string formatting, not the comparison logic). A rejected resource name is modeled as its own failure category (REJECTED, layer=injection) distinct from a per-layer MISMATCH/UNREACHABLE, and always forces overall FAIL, per design.md's Error Handling ("unrecognized resource name is a run-level failure") -- this is NOT implied by Requirement 6.2/6.3 alone (those only speak to "checked layers"), it's a deliberate design-level extension. The script's bottom dispatch block is now guarded ([[ "${BASH_SOURCE[0]}" == "${0}" ]]) so it can be sourced to get aggregate_verdict() et al. in scope with zero side effects (no main()/deployment triggered) -- this is the intended integration point for task 8.2's dedicated test harness.
• resource-value-verification task 7.1 round 2 (2026-08-18, bug-1582, round-1 REJECTED -- see Do-Not-Repeat entry for the mechanism): main()'s OWN record-building loop (feeding aggregate_verdict(), distinct from aggregate_verdict()/build_verdict_record() themselves, which were and remain correct) had the actual bug -- sched_ok/api_ok derived from *_OBSERVED array-key existence instead of the *_CHECK_RESULTS classification strings, masking every genuine scheduler/apiserver UNREACHABLE as MISMATCH. Fixed by switching to a leading-UNREACHABLE: check on SCHEDULER_CHECK_RESULTS/APISERVER_CHECK_RESULTS (the controller layer's ctrl_ok derivation from CONTROLLER_OBSERVED array-key existence was independently confirmed fine and left untouched -- that layer has no separate UNREACHABLE concept). Also extracted the loop into a standalone build_verdict_records_from_tsv() function (populates global VERDICT_RECORDS) specifically so a synthetic test can exercise main()'s actual data-marshaling code, not just aggregate_verdict() called with hand-built records -- the round-1 gap was that no test exercised the code that BUILDS records from the *_OBSERVED/*_CHECK_RESULTS globals, only the pure aggregation function itself. Live-reproduced by killing the scheduler process right after confirm_deployment_up() returns (before inject_resources()/query_scheduler_resvals() ever run) -- deterministic, no race window, and confirmed the controller/apiserver layers are unaffected by an isolated scheduler kill (each layer's check is independent).
• build-time-versioning task 4.2 (2026-08-20): apiserver/meson.build/xcecli/meson.build's existing CURRENT_SOURCE_DIR="@CURRENT_SOURCE_DIR@"/BUILD_ROOT="@BUILD_ROOT@" command-array entries (passed to env_program = find_program('env'), no shell involved) put literal double-quote characters INTO the env var value seen by the child process — meson's command: list is exec'd directly, so shell-style quoting around a placeholder is not stripped by anything; go_build's own fix_input_envs() explicitly sed -e 's@"@@g's those two vars to compensate, proving this empirically. When adding new KEY=VALUE entries to this SAME mechanism (XCE_GIT_COMMIT/XCE_BUILD_TIME, sourced from get_option('commit_id')/get_option('build_time'), task 1's options — no meson placeholder substitution needed since they're already resolved strings), deliberately did NOT wrap them in quotes ('XCE_GIT_COMMIT=' + get_option('commit_id'), not 'XCE_GIT_COMMIT="..."') — go_build's task-4.1 code reads $XCE_GIT_COMMIT/$XCE_BUILD_TIME directly with no quote-stripping, so quoting these would have embedded literal " characters into the linked -X main.GitCommit=... value. Live-verified via DEBUG=1 (go_build's own set -x trace mode) meson-driven build: clean values (18ae0cc3, ISO8601 timestamp) reached go build -ldflags, and xce-build.sh apiserver/xcecli end-to-end produced binaries with -X main.GitCommit=<hash>-dirty -X main.BuildTime=<ts> in their go version -m build info, no stray quotes. General lesson: "mirror the existing mechanism" doesn't mean copy-paste every byte of an existing established pattern — check whether the DOWNSTREAM consumer (here, go_build) actually expects/strips the same decoration before reusing it verbatim.
• resource-value-verification spec (2026-08-17): verify scheduler/apiserver resource-value propagation via a one-line trace/debug log at each layer's point of application (core_events.c's RESVALS_DELTA case; apiserver's OnResourceLocationUpdated), grepped live, instead of record+replay+REPL or a REST/CLI query — matches how bug-1568/1569 were actually root-caused, and is a stronger read of "the scheduler's own resource state" than a replayed reconstruction. Trade-off accepted: doesn't exercise the real query/REST path, only storage correctness. Load-bearing gotcha: TRACE=tlog_debug3 needs -vvv; apiserver's logger.Debug needs logger.yaml's logging.level: debug — both silently no-op otherwise, producing false timeouts. Also: extend xce-deploy-ha's deploy-ha.sh (opt-in, conditional-on-binary-present, mirroring its existing worker/scheduler pattern) plus a generated start-all.sh, rather than building a parallel standalone deploy script — avoids re-implementing the same deploy conventions twice.
• bug-1690 (2026-08-25, WAL_JOB_RUSAGE/job.run_usage restart-drop fix): established a NEW apply-seam pattern, distinct from bug-1678/1687/1689's "decode, free-and-replace" shape — merge, don't replace, whenever the field being restored can also be the job's LIVE structure (a fact type whose live-mutation call site runs synchronously BEFORE persisting, the same "mutate-then-persist" shape DISPATCH/START/FINISH/RUSAGE all share) and any of that structure's nested pointers can originate from something other than a plain malloc() (here: makeString()'s xstring pool). merge_job_runusage() (job.c) is the reference implementation: overwrite scalar fields unconditionally in place, match-and-update existing nested entries by key (never freeing their pointers), only adopt genuinely-new nested entries by ownership transfer. Also established: for a fact type whose LIVE mutation almost never itself changes job.state (a rusage report keeps reporting the same RUN state), gate the type-specific field adoption on nothing narrower than "trailing bytes present" — do NOT reuse START/FINISH/DISPATCH's result==JOB_TRANSITION_APPLIED gate, since that would make the new field's adoption dead code for the realistic repeated-report case. Both of these are genuine deviations from the just-established sibling-bug precedent, found by reading the real code (update_host_runusage(), persist_apply_pump_run()'s doc comment, apply_job_update_transition()'s guard) rather than pattern-matching the fix shape — worth treating "mirror the precedent" instructions as a strong prior to verify against the actual code for THIS field, not a template to copy blindly, especially around allocator provenance and state-transition gating.
• This project's existing home for a standalone, CLI-runnable regression/integration check that needs a REAL build or live deployment (not a meson test()) is src/tools/test-<topic>.sh — test-commit-durability.sh and test-heartbeat-tolerance.sh are the established precedent (shared header-comment style documenting the invariant under test + its spec, green()/red()/yellow() colour helpers, a pass()/fail() counter convention). Followed this for build-time-versioning task 6.1's test-build-version-regression.sh rather than inventing a meson test() (two full rm -rf+meson setup+compile cycles per profile is far too expensive for meson test) or a new top-level test-framework directory. When a task explicitly says "use judgment / follow existing convention if the design doc doesn't pin down an exact path," check src/tools/ first for this class of check before reaching for testsuite/ or inventing something new.
• xce-build.sh has set -euo pipefail, and every profile's RUN_TESTS block (asan only, currently) runs the FULL component meson test suite unconditionally as the LAST step of the invocation — so ANY single failing test in that suite (even a pre-existing, unrelated one, e.g. bug-1560's scheduler symlink-escape flake) makes the WHOLE xce-build.sh <component> <profile> ... invocation exit non-zero, even though meson compile already fully succeeded and the real binary exists on disk. A caller/wrapper script that treats a non-zero xce-build.sh exit as "the build failed, binary unusable" will incorrectly skip/abort real post-build work (e.g. checking --version output, as test-build-version-regression.sh needs to). Fix pattern: never bare-call it under your own set -e — capture the exit code via cmd || rc=$? (NOT if cmd; then ...; fi with no else, which loses the real nonzero code — POSIX defines an if with a false condition and no else as exiting 0, discovered live re-deriving this), print a WARNING distinguishing "compile succeeded, only the bundled test run failed" from a genuine failure, and let a SEPARATE binary-existence check be the actual pass/fail signal for "did the build produce something real."- A sched_account_jobs()-shaped no-op stub already present in SOME white-box test files that #include "../src/sim_loop.c" directly does NOT mean it's present in ALL of them — adding a genuinely NEW production call inside a white-box-#include'd file breaks every sibling TU that lacks the matching stub, and the only reliable way to find them all is to build and read the linker errors, not to grep-and-guess. scheduler-sim-decision-parity task 2.1 (2026-08-22): sim_loop.c's round boundary gained a real call to sched_account_jobs(&(sim->core)) (previously never called there at all, bug-1613). test_sim_equivalence.c/test_sim_resource_contention.c (task 1.2-era files) already carried a no-op stub anticipating this; 8 OTHER sibling files that also #include "../src/sim_loop.c" (test_sim_event_loop.c, test_sim_verify.c, test_sim_alt_events.c, test_sim_record_output.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c, test_repl_execution_control.c, test_repl_equivalence.c) did NOT, since none of them ever needed the symbol before. grep -rl '#include "../src/sim_loop.c"' testsuite/ first, then just build — the linker's undefined reference to sched_account_jobs errors enumerated the exact fix list precisely, faster and more reliably than reasoning about it from source alone.
• xce_request()/every xce_*_req() wrapper (src/lib/request.c) is a genuinely non-blocking, fire-and-forget send — it capnp-encodes the request and endpoint_enq()s it onto ep->outq, then returns; it never waits for a reply itself (the synchronous xceapi.c wrappers like xce_hello/xce_jobsubmit layer a separate cbuf_timedget() wait on top, but the raw xce_*_req() call a controller-side handler like dispatch_a_job_() makes does not). endpoint_enq() (netio.c) succeeds unconditionally against a zeroed endpoint_t with ep->iomgr == NULL: dataq_enq() has no failure path at all, and endpoint_update_epoll() is a pure no-op whenever ep->iomgr == NULL (skips the whole iomgr_update_poll branch). Net: to make a controller-side xce_*_req() call "succeed" in a unit test, you do NOT need a live peer, a synthetic reply thread, or a socketpair — a static endpoint_t (or embedded field) zeroed with .fd = -1 and left with no iomgr is sufficient, and packets just accumulate on ep->outq for the test to endpoint_deq()/free_packet() afterward. This exact pattern is already established/proven live by test_commit_barrier_reply.c's g_wkr_disp_ep/cbrp_arrange_jobdispatch() fixture (its own file header even documents it: "a zeroed static endpoint_t with no iomgr... packets stay queued") — reused directly for scheduler-dispatch-commit task 3.1's test_dispatch_commit.c success-path tests, no mocking/round-trip machinery needed.
• controller_unit_test (and presumably every other argp-based Unity runner built the same way in this project) supports a repeatable -t <group-name> (or --run <group-name>) CLI filter (controller_test.c's TEST(x) macro checks check_test(&arguments, #x) before calling x() — with zero -t args every group runs, with one or more only the named TEST(...) group(s) run, matched against the literal TEST(test_foo) identifier as a string). Invaluable for isolating ONE new test group from a suite-wide hang/flake without touching any source: run the already-built binary directly, e.g. ./controller_unit_test -t test_dispatch_commit, or build a -t list for "everything except the one broken/hanging group" via grep -oP '(?<=^extern void )test_\w+(?=\(void\);)' controller_test.c | grep -v '^test_broken_group$' and pass each as its own -t name. This bypassed a pre-existing, already-tracked hang (bug-1429, test_commit_barrier_reply.c's test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs, unrelated to the diff under test) cleanly — confirmed a new test group's 3 tests all passed AND that every other group (91 of 92) still matched the pristine-tree failure count exactly, without needing to touch/skip/comment out any test source.
• bug-1429 (controller_unit_test TIMEOUT 300s/SIGTERM, NOT root-caused) has now recurred a 3rd time (2026-08-23, scheduler-dispatch-commit task 3.1), each time with a COMPLETELY unrelated diff and a DIFFERENT hang location (test_demotion_replay_yields_committed_only → test_configack_* group → test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs, all in test_commit_barrier_reply.c/test_commit_barrier_rollout.c). Before concluding a suite-wide TIMEOUT is caused by your own change, check .wolf/buglog.json for bug-1429 first, then get an independent data point on a genuinely pristine tree (git stash -u your own diff, rebuild+retest once) — a pristine run completing cleanly while your-diff runs hang 2-for-2 does NOT by itself prove causation when the hang location has zero code-path relationship to your diff (traced directly: the hanging test never calls the function you changed) and the SAME symptom is already logged against two prior, unrelated diffs. The -t filter (above) is the fastest way to get a real, uncontaminated pass/fail answer for your OWN new tests when a pre-existing hang blocks the full suite.
• In the current (pre scheduler-sim-decision-parity tasks 2.2-2.4) codebase, a RUN-state job's job->accounts/job->allocs is ALWAYS NULL in simulation mode — there is no live path that ever sets it. Two independent, compounding reasons, both confirmed by reading the real code: (1) sim_picker_thread_fn()'s (sim_loop.c) own picker thread unconditionally PENDs every candidate (picker->picker_iface->on_pend() + make_pend_decision()) — it never emits a live DISPATCH (documented "Known scope boundary": real allocation needs a live xce_scheduler_t* the sim driver deliberately doesn't have, belongs to the not-yet-designed scheduler-allocation spec). (2) Even if it did dispatch, sched_commit_decision() (scheduler.c) — the ONLY function that ever does job->allocs = decision->u.alloc; job->accounts = make_job_account(...) — is only ever called from sched_allocator() (scheduler.c's own real allocator thread), which sim mode never runs; sim's own decision-drain loop (sim_loop.c) reads core->decisions directly and just records/prints/REPL-checks each one, with no sched_commit_decision() call anywhere in that path. Consequence for task 2.1: sched_account_jobs()'s new round-boundary call (which only charges a job whose job->accounts != NULL) has genuinely nothing to charge yet in isolation — it only becomes behaviorally meaningful once Recorded-Decision Reconciliation (tasks 2.2-2.4) populates job->accounts from decoded_session->decisions. Task 2.1's own regression test therefore has to prove the WIRING (is sched_account_jobs() called, once per round, inside the session-active window) via a spy, not an end-to-end "resInUse becomes nonzero" assertion — that end-to-end behavior only becomes true after 2.2-2.4 land.
• sched_sim_driver_run() (sim_loop.c) is a single monolithic entry point — unlike driver.c's decoupled, single-round xce_sched_driver_run_normal_mode_round() — that opens the replay file, spawns its OWN picker thread (sim_picker_thread_fn) and bootstraps the real builtin_fcfs plugin (sim_bootstrap_picker(), sim_picker.c) internally, all inside one call. This means test_driver_session_ordering.c's cross-thread wbuf-rendezvous technique (substitute a test-controlled picker plugin, spawn it in a test-owned thread, call the driver's one-round function directly) is NOT reproducible against the sim driver without either modifying sim_picker.c (routinely outside a narrowly-scoped task's boundary) or restructuring sched_sim_driver_run() into smaller testable pieces (not what design.md's File Structure Plan asks for — it's described as an inline round-boundary edit, not a new decomposed interface). Pragmatic substitute used for task 2.1's test_sim_session_ordering.c: since xce_core_begin_session()/sched_account_jobs()/xce_core_signal_go() are three unconditional, sequential, same-thread calls in sim_loop.c's own source (no branch can reorder them), the only genuinely runtime-observable fact is "does it actually get called, and does it see core->session_active==1" (proven via a spy) — NOT a full cross-thread ordering race, which doesn't exist to prove in the first place given cbuf's already-tested blocking rendezvous semantics.
• controller_unit_test's notify() capture (test_commit_barrier_reply.c's g_notify_* arrays) is static/file-local, so no OTHER test file in the same binary can read it directly — even though that file is the sole notify() definition linked into the binary (dataex.c's real one is excluded). scheduler-dispatch-commit task 3.3: to let a new test file (test_dispatch_fail_retry.c) observe notify_dispatch() (job.c) actually firing, added a handful of tiny non-static cbrp_notify_capture_*_ext() accessor functions to test_commit_barrier_reply.c (count/event/jobid/code by index, plus a reset wrapper) rather than duplicating a second notify() stub (impossible — duplicate-symbol link error across a single binary) or skipping the observation entirely. Read-only, additive, does not touch that file's own capture/reset logic or any of its existing tests — a low-risk way to extend an existing test's shared mock surface to a sibling file in the same binary.
• dispatch_a_job_()'s key_find(c->workermgr, ...) for a genuinely UNREGISTERED worker key returns rc>=0 with w left NULL (hits XCE_DISPATCH_FAIL_SITE_WORKER_NULL), not rc<0 (XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP) — confirmed empirically (task 3.3), matching test_dispatch_commit.c's own prior comment. A genuine key_find() internal failure (rc<0, the WORKER_LOOKUP site) has no known black-box trigger from a test without reaching into librtti's own (external, opaque) internals — test_dispatch_fail_retry.c's test_worker_notfound_resolves_retryable deliberately documents that it covers "whichever of WORKER_LOOKUP/WORKER_NULL the runtime takes" rather than claiming to isolate one, since both sites share the exact same classify/resolve code shape and the test's assertions hold for either.
• A real, unregistered-with-its-iomgr socket fd (socket(AF_INET,...) + a live iomgr_t via iomgr_init(), ssl_handshake_done=1, flags=0, never EPOLL_CTL_ADD'd) is a reusable, deterministic technique for forcing an xce_*_req() call to fail — EPOLL_CTL_MOD on the unregistered fd fails ENOENT every time (endpoint_update_epoll()), and since ep->flags never advances past 0 on failure, the SAME broken endpoint can be dispatched against repeatedly in a loop for a bounded-retry-escalation test, no per-attempt endpoint reconstruction needed. First established by src/lib/testsuite/test_netio_arm_failure.c/test_request_send_propagation.c (bug-646/650); reused directly at the controller layer for test_dispatch_fail_retry.c's XCE_DISPATCH_FAIL_SITE_RPC_FAILURE tests (task 3.3) by pointing a worker_t.ep at the broken endpoint instead of the established "zeroed, no-iomgr ep always succeeds" fixture task 3.1 uses for the success path.
• A prior task's own test file can pre-emptively flag a specific assertion as "this will need to change once task N.M lands" in its own comment/message string — grep for the LATER task's own number in sibling test files before assuming "task boundary == only my new file changes." Task 3.1's test_dispatch_commit.c::test_dc_worker_not_found_leaves_job_pend_and_alloc_uncommitted asserted dispatch_fail_count stayed unchanged with the message "incrementing it is task 3.3's job, not 3.1's" — a deliberate, anticipated staleness, not an accidental regression. Confirmed via the mandated RED (git-stash job.c only, rebuild, run new tests → fail as expected) → GREEN (restore, rebuild) cycle: the GREEN pass surfaced this ONE pre-existing sibling-test failure (seeded counter 2 → now correctly 3), fixed by updating that one assertion + its message, not by touching my own new file. This is explicitly within "test infrastructure under testsuite/ may change" boundary language even when the task brief's boundary section only names the NEW file — a task boundary permitting "test infrastructure" changes covers fixing a sibling test's own foreseen staleness, not just adding new tests.
• A "mirror the precedent's free-and-replace pattern exactly" instruction (bug-1678/1687/1689's own established shape) can be actively WRONG for a new field if that field's live-mutation path sources any nested pointer from makeString() (bug-1690, WAL_JOB_RUSAGE/job.run_usage). update_host_runusage() (misc.c) builds xce_host_runusage_t.host_name from makeString() — an RTTI-managed, refcounted, HEADER-BEFORE-DATA pointer (confirmed by reading librtti's actual source in the nix store, RTTI_DATA_TO_OBJ: ((meta_obj_t*)(((char*)v) - RTTI_OBJ_HEADER_SIZE))), NOT a plain malloc(). persist_apply_pump_run() (the LIVE online-apply pump, confirmed via its own doc comment to run continuously on the primary itself, not just boot replay) re-applies every committed fact — including one the SAME process just live-mutated moments earlier via the synchronous "mutate-then-persist" pattern (the SAME pattern bug-1678's dispatch_a_job_() uses for job.allocs). For DISPATCH this is safe because job.allocs's nested host field is genuinely capnp-wire-decoded end-to-end (arrives from the scheduler already deserialized) — free_JobAlloc_ptr() on it is always consistent. For RUSAGE it is NOT safe: calling the capnp-generated free_JobRunusage_ptr() on the LIVE (xstring-backed) structure would free() a non-malloc'd pointer — guaranteed ASAN heap-corruption abort on the very first live report for a new host. General check before reusing this "free-and-replace at apply time" precedent for any NEW WAL fact type: grep the field's LIVE-mutation call site for makeString() (or any other non-malloc allocator) on any pointer the apply seam might later free — if found, replace with a MERGE (overwrite scalars in place, only adopt genuinely-new nested pointers by ownership transfer, never free an existing one) instead of a free-and-replace. Proved with a real regression test using an actual makeString()'d pointer and asserting POINTER IDENTITY survives the merge (not just value equality) — a value-only check would not have caught this class of bug. Also found, but explicitly NOT fixed (separate file/component, out of scope): job_destroy() (src/lib/jobj.c)'s own existing manual FREEUP-based walk over run_usage has this exact same latent host_name/xstring mismatch, independent of and pre-dating this fix.
• This project's Unity build has TEST_ASSERT_EQUAL_DOUBLE silently non-functional (fails every call with "Unity Double Precision Disabled") because unity_dep = dependency('unity') is a prebuilt nix package with no -DUNITY_INCLUDE_DOUBLE (Unity's own default is double support OFF unless explicitly opted in; float support is ON by default). Grep any existing test file comparing a C double field before assuming TEST_ASSERT_EQUAL_DOUBLE works — the established codebase convention (e.g. test_apply_committed_entry.c, test_dispatch_commit.c, test_worker_join.c) is TEST_ASSERT_EQUAL_FLOAT even for genuinely-double-typed fields (implicit narrow to float; fine for clean/round test fixture values). Caught only by actually running the RED build and reading the real failure message, not by reasoning from Unity's public API alone.
• When offloading a newly-discovered delaymgr/iomgr blocking-I/O violation, "vtask_enq_classified(pool, task, VTASK_LIVE_BLOCK)" is NOT automatically the right destination even though it's this codebase's most commonly cited fix-pattern name — check whether the SAME pool already carries a lock the new work also needs, via a DIFFERENT already-offloaded caller (bug-1694, checkpoint_task_fn offload, 2026-08-25). This codebase actually has TWO distinct offload destinations, easy to conflate: the vtaskmgr N-thread pool (vtask_enq_classified, VTASK_LIVE_BLOCK/VTASK_REQ_BLOCK — used for bounded handler work, e.g. persist_apply_pump_kick's apply-pump drain and commit.c's dispatch_continuation) vs. the dedicated SlowLane (slowlane_submit, c->slowlane, n_workers separate pthreads — used for genuinely long/unbounded blocking I/O: snapshot send, config-push, reconnect, catch-up-streaming, demotion-truncate). ha-async-io.md's own table already states this split ("vtaskmgr must NOT do: long/unbounded blocking I/O ... starves the pool") but it's easy to miss which category new work falls into. Concretely: persist_apply_pump_run() (the live WAL-apply walk) holds apply_pump_lock across its ENTIRE walk and is ITSELF dispatched onto the vtask pool via persist_apply_pump_kick()'s vtask_enq_classified(..., VTASK_LIVE_BLOCK) — so any OTHER unbounded, lock-holding work (checkpoint's write_snapshot(), also under apply_pump_lock) put on that SAME pool would risk starving it (a worker parked on the lock for the whole snapshot-write duration, stealing capacity from commit-continuation/apply-pump liveness work) — not a classic deadlock, but the exact "starves the pool" failure mode the split exists to prevent. The fix used the slow lane instead, exactly mirroring ha_snapshot_do_send()'s pre-existing (already-in-production, default-ON) call to the SAME function (persist_checkpoint()) from a slow-lane thread — i.e. the safest new offload destination is often "wherever an already-proven-safe sibling call already runs the identical function from," not a fresh pattern match on a fix-recipe's literal wording. General technique: before picking an offload destination, grep every existing caller of the SAME blocking function/lock for its current thread context — if a sibling caller already runs it safely from candidate destination X, reuse X; if a DIFFERENT already-offloaded consumer of the same lock lives on candidate destination Y, that's evidence AGAINST Y.
• A buglog entry's own "root cause: NOT YET ROOT-CAUSED" (or similarly hedged) status is a real signal to re-derive the causal claim from primary evidence before citing it as settled, especially when later re-occurrences of the SAME symptom show it firing on completely unrelated diffs. bug-1429 (cited by config_repl.c/ha_convergence.c's catchup/demotion offload-flag comments as "the vtask pool that deadlocked controller_unit_test on 2026-07-10") is itself logged as never root-caused, and its OWN later occurrences (2026-08-20, 2026-08-23) reproduced the identical TIMEOUT/hang signature on diffs that touched neither the vtask offload code nor persist.c's checkpoint path at all — "strongly corroborates this is a pre-existing, systemic, timing/order-dependent hang in the controller_unit_test harness," per the buglog's own later entries, not proven caused by VTASK_LIVE_BLOCK dispatch specifically. Don't repeat a downstream comment's causal claim as fact in a new design doc/comment without checking the buglog entry it cites — the comment can be more confident than the evidence it's citing.
• controller_unit_test (the main shared meson.build controller_unit_test_src binary) does NOT link the real sched.c/dataex.c at all (bug-1693, 2026-08-25) — sched.c's own on_scheduler_join()/collect_confvars() and dataex.c's notify() collide at link time with fake local definitions several OTHER files in that same binary already provide (test_commit_barrier_reply.c, test_worker_join.c), so test_config_repl.c instead ships a trivial no-op handle_scheduler_leave(cluster_t *c) { (void)c; return 0; } stub purely to satisfy the linker. Any new/changed behavior added to a real sched.c function is therefore SILENTLY UNTESTED by any controller_unit_test-based test that calls that function by name — the call resolves to the stub, not the real implementation, with no compile/link error to flag the mismatch (confirmed empirically: a new regression test asserting sub-second wake timing after calling handle_scheduler_leave() passed compilation and linking fine, then failed by taking the full 5s deadline — the stub had silently swallowed the call). nm '<obj>.o' | grep ' T <symbol>$' across every .o in the target's build dir is the fast way to confirm which TU actually provides a given symbol in a given binary before trusting a test's result. The two standalone binaries that DO link the real sched.c (sched_push_unit_test, sched_worker_change_unit_test — see their own meson.build comments for the collision rationale) don't link cluster.c, so neither binary alone can exercise a fix that spans both files for real; bug-1693's fix split verification across both (a direct-call test against cluster.c's real new function in controller_unit_test, plus a small counting-spy test in test_sched_worker_change.c proving sched.c's real caller invokes it) rather than inventing a third binary that links both (would need to duplicate most of controller_unit_test_src's already-large dependency list to satisfy cluster.c's own huge fan-out). General lesson: when a fix spans a function in sched.c/dataex.c AND a function in any of the ~15 other files real-linked into controller_unit_test_src, check with nm/grep for a same-name fake stub in a sibling test file in that binary BEFORE trusting a test that calls the sched.c/dataex.c side by name — it may be exercising a stub, not your change.