chore(wolf): checkpoint buglog.json (1062 bugs)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

.wolf/cerebrum.md

@@ -10,6 +10,7 @@
 - Keep test logs for analysis, don't skip intermediate output even if a test hangs.
 
 ## Key Learnings
+- **`fopen(path, "r")` SUCCEEDS on a directory path on this platform (glibc/Linux) — it does NOT return NULL/set errno the way "could not open the file" naively suggests.** (2026-08-26, xce-test-rc-file task 1.2, bug-1716.) The open failure only surfaces on the FIRST READ: `getline()`/`fread()` returns -1/0 with `errno==EISDIR` and `ferror(f)` set (`feof(f)` stays 0, so don't gate on that). Confirmed via a standalone repro (`fopen("/tmp","r")` then `fread`). Any function whose error-handling contract says "print a warning if the file can't be opened for reading" needs to check BOTH the `fopen()`-returns-NULL case AND a post-read `ferror(f)` check — checking only `if (!f)` right after `fopen()` silently misses the directory case (rc stays 0, warning never printed, but also no crash/runner-call — a quiet contract violation, not a loud failure, so it only surfaces via a test that actually asserts the warning text was printed, not just the return value).
 - **xcecli Cobra rootCmd location**: `src/frontend/xcecli/cmd/main.go` imports `cmd "xcecli/router"` (the import alias is `cmd`, but the actual package lives at `xcecli/router/router.go`, package-named `cmd`). Cobra's `rootCmd` is declared there as an **unexported** package-level var (`var rootCmd = &cobra.Command{...}`, router.go:21). ~~The only exported symbol is `Execute(cm *config.ConfigManager) error`, which takes no version param... boundary conflict~~ — **SUPERSEDED, resolved by build-time-versioning task 4.5 (commit fb09afe) and confirmed still true as of dirty-content-fingerprint task 4.5 (2026-08-21)**: `Execute` was extended to `Execute(cm *config.ConfigManager, version string) error`, which itself sets `rootCmd.Version = version` (router.go:29-34) before `rootCmd.Execute()`. main.go now composes the full version string (`GitCommit`/`BuildTime`, plus task 4.5's `composeVersionDisplay(GitCommit, DirtyFingerprint)` strip-and-append) and passes it into `cmd.Execute(configManager, version)` — no further router.go changes needed for version-string composition; only main.go is the boundary now.
 - **Build entry point**: `flake.nix` lives in `src/`, not the repo root. Always `nix develop /abs/path/to/src --command bash -c '...'` with an explicit path — Bash cwd isn't stable across calls, and a bare `cd src && nix develop` can silently short-circuit.
 - **Test coverage gaps**: documented full-suite commands (`xce-build.sh <component>`, `make test`) each cover only a subset — e.g. a component's default meson run misses newly-added standalone test binaries unless wired into its `RUN_TESTS` case, and the CGO/apiserver backend suite needs `make test-backend` with CGO_CFLAGS/LDFLAGS reconstructed via pkg-config (package is `c-capnproto`, not `capnp_c`). A green default build is not proof of full coverage — verify every new test binary actually runs.
@@ -131,9 +132,27 @@
 - **(2026-08-25, bug-1695) `meson.build`'s `controller_unit_test_src` deliberately does NOT link `myapp.c` (or `main.c`) — only `startup_validation.c`, `ha.c`, `cluster.c`, `persist.c`, etc.** `myapp_run()` is documented as an "untestable integration entry point" (test_ha_startup.c's own Task 5.1 comment) requiring a live network socket, so it's excluded from the unit-test binary entirely. If you need to unit-test logic that currently lives inline inside `myapp_run()`, do NOT just add a non-static function to `myapp.c` and forward-declare it in a test file — it will compile but fail to LINK (`undefined reference`) because `myapp.c` itself never gets compiled into `controller_unit_test`. Instead, extract the logic into `startup_validation.c` (already linked into both `xce_controller`/`main.c` AND `controller_unit_test`, and explicitly documented as existing "for use in main.c and the controller unit test suite") and have `myapp.c` forward-declare + call it — exactly mirroring the pre-existing `validate_startup_config()` pattern (defined in `startup_validation.c`, forward-declared at the top of `myapp.c`, called from `myapp_run()`). Confirmed via a real RED/GREEN round-trip: extracting into `myapp.c` first produced a clean compile + link FAILURE across all of `controller_unit_test`'s ~90 test files; moving the same function body to `startup_validation.c` fixed it with zero other changes.
 - **(2026-08-25, bug-1695) `ha_state_t` has TWO separate term fields that look similar but are written by different code paths and must not be conflated**: `ha->term` (top-level, `_Atomic uint64_t`, `ha.h:157`) is the adapter's own copy, written by `ha_restore_term()` at boot and by `raft_cb_persist()` on every `on_persist` callback; `ha->raft.term` (nested inside the embedded `raft_state_t`, `ha.h:198`/`raft.h:29`) is the raft LIBRARY's own internal term, seeded ONCE at `ha_init_raft()` time from `ha->term` (`rcfg.initial_term = atomic_load(&ha->term)`) and thereafter updated only by the library itself (`raft_start_election`/`raft_on_vote_request`/`raft_on_append_entries_request`, all via the SAME `on_persist` callback that also updates `ha->term`) — so they stay in sync after boot, but the ONLY correct place to read "the term as of right after this boot's restore" is `ha->raft.term` captured immediately after `ha_init_raft()` returns (not `ha->term`, and not before `ha_init_raft()` runs). myapp_run's bootstrap check reads `ha->raft.term` specifically (not `ha->term`) — a fix that snapshotted the wrong field would silently do nothing.
 - **(2026-08-25, bug-1695) A `--chaotic` HA soak's per-round "wait_for_new_primary window" log line is a WEAK signal on its own — always cross-check the raw controller logs (`grep -n "CTRL_PRIMARY\|CTRL_STANDBY\|PreVote round" logs/ctrl-*.log`) before trusting it.** The chaotic driver's own per-round polling window (~90s, 30×3s probes) can time out ("NO new primary within window") even when a genuinely NEW node DID win a real election shortly after — the log line only reflects whether the driver's OWN bounded poll happened to catch it, not whether the underlying HA mechanism worked. In this session's soak, the fixed run showed "NO new primary within window" on 5/6 rounds (superficially similar to the unfixed baseline's 6/6) while the RAW logs proved something structurally different was happening: term genuinely progressing 1→2→3→4→7 and a different node (`ctrl-2`) achieving a real, log-verified `CTRL_PRIMARY` transition via a won PreVote — vs. the baseline where term was frozen at a single value for the entire run and NEITHER peer ever showed any PreVote/election activity at all, anywhere. **The A/B technique that resolved this ambiguity**: `git stash push -- <the exact files you changed>` (reverts just those files, keeps everything else including the same freshly-built binaries' surrounding tree state), rebuild, re-run the IDENTICAL soak invocation against a fresh scratch dir, compare raw logs side by side, then `git stash pop` to restore the fix. Far more conclusive than trusting either run's summary line alone.
-- **(2026-08-25, bug-1695) A chaotic-soak `expect-job`/`verify_classify` "COMMITTED JOB LOSS" verdict is NOT self-verifying — cross-check the raw WAL with `xce_test`'s `waldump <data-dir>` REPL command (built into `xce_test`, not a separate binary) before treating it as a real controller data-loss bug.** `verify_classify` reads through the apply-pump/materialized-jobmgr layer (an RPC-level `expect-job` query), which per already-documented bug-1487 can legitimately lag behind the raw committed WAL for several seconds after a genuinely-new election (the new leader's OWN-term commit-barrier gate, `ha_arm_own_term_gate`, requires a fresh majority ack for its own-term no-op before folding in prior-term entries) — a RETRY-EXHAUSTED "not found" there does not by itself prove the entry is actually missing from the durable log. `walcompare`'s `committed_divergence` count (also built into `xce_test`) is the AUTHORITATIVE data-safety signal — it stayed 0 in this session's soak even while `verify_classify` counted 6 losses, and a direct `waldump` of all 3 nodes' `data/wal/seg-*.wal` files confirmed the ONE specific jobid in question genuinely does not appear on ANY node's WAL at all (not divergent between nodes — universally absent), tracing it to the cluster's very first cold-boot config-bootstrap window rather than anything this specific fix touched.
+- **(2026-08-25, bug-1695) A chaotic-soak `expect-job`/`verify_classify` "COMMITTED JOB LOSS" verdict is NOT self-verifying — cross-check the raw WAL with `xce_test`'s `waldump <data-dir>` REPL command (built into `xce_test`, not a separate binary) before treating it as a real controller data-loss bug.** `verify_classify` reads through the apply-pump/materialized-jobmgr layer (an RPC-level `expect-job` query), which per already-documented bug-1487 can legitimately lag behind the raw committed WAL for several seconds after a genuinely-new election (the new leader's OWN-term commit-barrier gate, `ha_arm_own_term_gate`, requires a fresh majority ack for its own-term no-op before folding in prior-term entries) — a RETRY-EXHAUSTED "not found" there does not by itself prove the entry is actually missing from the durable log. `walcompare`'s `committed_divergence` count (also built into `xce_test`) is the AUTHORITATIVE data-safety signal — it stayed 0 in this session's soak even while `verify_classify` counted 6 losses, and a direct `waldump` of all 3 nodes' `data/wal/seg-*.wal` files confirmed the ONE specific jobid in question genuinely does not appear on ANY node's WAL at all (not divergent between nodes — universally absent), tracing it to the cluster's very first cold-boot config-bootstrap window rather than anything this specific fix touched. **CRITICAL CAVEAT discovered later the same day (bug-1696): do not stop at "walcompare says 0 divergence" for a soak run driven by the bug-1695 fix specifically — that fix newly enables a restart-then-immediately-self-promote path that hits a SEPARATE, far more severe bug (WAL truncated to empty on promotion, see bug-1696 below). A truly clean soak needs BOTH committed_divergence=0 AND a raw `waldump` entry count that matches the real number of jobs submitted, not just "the specific jobid I went looking for is present somewhere."**
+- **(2026-08-25, bug-1696, CRITICAL — Leader Completeness / bug-1060-guard defeated by boot ordering) `raft_cb_become_leader()` (ha.c:1615) computes the promotion WAL-truncation floor and evaluates the ENTIRE promotion-eligibility gate using `persist_durable_floor()`/`persist_recovery_truncated()`/`persist_apply_verdict()` BEFORE `cluster_init_phase2()` — the only call that invokes `persist_recover()` for this process's first-ever boot — has run.** `is_first_boot` handling (`cluster_init_phase2()` call) sits at ha.c:~1873-1899, AFTER the safe_committed_lsn computation and `persist_truncate_wal_above()` call (~1637-1662) and AFTER the full eligibility gate (~1699-1823) in the SAME function. Before recovery, a freshly-`persist_init()`'d `persist_t` reports `durable_lsn=0` — not the real on-disk tip. So on a node that RESTARTS with real committed data on disk and then wins its OWN promotion on that same boot (a path bug-1695's fix newly makes reachable — previously a restarted node's stale term made it always defer to a peer instead), `ha_promotion_truncate_floor(safe_lsn≈0, durable_tip≈0)` computes ≈0 and `persist_truncate_wal_above(p, 0)` physically destroys the real on-disk WAL before it's ever been read. bug-1445's comment (ha.c:1901) already flagged this EXACT "gate evaluated blind before recovery" hazard for one flag (`apply_resource_fenced`) with a narrow re-check-after-phase2 patch — but never for the far more consequential `durable_floor`/truncation computation, which runs even earlier and physically deletes data rather than just making a stale eligibility decision. **General lesson: any promotion/election-safety code that reads `persist_durable_floor()`/`persist_recovery_truncated()`/`persist_apply_verdict()` must be checked against whether `cluster_init_phase2()` (this process's `persist_recover()`) has actually run yet THIS boot — `is_first_boot` is the tell; a function that both checks `is_first_boot` and reads recovery-derived persist state earlier in its own body than the `is_first_boot` phase2 call is almost certainly reading pre-recovery garbage.** Root-caused via source trace (not guesswork) after direct `waldump` evidence (entire on-disk WAL = 1 NOOP entry despite 21+ committed jobs, no snapshot dir, zero checkpoint log lines) plus the smoking-gun log line `ha_compute_safe_committed_lsn: safe_committed_lsn=0 ... (from 2 granted voters)` immediately preceding `raft_cb_become_leader: → CTRL_PRIMARY` on ctrl-1's SECOND boot. Fix pending — see bug-1696 in buglog.json for the exact proposed reorder and required A/B soak proof.
+
+- **(2026-08-26, bug-1688) When a zarray's `.count` field does something numerically impossible (doubles across a repeat push carrying the exact same keys, or drifts above `.caps`), do NOT jump straight to "something external corrupted the array between the two calls" or "the same key must be resolving to two different indices" — first check whether the CURRENT call's own writes are storing NULL.** `zarray_put_ex()` (zarray.c:267-303) only increments `.count` when the slot being overwritten currently reads NULL; it has NO NOTION of "is the new value itself NULL" — writing NULL into an already-occupied slot leaves `.count` unchanged while silently zeroing that slot's data, and a LATER write to that now-NULL slot (even with a perfectly stable index) then legitimately increments `.count`, looking exactly like "the array got corrupted between calls" if you only watch `.count`/the slot's occupancy and not the actual VALUE being written. Root-caused here only after adding a `zarray_put_ex()`-level watch-trace (a temp global pointer comparison, since the bug lived in library code called from many sites) that printed `data[i]` itself, immediately revealing `new_data=(nil)` on every write of the "corrupting" push — the two more exotic hypotheses (idx-resolution bug; core_reset_workers()/session-boundary code clearing the array) were both live-traced and definitively ruled out first, costing significant time that a "trace the actual value being written, not just count/occupancy" instinct earlier would have saved. Separately: `decode_Value_ptr()` (controller.capnp.c:2966) legitimately decodes to NULL whenever the wire's capnp pointer for that field is `CAPN_NULL` — this is not a decode bug, and any C struct field populated via `decode_X_ptr()` from a capnp-generated decoder should be treated as "can legitimately be NULL even when the enclosing struct/list entry itself is non-NULL," with call sites guarding accordingly (the same discipline this file's existing `nv == NULL`/`nv->name == NULL` guards already applied to the OUTER struct, just not to this one nested pointer field).
+
+- **(2026-08-26) `controller_unit_test`'s meson `test()` registration sets `env: ['ASAN_OPTIONS=detect_leaks=0']` (src/controller/meson.build:239) — running the built binary DIRECTLY (bypassing `meson test`) without setting that same env var yourself makes ASAN's default leak detection fire at process exit, reporting ~7400 pre-existing allocations (~954KB) across dozens of unrelated test files as "leaked" and returning a nonzero exit code, even when the actual Unity test count (`N Tests M Failures`) is 100% clean.** This is NOT a regression signal — it's the project's own accepted, documented convention (also referenced in `xce-build.sh`'s own "skipped tests" hint: `ASAN_OPTIONS=detect_leaks=0 ./$BUILD_DIR/controller/controller_unit_test`). Always grep the target's own `meson.build` `test()` block for its `env:` before treating a direct-binary-run's exit code or ASAN output as ground truth; the Unity summary line (`"<N> Tests <M> Failures <K> Ignored"`) is the real pass/fail signal, not the process exit code alone.
+- **(2026-08-26) A Unity `TEST_ASSERT_*` failure `longjmp`s out of the CURRENT test function immediately — skipping any teardown call written AFTER the assertion in that same function body.** If that teardown would have stopped a live background thread (e.g. `persist_stop()`/`persist_fini()` tearing down a real `fsync_worker` pthread), the thread leaks alive holding a pointer into that test's now-returned (and ASAN-poisoned) stack frame — the NEXT test in the same binary that happens to touch overlapping stack memory can then crash with a `stack-use-after-return` ASAN report that looks totally unrelated to the real, intentionally-failing assertion (confirmed while proving bug-1697's RED phase: a deliberately-reverted assertion failure in `test_wal_job_start_fact.c` cascaded into an unrelated ASAN abort deep inside `test_dispatch_commit.c`, later in the same run). This is a known, already-documented pathology in this codebase (see `test_upgrade_snapshot_apply.c`'s own header comment: "gather every assertion-relevant observable into a plain local variable FIRST, call teardown UNCONDITIONALLY, and only THEN assert") but most test files in this suite do NOT follow that discipline (assert-then-teardown is still the norm). When a RED-phase test run produces both the EXPECTED assertion failure(s) AND a seemingly-unrelated ASAN crash later in the same run, check whether the crash is this cascade (a live thread from an earlier failed-and-un-torn-down test) before treating it as a new bug — capture the actual observable into a variable, run teardown, THEN assert, if you need a RED-phase test to survive to completion without corrupting siblings.
+- **(2026-08-26) When a live HA soak or other long-running deployment is using `src/build-asan` (or any shared default build dir) from a concurrent session, do NOT run `/xce-build` (or any `xce-build.sh`/`meson compile`) against that same directory — even for a "just checking" build — since a concurrent `ninja`/`meson setup` can race the other session's own build, and relinking a binary a live soak's processes still hold open risks corrupting or confusing that process's on-disk image.** Always pass an explicit, unique custom `build_dir` (`xce-build.sh <component> <profile> <custom-dir>`) for any build done "on the side" of another session's live deployment — check `ps aux | grep xce_controller` (or the relevant daemon name) FIRST to see which build dir(s) are actually live before choosing where to build. Also verify a dispatched build `Agent` subagent's build_dir argument actually took effect (compare the binary's own mtime against your most recent source edit's mtime) rather than trusting its self-report — one such subagent in this session silently ended up building into the script's own default dir (`xce.build`, from a malformed/dropped custom-dir argument) and initially reported stale/incorrect RED results from a build that predated the real edits; independently re-running the build directly (not via a sub-dispatch) and cross-checking mtimes caught it.
+- **(2026-08-26) `deploy-ha.sh <binary> <top> <peers>`'s `peers` argument is a plain comma-separated `host:port` list — do NOT prefix each entry with its own `name=` (e.g. `ctrl-1=localhost:19991`); the script adds that prefix itself internally.** Passing an already-prefixed peers string produces a doubled `ctrl-1=ctrl-1=localhost:19991` in every generated start script, which silently breaks worker/scheduler cluster-join (`Failed to join cluster, retrying in 5 seconds` in the worker log) while the controller itself still starts and elects fine (its own `--peers` parsing tolerates/ignores the malformed self-reference for a single-node cluster) — the failure only surfaces once you try to verify a worker/scheduler joined, not at controller-startup health-check time. Confirmed the fix by re-deploying with a bare `host:port` peers string.
+- **(2026-08-26) `xce_test`'s REPL command set is SPLIT by connection mode — `jobsub`/`joblist`/`jobinfo`/`wait-primary`/etc. live under `client` mode's command table, while `jstart`/`jfinish`/`jfail`/`resupdate` live under a SEPARATE `worker` mode's command table (test.c's `client_commands[]` vs `worker_commands[]`).** Typing `jstart` while still in `client` mode fails with a generic `"jstart: No such command for xce test."`, not an argument or connection error — easy to misdiagnose as a jobid-format problem. To send a synthetic worker-side fact (e.g. to test a WAL_JOB_START collision scenario against a job a REAL worker is also concurrently racing to finish), you need a SEPARATE `xce_test` invocation that switches into `worker` mode (its own `join`) first — you cannot mix `jobsub` and `jstart` in one client-mode session. The `$jobid`/`$jobcount` REPL auto-variables (populated by the most recent `jobsub`/`joblist`) are real and usable across subsequent lines in the SAME script (test.c:166-179), useful for exactly this kind of immediate-follow-up scripting once you're in the right mode.
+
+- **(2026-08-26, bug-1700/bug-1701) `ha_state_t.primary_host`/`primary_port` (ha.h:161-162) are NEVER WRITTEN by any production code path — they are only ever set by test fixtures manually poking the struct.** `fill_redirect_reply()` (join.c:27) and anything that calls it (OPCode_join's handler at join.c:277, and bug-1700's new myreq_handler gate-rejection call) is therefore a NO-OP in a real deployment: the redirect it sends is always an empty host / zero port, which the xceadmin client correctly (per its own documented contract) treats as "no redirect." This was independently confirmed by grepping every `.c`/`.h` in `src/` for an assignment site (none outside tests) and is already explicitly named in `ctrlstatus.c`'s own doc comment ("the vestigial, never-written ha->primary_host") and in the already-closed bug-854's root-cause writeup — but nobody had connected the dots that this ALSO silently breaks `fill_redirect_reply()`'s OTHER caller, `join.c:277`, not just whatever new caller you're adding. **Before wiring any new caller of `fill_redirect_reply()`/relying on `ha->primary_host` for anything live, verify with a real 3-node deploy (not just a unit test that manually sets the field) that the value is actually populated on a standby** — a unit test proves the helper copies the field correctly, not that the field ever holds anything real in production.
+- **(2026-08-26) When a task's own "critical correctness check" section asks you to verify a field's value on a standby "that has NOT yet learned of any live primary (e.g., very early boot, mid-election...)", read that as a hint to check the PERMANENT-vs-transient distinction, not just the transient one it names.** bug-1700's task prompt explicitly asked this question but framed it as a transient edge case; the actual answer was "permanently empty in every state, forever" — a much bigger finding than "empty during a brief startup window." Don't let the framing of a verification question limit how far you check.
+- **(2026-08-26, bug-1701 FIX) `ha_state_t.primary_host`/`primary_port` (bug-1700/bug-1701's vestigial-field finding above) is now actually populated in production, closing the gap.** Fix is a pure LOCAL lookup in `handle_ctrlhello()` (config_repl.c:2170-2246, inside the existing `primary_ctrl_fd == -1` idempotency guard) — no new wire field needed, because every node already statically knows every candidate's `host:port` from boot (`c->myconf->peers_cfg`, parsed once from `--peers`). Match `req->from->name` (populated for EVERY incoming request at `request.c:379`, `req->from = ep->peer;`) against `peers_cfg.candidates[]` by name and copy the matched candidate's `hostname`/`port`. **General pattern worth remembering**: when a per-connection identity field (`req->from->name`) needs to resolve to a richer static record, check whether that record is ALREADY fully known locally (a config parsed at boot) before assuming you need a new heartbeat/wire-protocol field to carry it — `ha_init_raft`'s existing `voted_for_name`-to-index lookup (ha.c:7822-7829) was the exact precedent pattern to copy, found by grepping for other `peers_cfg.candidates[i].name` comparisons in the same file family. Also: `ha.h`'s field-level doc comments (e.g. `pthread_mutex_t lock; /* Protects primary_host/port + voted_for_term/name */`) are authoritative for which lock protects a field — trust that over inferring locking convention purely from a nearby unrelated field's (`primary_ctrl_fd`'s) actual call-site behavior, which can itself be under-protected without it being the intended convention for the field you're adding.
+- **(2026-08-26) `xce-build.sh <component> <profile> <build_dir>`'s `build_dir` argument is resolved AFTER the script's own `cd "$SRC_DIR"` (`cd "$(dirname "$0")/.."`, i.e. `src/`) — passing an already-`src/`-relative path like `src/xce.build` from the repo root produces a bogus DOUBLED path (`src/src/xce.build`) that silently creates a whole second build tree instead of erroring.** Caught immediately (binary just wasn't at the expected path) but wasted a build cycle. When explicitly passing `build_dir` to `xce-build.sh`, give it relative to `src/` (e.g. bare `xce.build`), or omit it and let the profile default (`asan`→`xce.build`, `debug`→`build.dbg`, `release`→`build.release`) resolve it.
+- **(2026-08-26) To produce a clean "before/after" pair of binaries for live E2E verification when OTHER unrelated files have uncommitted in-flight changes you must not touch or rebuild against**, use `git stash push -- <only the files for this fix>` (explicit pathspec, NOT a bare `git stash`) to isolate exactly the fix's own diff — rebuild for the "before" binary, `git stash pop` to restore the fix, rebuild again for the "after" binary. This leaves every other uncommitted file (the unrelated in-flight `ha.c`/`job.c`/`persist.c` work in this session) completely undisturbed throughout, unlike a bare `git stash` which would have stashed everything. For a PYTHON-only fix specifically, this same before/after swap works without any rebuild at all — just `cp` the two scheduler.py variants in and out of `src/xceadmin/src/xceadmin/` and re-run the SAME editable-install venv's `xceadmin` entry point; a `pip install -e` venv reads live from the source tree, no reinstall needed between swaps.
+- **(2026-08-26, bug-1713) In this sandbox's local-loopback 3-node deploy, the FIRST-BOOT election-settling window varies wildly run to run (sometimes near-instant, sometimes ~10-12s via a term-1 election-timeout-then-PreVote-failover fallback path) — a fixed "wait for start-all.sh's health-check, then fire N requests" tight loop is NOT a reliable way to catch a transient `E_XCE_NOT_READY`/`E_XCE_NOMASTER` window: by the time start-all.sh's own 1s-granularity health-check first observes `CTRL_PRIMARY` in a log and returns, `ha_has_write_quorum()`'s `standby_endpoints` (populated synchronously inside `raft_cb_become_leader` from the raft library's own connected-voter set) is typically ALREADY satisfied, so the gate is already open. A REAL FAILOVER (kill -9 the current primary, then hammer the survivors with NO sleep, no health-check wait) reproduced the exact live symptom reliably instead — got a genuine bare `-10013` at ~t+8.6s post-kill on an unfixed client, and confirmed zero such failures surfaced across the equivalent window on the fixed client. The bug's own root-cause writeup already named this as an equally-valid trigger ("...or any time a real failover is in flight") — prefer the failover trigger over racing first-boot timing when a live repro is needed and cold-boot timing proves flaky.
+- **(2026-08-26) A hand-rolled multi-connection mock TCP/TLS server in a test file needs a genuinely FINITE accept loop (`for handler in handlers:`), not `while self._handlers:` guarding a list that's never mutated — the latter is an always-true condition that leaves the server thread blocked in `accept()` indefinitely once the test's real client stops connecting, which does not fail the test but silently adds `thread.join(timeout=N)`'s FULL timeout to every such test's teardown (measured: 3 tests each added a flat 5s of pure teardown wait with zero effect on pass/fail).** Match the mock's handler-list length exactly to the number of connections the test actually expects (assert on `connection_count` for the "stops on its own budget" property instead of over-provisioning handlers "just in case") and additionally set a short `socket.settimeout()` on the listener as a safety net so a hypothetical future regression that causes MORE connections than expected still fails within one timeout window instead of silently reintroducing the same multi-second stall.
 
 ## Do-Not-Repeat
+- **A TDD-implementer SUBAGENT (spawned by a kiro-impl coordinator) has NO external wakeup mechanism — arming a `Monitor` tool call or a `Bash run_in_background` call and then ending the turn to "wait for a notification" just stops the subagent forever; the coordinator has to manually resume it.** (2026-08-26, xce-test-rc-file task 1.2, recurred across multiple tasks in this same spec before being called out explicitly.) Only the top-level session genuinely receives async task-notification events between turns; a subagent's turn ending IS the end of its execution for that dispatch. When a validation command (e.g. `controller_unit_test`) risks exceeding the default 120-300s Bash timeout, pass an explicit large `timeout` (e.g. 300000-600000ms) to a PLAIN FOREGROUND `Bash` call and let the tool call itself block until real output returns — never `run_in_background`, never `Monitor`, never end a turn on an unfinished command in this role. If a command is auto-moved to background because the timeout already elapsed, immediately re-run it in the foreground with a longer timeout rather than trying to "wait" for the backgrounded one.
 - **When translating a `fp_ok`-tracked, per-step-gated failure-detection pattern (like `xce-build.sh`'s dirty-fingerprint computation) into a Make `$(shell sh -c '...')` one-liner, do NOT collapse two commands into `{ cmd1; cmd2; } >"$$fp_tmp"` and gate the whole group with one `if`** — a POSIX sequential group's exit status reflects only its LAST command, so `cmd1`'s failure is silently discarded whenever `cmd2` still succeeds (2026-08-21, bug-1612: `git diff HEAD` failing was swallowed because the trailing `git ls-files | sort | xargs cat` pipe still exited 0, producing a fabricated fingerprint instead of degrading to empty). Give each step its own explicit `if ! step; then fp_ok=0; fi` (or `&&`-chain), exactly mirroring the reference script's discipline, even inside a cramped one-liner. Caught by adversarial code review with a targeted failure-injection test (wrapper `git` on `PATH` that fails only the targeted subcommand) — that test pattern is the reusable way to verify this class of fix.
 - Don't run `clang-format -i` on whole existing files here — the repo isn't clang-format-clean at baseline, so a full reformat buries the real diff; recovering via `git checkout HEAD -- <file>` on a file with other uncommitted work destroys that work irrecoverably.
 - **(2026-08-22, scheduler-sim-decision-parity task 2.3) Actually ran `clang-format -i` on two files mid-task despite this rule already being logged above — caught it via `git diff` before finishing, not before running it.** If this happens again: do NOT `git checkout` the file (destroys real work). Instead: `git diff -- <file>` and read every hunk; any hunk whose ONLY change is whitespace/line-wrapping on code you did NOT intentionally write/edit this turn is collateral reformatting — manually `Edit` just that hunk back to its pre-format text (find each such hunk, revert by hand). Verify success with `git diff -- <file> | grep -c '^-[^-]'` on regions you didn't touch dropping to 0, then rebuild+retest to confirm nothing broke. Cheap enough to be a non-event once caught, but the real fix is: check this Do-Not-Repeat list (or just never run `clang-format -i` without `--dry-run`/`-output-replacements-xml` first to preview scope) BEFORE running it, not after.
@@ -153,6 +172,7 @@
 - **A REPL/replay test fixture that reuses the SAME jobid for both a "real" event (job push) and a "cheap padding" event (job state change, added purely to exercise a second event type) accidentally mutates the real job's state, not just decorates the log.** bug-1595 test regression (task 12.5, 2026-08-19): `test_step_echoes_events_before_session_decisions` pushed job 6161 then sent a JOBSTATECHANGE event for the SAME jobid 6161 (PEND->RUN) purely to get a second event type echoed — this genuinely transitioned the real job to RUNNING before the session ran, so the picker's `next_job()` never offered it and the session's own PEND decision (which the test also asserted on) was never produced. The sibling test `test_event_breakpoint_pauses_repl_at_matching_event` already documents the correct pattern in its own comment (`append_jobstatechange_event() establishes: core_events.c WARN+skips an unknown jobid rather than fail, so ... below needs no real job to exist`) — it deliberately used unrelated jobids (111/222) precisely to keep the padding event inert. When adding a "just need any event of type X to be logged" filler event to a fixture that also has a real job in flight, always target an unrelated/nonexistent jobid unless the state transition itself is the thing under test.
 - **`[ -n "${arr[key]+x}" ]` (associative-array key-EXISTENCE) is NOT a safe proxy for "the value is meaningful/reachable" when the producer unconditionally sets that key even on failure.** bug-1582 (resource-value-verification task 7.1, round-1 REJECTED, round-2 fixed): `main()`'s record-building loop derived `sched_ok`/`api_ok` from whether `SCHEDULER_OBSERVED[$vname]`/`APISERVER_OBSERVED[$vname]` existed — but `query_scheduler_resvals()`/`query_apiserver_resvals()` (already-approved tasks 6.3/6.4) always set that key, setting it to `""` on a genuine UNREACHABLE/timeout rather than leaving it unset. Key-existence was therefore always true, silently masking every real UNREACHABLE as a MISMATCH (comparing the injected value against an empty string). Fix: derive the "is this layer reachable" signal from the layer's own pre-classified status string (`SCHEDULER_CHECK_RESULTS[$vname]`/`APISERVER_CHECK_RESULTS[$vname]`, which DOES distinguish `UNREACHABLE:*` from `MATCH`/`MISMATCH:*`) instead of from a sibling data array's key presence. General lesson: before using `${arr[k]+x}`/`-v arr[k]` as a "was this checked / did this succeed" signal, verify the producer actually leaves the key UNSET on failure — if it sets it to an empty/placeholder value instead (a common pattern for "always populate every requested key so downstream code doesn't need existence-checks"), key-existence collapses to "was this key in the requested set" a wholly different, weaker signal than "did this succeed". Also: a synthetic test that only calls the pure aggregation function directly with hand-built records (task 8.2's original approach) cannot catch a bug living in the data-marshaling code that BUILDS those records — this class of bug needs the marshaling code itself extracted into a directly-callable function (`build_verdict_records_from_tsv()`) and exercised by its own test.
 
+- **(2026-08-26, xce-test-rc-file task 1.1, bug-924) When `controller_unit_test` invoked DIRECTLY (not via `meson test`) dies instantly (well under 1s) with a plain exit code 1 and NO crash/ASAN signature — the raw log just stops mid-file, often right after some earlier, unrelated `test_*.c` group's last line — check `df -h /tmp` before suspecting a real regression.** This sandbox's `/tmp` accumulates `mkdtemp()`-created `xce_*` fixture directories across EVERY session indefinitely (no test file in `testsuite/` cleans up its own `/tmp/xce_*` dirs); once the filesystem backing `/tmp` hits 100% used, `mkdtemp()` fails with ENOSPC inside whichever test file's fixture setup runs next, and several such files (e.g. `test_repl_apply.c`'s `make_test_persist_for_apply()`/`make_test_persist_in()`) hard-`exit(1)` on that failure rather than reporting it through Unity — silently killing the ENTIRE combined-binary test process before any later `test_*` group (including one you just added) gets a chance to run, with no signal distinguishing it from your own new code being broken. Confirmed live at 14731 leftover dirs / 653.6G / 100% used, 0 available — `find /tmp -maxdepth 1 -name 'xce_*' -exec rm -rf {} +` reclaimed 630G (100%→29%) and the exact same binary then ran to completion (1208 tests) cleanly. Also check for and kill any orphaned `controller_unit_test`/`xce_controller` process left over from an earlier invocation in the SAME session (`ps aux | grep controller_unit_test`) before re-running — one was found stuck in `futex_do_wait`, presumably retry-looping a WAL fsync against the full disk, and could confuse a fresh run's process/port state. See buglog.json bug-924 (now updated with this occurrence) for the original, smaller-scale sighting.
 - **This coding-agent harness's interactive shell aliases plain `grep` to a PCRE-capable (ugrep-backed) wrapper via a `.bashrc`-sourced shell FUNCTION — but that function is interactive-shell-only and is NOT inherited by a script invoked as `bash script.sh`, which falls back to whatever the real `grep` binary on PATH is (in this environment: BusyBox, no `-P` support at all).** (2026-08-20, build-time-versioning task 6.2): `echo x | grep -oP '(?<=x)y'` succeeded fine when typed directly as a Bash-tool command, but the identical `grep -oP` call inside a standalone `.sh` file, run via `bash thefile.sh`, failed with `grep: invalid option -- 'P'` (BusyBox usage dump). Confirmed via `type grep` in an interactive call (shows a `_cc_a`-prefixed shell function shelling out to the Claude Code binary itself acting as ugrep) vs. `which -a grep`/`/sbin/grep --version` showing BusyBox as the real underlying binary. **Any shell script file meant to be run standalone (not just typed inline as a Bash-tool command) must assume POSIX-only grep/sed (BRE, `-E`, `-F`, `-o`) — no `-P`/PCRE, no lookbehind/`\K`.** Use `sed -n 's/.../\1/p'` (POSIX BRE, works under BusyBox) instead of `grep -oP '...\K...'` for field extraction inside any committed script.
 - **A direct `go build` (even a real, working one) without `-mod=readonly` can silently rewrite a COMMITTED `go.sum`, adding unrelated indirect-dependency checksum lines as a side effect of resolving the full module graph — even though `go env GOFLAGS` was empty and go.mod itself was never touched.** (2026-08-20, build-time-versioning task 6.2): building `apiserver/cmd/main.go` directly (bypassing the network-hung `go_build` wrapper, see below) added ~65 new `/go.mod`-hash-only lines to `src/frontend/apiserver/go.sum` on the first attempt — caught only via a post-hoc `git status --porcelain` sweep, not by the build itself (exit 0, no warning). Go 1.16+'s `-mod=readonly` default blocks go.mod EDITS but does NOT by itself block go.sum EXTENSION for modules already implied by go.mod's requirements. Fix: pass `-mod=readonly` EXPLICITLY on every ad-hoc `go build` invocation that isn't supposed to touch repo state — confirmed live that the same build still succeeds under explicit `-mod=readonly` with zero go.sum diff, so nothing was genuinely missing; the extra entries were pure side effect. **Any script/agent that runs `go build` directly against a checked-out repo (not via the project's own wrapped build path) must always pass `-mod=readonly` and verify `git status --porcelain` on go.sum/go.mod afterward, every time, no exceptions.**
 - **`src/tools/go_build`'s hardcoded `GOPROXY=https://goproxy.cn` + a fresh-per-invocation `GOCACHE`/`GOPATH` under meson's own `$TMPDIR` sandbox can hang indefinitely in this environment — not fail fast, HANG: an established (not refused/reset) TCP connection to goproxy.cn:443 sits in `futex_do_wait` with near-zero CPU time for 8-10+ minutes across repeated attempts.** (2026-08-20, build-time-versioning task 6.2): `xce-build.sh apiserver <profile> <dir>` (which builds apiserver AND xcecli together via this wrapper) reliably hung on xcecli's fresh module downloads; confirmed via `ss -tnp` showing `ESTAB` to `8.45.176.179:443` (=goproxy.cn, independently reachable fine via plain `curl`/`/dev/tcp` from the SAME shell) with the `go` process's CPU time frozen — a genuinely stuck data path, not slowness, since it never progressed no matter how long left running. Root cause isolated to `go_build`'s own env overrides (`export GOCACHE=$TMPDIR/go-cache; export GOPATH=$TMPDIR/go; export GOPROXY=https://goproxy.cn`), which bypass the ALREADY-POPULATED default `$GOPATH/pkg/mod` cache (`go list -m all` from an ordinary `nix develop` shell, no wrapper, resolved apiserver's full module graph in under a second, zero downloads). **Workaround (do NOT modify go_build/Makefile — this is this specific sandbox's networking quirk, not a real bug in those files): invoke `go build` directly with the default module cache (`GOPROXY=off` is fine since everything needed is already cached) plus the same CGO_CFLAGS/CGO_LDFLAGS pkg-config derivation `test-backend`/apiserver's own meson.build already uses (see bug-1597's documented fix) whenever a real apiserver/xcecli build is needed standalone in this sandbox and `xce-build.sh apiserver`/`make build-server` hang or fail.** Do not blindly increase the timeout and retry — a truly stuck TCP data path does not resolve itself no matter how long you wait; diagnose via `ss -tnp`/`/proc/<pid>/wchan` first.
@@ -164,11 +184,17 @@
 - **Reinforcement of the bug-1615 same-statement `local a=... b="$a..."` gotcha**: recurred independently in a brand-new file (`extract_flag_from_json()`, dirty-content-fingerprint task 6.3, logged as bug-1616) less than a day after being first documented — this pattern is easy to reach for out of habit when writing a "compute one derived value from a parameter" helper. When writing ANY new bash function with `local x="$1" y="...$x..."`, default to splitting the dependent assignment onto its own `local` statement, don't rely on remembering the gotcha in the moment.
 - **scheduler-sim-decision-parity task 1.1 (2026-08-22)**: relocating an `xce_scheduler_t*`-signature function to `xce_sched_core_t*` (the `sched_resolve_job_picker()` precedent, now repeated for `sched_account_jobs()`/`sched_charge_account()`/`sched_uncharge_account()`/`sched_update_account()`/`sched_update_res_accounts()`/`sched_commit_job_alloc()`) has a hidden fan-out cost: EVERY testsuite file that white-box `#include`s the driver/scheduler source and provides its OWN link-satisfying spy/stub for that function (grep the whole `testsuite/` dir for the function name, not just the file(s) named in the task boundary) must have its stub's signature updated too, or the build fails with a type mismatch/duplicate-declaration error — 6 separate test files needed this for `sched_account_jobs()` alone (test_driver_decision_cb.c, test_resval_ownership.c, test_admin_request.c, test_sim_equivalence.c, test_record_lifecycle.c, test_driver_translate.c), none of them named in the task's own boundary list but all mechanically required by it. This project's codebase convention on the relocated parameter's name is to rename it from `sched` to `core` (confirmed via `core.c`'s own `sched_get_job`/`sched_add_worker`/etc., not just keeping the old name with a retyped pointer) — match that, don't just retype in place.
 - **Testing "does function X behave identically given a core from container A vs container B" (Req 1.4-style cross-container proof) does NOT require white-box `#include`-ing the heavyweight driver file that defines the thin iterate-and-call wrapper** (here, `scheduler.c`'s ~20-line `sched_account_jobs()`) if the real accounting logic lives one level down in a lighter, already-provenincludable file (`resource.c`, per `test.c`'s own existing `#include "../src/resource.c"`) — no test in this project's `testsuite/` white-box-includes `scheduler.c` at all (confirmed via grep before writing), because its link-time dependency surface (controller-join, capnp publish, HA candidates) is much larger than `resource.c`'s. Test the real lower-level functions directly instead (`sched_charge_account()`/`sched_uncharge_account()`/`sched_commit_job_alloc()` here) and document in the new test's header comment exactly why the thin wrapper itself isn't independently exercised (already covered elsewhere, e.g. by an existing ordering-proof spy test) — this is honest, still exercises 100% real relocated code, and avoids a huge stub-writing burden.
+- **(2026-08-26) Before treating a fresh HA deploy's anomalous behavior (term jumping impossibly fast, a standby permanently failing to load config, a node never converging with peers) as a real product bug, check for and kill ANY leftover controller/scheduler/worker processes from EARLIER in the SAME session — a same-session soak/investigation left running and forgotten (`ps aux | grep xce_controller`) can bind the exact same default ports (`9991`/`9992`/`9993`, this project's `/xce-test-ha` skill default) a brand-new deploy also uses.** Concretely: an abandoned `xce-ha-chaotic2` 3-node cluster from an earlier bug-1696 investigation, still running hours later, was still bound to 9991-9993 when a fresh `run-ha-soak.sh --sequential` deploy was launched at the same default ports; the new cluster's vote traffic cross-talked with the ancient zombie (which had organically reached a high term over its own long runtime), producing an apparently-impossible term jump (term=1→23 within 50ms of a genuinely fresh first boot) and a standby that could never load cluster_config for the whole test — both fully explained once the zombies were found (`ss -tlnp | grep -E "999[123]"` showing FOUR processes across two unrelated deploys all bound to the same three ports) and killed; a clean re-run on the exact same code, same binaries, same script passed cleanly (`PASS: 2 rounds, 12 jobs, zero job loss; all 3 nodes led`). **Always run `ps aux | grep -i "xce_controller\|xce_scheduler\|xcecd"` (and `ss -tlnp` for the ports you're about to use) before launching a "fresh" HA deploy in this sandbox, especially after any earlier soak/investigation in the same session — a `/xce-test-ha` skill soak that "completed" or whose agent died without explicit cleanup does not guarantee its processes actually exited.** This directly extends the earlier bug-1487-era lesson ("a chaotic-soak verdict is not self-verifying, cross-check the raw WAL") one level further back: an anomalous LIVE symptom from a fresh-looking deploy is not self-verifying either until same-session process/port contamination is ruled out first.
 - **`xce_sched_core_init()` is safe to call twice in the same test process/function to build two independent, simultaneously-alive `xce_sched_core_t` instances for a side-by-side comparison** — established precedent `test_snapshot_save_load.c` (`core_a`/`core_b`) already relies on this; it does re-init a couple of process-global singletons internally (`xstring_table_init()`) but that's a harmless re-init, not a crash/corruption risk, confirmed by that pre-existing test passing. White-box-including `../src/core.c` + `../src/resource.c` (NOT `core_events.c`/`core_session.c`/`decision.c`/`plugin.c`, unneeded if the test only calls `xce_sched_core_init()`/`sched_add_worker()`/`sched_charge_account()`-family functions directly, bypassing the event-push/session-begin/decision-drain APIs entirely) needs exactly 7 link-satisfying no-op stubs: the `kill_sched_job/queue/worker/user/project` family + `kill_sched_resval` (core.c's kvmgr/zarray datakillers) + `plugin_unregister` (core.c's plugins-kvmgr datakiller) + `plugin_prepare_allocation`/`sched_ctx_get_uncomit_host` (resource.c's `sched_try_alloc_on_host_()`'s own callees, needed only because that whole file is pulled in, even though this test path never calls it) — the last two are the exact same stub pair `test.c`'s own resource.c-only white-box test already establishes needing.
 - **Never use `date -d` with a RELATIVE expression in this environment (`-7 days`, `+5 days`, `last monday`) — `date` first on PATH is BusyBox v1.36.1, and its failure is silent-by-default rather than loud.** (2026-08-23, `/weekly_report` skill authoring, bug-1628): BusyBox date *does* parse absolute dates (`date -d 2026-06-29 +%A` → `Monday`), which masks the gap during casual probing, but rejects all relative arithmetic with `invalid date` **on stderr** — so `START=$(date -d "-7 days" +%F)` captures only stdout and yields an EMPTY string. The empty value then flows into `git log --since="" --until=""`, which git treats as *no time filter at all* (unbounded history) instead of erroring, so a broken date computation silently produces a plausible-looking-but-wrong result. Note `/bin/date` IS GNU coreutils 9.8 — the nix profile bin dir simply precedes `/bin` on PATH. **Do date arithmetic in `python3` (`/bin/python3`, always present) via `datetime`/`timedelta`, emitting `VAR=value` lines consumed by `eval "$(python3 - <<'PY' ... PY\n)"`;** call `/bin/date` by absolute path only if GNU date is genuinely required. Corollary that generalizes beyond dates: whenever a computed shell variable feeds a `git log --since/--until`-style range filter, assert it is non-empty first — git's "empty arg means no filter" behavior turns an upstream failure into silent wrong output rather than a crash. Same BusyBox-shadows-GNU family as the `grep -P` and `mktemp` entries above.
 
 - **A scheduler resource-accounting test fixture that lets a job's `xce_res_account_t.residx` resolve LAZILY (left at -1, relying on `sched_update_res_accounts()`'s own first-use `key_find()`/`add_vres()` fallback) can SEGV inside `zarray_get_ex()` instead of failing loudly.** (2026-08-23, scheduler-dispatch-commit task 2.1, bug-1629): a minimal test fixture that never pre-registers a resource name in `core->resmgr` (no real resource-config load, unlike production) hit a silent failure path — `sched_update_account()` discards `sched_update_res_accounts()`'s return value entirely, so if that inner call's `key_find()`/`add_vres()` resolution doesn't complete, `residx` silently stays `-1` with no error surfaced anywhere up the call chain (`sched_commit_decision()` still returns success). `zarray_get_ex()` (`lib/zarray.c:332`) only guards `idx >= array->caps`, NOT `idx < 0`, so passing that leftover `-1` to `get_resvals()`/`put_resvals()` reads `array->data[-1]` — an underflow read ASAN reports as a high-address SEGV, not a clean bounds error. Fix: in any test fixture exercising real charge/accounting code without loading real resource config, pre-register the resource explicitly first via `add_vres(&vres, core.resmgr, name, 0)` and use its returned (non-negative) index for `get_resvals()`/`put_resvals()`, rather than trusting the production lazy-resolution path to run to completion inside a minimal fixture.
 
+- **(2026-08-26, bug-1715) Running xceadmin's pytest suite under `nix develop --command bash -c "... PYTHONPATH=src python3 -m pytest ..."` with a bare `PYTHONPATH=src` (no `:$PYTHONPATH`) silently DISCARDS the whole nix-provided PYTHONPATH (pycapnp/click/jsonschema/pytest itself all live there, per the PYTHONPATH-entries Key Learning above) instead of extending it — fails with `No module named pytest`, easy to misread as "pytest isn't installed in this devshell" when it actually is.** Always use `PYTHONPATH=src:$PYTHONPATH` (prepend, don't replace) when adding `src/xceadmin/src` to the path for a one-off `python3 -m pytest`/script invocation inside `nix develop`.
+- **(2026-08-26, bug-1715) The AUTHORITATIVE, complete list of every literal `sched_admin_reply_t.error = "..."` string the real controller/scheduler can send (confirmed by grepping `src/controller/` and `src/scheduler/` for the assignment, then reading each site's surrounding code — not from requirements.md's own worked examples, which under-lists them):** transient/connection-timing (self-resolving, safe to retry): `"scheduler disconnected"` (cluster.c:1062, bug-1693's disconnect path), `"scheduler not connected"` (cluster.c:958, `c->sched.ep==NULL`), `"request timeout"` (cluster.c:1078, `SCHEDADMIN_TIMEOUT_MS` wait expired), `"server busy, try again"` (cluster.c:937, `vtaskmgr_blocking_acquire()` admission-reservation exhaustion — NOT previously documented as transient anywhere, found by reading the admission-gate comment, not the string alone). Permanent/domain (must never retry): `"already recording"` / `"not recording"` (driver.c:390/426, `rc==-EALREADY`) and `validation_error_text()`'s six filename/path-rejection strings (driver.c:298-316: `"invalid record file name"`, `"record directory not found"`, `"record directory path is not a directory"`, `"permission denied accessing record file"`, `"a non-file entry with this name already exists"`, `"record file path too long"`), plus a dynamic (non-literal, can't exact-match) `"cannot write to %s: %s"` I/O-failure message (driver.c:398-404). When classifying an error-reply reason string as retry-safe vs not, read the ERROR-SITE's surrounding code/comments, not just the string's English wording — "server busy, try again" was only correctly classified transient by reading `vtaskmgr_blocking_acquire()`'s admission-gate design comment, not from requirements.md.
+- **(2026-08-26, xce-test-rc-file task 3.3, bug-1719) `src/tools/xce-build.sh <component>` with NO `--incremental` does a full `rm -rf "$BUILD_DIR"` + fresh `meson setup` by design (per its own top-of-file comment), then `meson compile` only rebuilds the ONE requested component — so building a second component (e.g. `scheduler`) into an `xce.build` dir that already has a different component's binaries (e.g. `controller`'s `xce_controller`/`xce_test`) silently WIPES the earlier ones.** Always pass `--incremental` when you need a component built alongside another already-built one in the same dir, and never pass an explicit build_dir positional arg unless it matches the script's own default relative-path resolution (`xce.build`, resolved from `src/tools/xce-build.sh`'s own cwd) — passing a repo-root-relative path like `src/xce.build` as the 3rd positional creates a bogus nested `src/src/xce.build` instead (harmless/gitignored via meson's own per-builddir auto `.gitignore: *`, but wastes a build cycle). Rebuilding the wiped component with `--incremental` restores it without disturbing the other.
+- **(2026-08-26, xce-test-rc-file task 3.3, bug-1720) `xce-test-resources`'s apiserver layer reporting UNREACHABLE for every resource does not by itself mean a propagation regression — check for a stale bind conflict on :8080 first (`lsof -i :8080` / `pgrep -af apiserver`).** An orphaned `/tmp/apiserver` process from an unrelated, long-concluded prior session (its own deploy dir already deleted, 7.5h uptime) was still bound to :8080, so THIS run's fresh apiserver instance failed `listen tcp :8080: bind: address already in use` and never logged anything — surfacing identically to a genuine trace-line-never-appeared UNREACHABLE. `run-resource-verify.sh`'s own trap-based cleanup only reaps processes IT started, not orphans left by an earlier, already-ended unrelated script/session.
+
 ## Decision Log
 - HA/consistency model follows Raft + CP strictly: majority ACK is the hard commit barrier for all state-changing writes (reject on timeout, never local-apply-and-degrade); the own-term commit rule (not a 4th "COMMITTING" state) is what gives a new leader safety.
 - A controller process must NEVER self-exit/self-abort as a designed response to an operational condition (only a genuine crash or admin/supervisor kill) — an unrecoverable/fenced member logs loudly, stays permanently non-promotable, stays alive, and keeps retrying.
@@ -197,3 +223,7 @@
 - **A buglog entry's own "root cause: NOT YET ROOT-CAUSED" (or similarly hedged) status is a real signal to re-derive the causal claim from primary evidence before citing it as settled, especially when later re-occurrences of the SAME symptom show it firing on completely unrelated diffs.** bug-1429 (cited by `config_repl.c`/`ha_convergence.c`'s catchup/demotion offload-flag comments as "the vtask pool that deadlocked controller_unit_test on 2026-07-10") is itself logged as never root-caused, and its OWN later occurrences (2026-08-20, 2026-08-23) reproduced the identical TIMEOUT/hang signature on diffs that touched neither the vtask offload code nor persist.c's checkpoint path at all — "strongly corroborates this is a pre-existing, systemic, timing/order-dependent hang in the controller_unit_test harness," per the buglog's own later entries, not proven caused by VTASK_LIVE_BLOCK dispatch specifically. Don't repeat a downstream comment's causal claim as fact in a new design doc/comment without checking the buglog entry it cites — the comment can be more confident than the evidence it's citing.
 - **`controller_unit_test` (the main shared meson.build `controller_unit_test_src` binary) does NOT link the real `sched.c`/`dataex.c` at all** (bug-1693, 2026-08-25) — `sched.c`'s own `on_scheduler_join()`/`collect_confvars()` and `dataex.c`'s `notify()` collide at link time with fake local definitions several OTHER files in that same binary already provide (`test_commit_barrier_reply.c`, `test_worker_join.c`), so `test_config_repl.c` instead ships a trivial no-op `handle_scheduler_leave(cluster_t *c) { (void)c; return 0; }` stub purely to satisfy the linker. **Any new/changed behavior added to a real `sched.c` function is therefore SILENTLY UNTESTED by any `controller_unit_test`-based test that calls that function by name** — the call resolves to the stub, not the real implementation, with no compile/link error to flag the mismatch (confirmed empirically: a new regression test asserting sub-second wake timing after calling `handle_scheduler_leave()` passed compilation and linking fine, then failed by taking the full 5s deadline — the stub had silently swallowed the call). `nm '<obj>.o' | grep ' T <symbol>$'` across every `.o` in the target's build dir is the fast way to confirm which TU actually provides a given symbol in a given binary before trusting a test's result. The two standalone binaries that DO link the real `sched.c` (`sched_push_unit_test`, `sched_worker_change_unit_test` — see their own `meson.build` comments for the collision rationale) don't link `cluster.c`, so neither binary alone can exercise a fix that spans both files for real; bug-1693's fix split verification across both (a direct-call test against `cluster.c`'s real new function in `controller_unit_test`, plus a small counting-spy test in `test_sched_worker_change.c` proving `sched.c`'s real caller invokes it) rather than inventing a third binary that links both (would need to duplicate most of `controller_unit_test_src`'s already-large dependency list to satisfy `cluster.c`'s own huge fan-out). General lesson: when a fix spans a function in `sched.c`/`dataex.c` AND a function in any of the ~15 other files real-linked into `controller_unit_test_src`, check with `nm`/grep for a same-name fake stub in a sibling test file in that binary BEFORE trusting a test that calls the `sched.c`/`dataex.c` side by name — it may be exercising a stub, not your change.
 - **(2026-08-25, bug-1695) A bootstrap "is a leader already known" fallback check must compare a term against a snapshot taken at THIS boot's start, never against a bare constant like `> 0`, whenever the compared value is restored from durable on-disk state that only ever increases (Raft terms, monotonic counters, etc.).** `myapp_run()`'s `raft.term > 0` fallback was trivially true forever after the cluster's first-ever election because `ha_restore_term()` restores a persisted, never-decreasing term at every boot. Fixed by capturing `term_at_boot = ha->raft.term` immediately after `ha_init_raft()` seeds it (right after restore, strictly before any new activity could legitimately advance it) and comparing `> term_at_boot` instead of `> 0` — the general pattern: "is X known/true" checks against monotonic restored state should ask "did X change SINCE I booted," not "is X nonzero," whenever a nonzero-forever steady state is a normal, expected outcome of prior successful operation (not an error state).
+- **(2026-08-25, bug-1697, CRITICAL — live, no-restart data bug) Whenever a semantic change makes one code path (e.g. a scheduler dispatch decision) start ANTICIPATING a state transition another, later code path (e.g. a worker's ack) used to be the sole author of, re-audit EVERY `result == JOB_TRANSITION_APPLIED`-gated (or equivalent "did this actually change anything" gated) side effect on that later path — the earlier path's own transition now permanently makes the later one a same-state no-op, silently killing every effect gated on "this was a real transition."** `dispatch_a_job_()` (job.c:1686, scheduler-dispatch-commit task 3.1) was changed to set `state = JOB_STAT_RUN` synchronously at DISPATCH time (the controller now decides RUN, it no longer waits for a worker ack) — but `apply_committed_entry()`'s WAL_JOB_START case (added by bug-1687, persist.c ~4594-4637) still gated its `start_time`/`pidinfo` wire-decode adoption on `result == JOB_TRANSITION_APPLIED` from `apply_job_update_transition()`, which now ALWAYS returns `JOB_TRANSITION_NOOP_DUPLICATE` for WAL_JOB_START (job.c:2795-2799, state already == RUN from dispatch) — so start_time/pidinfo silently never get set, for every job, live, no restart needed. The exact same class of bug had ALREADY been fixed for WAL_JOB_RUSAGE/CKPT/PREASON in this identical function (their own comments describe the identical hazard) — a fix for one wal_type in a shared-transition-guard case group does not imply sibling types sharing the SAME guard got the same fix; grep the whole case group whenever changing which code path is allowed to "arrive first" at a shared target state. bug-1679's own comment (job.c:1709-1719) had ALREADY independently hit and worked around the identical root collision for a different symptom (apiserver update-notification hook not firing) months/fixes apart from this one — the two were never connected as one root cause until a live "start_time is always 0" report forced re-deriving it from scratch. **General technique: after any change that makes state transition X reachable from a NEW, earlier code path, grep every existing consumer of "was this transition genuinely applied" (`result == *_APPLIED`, `state != cur`, etc.) that used to assume X's only author was the OLD path — each is a candidate for silently going permanently dead.** **FIXED (2026-08-26):** moved `start_time` ownership to the dispatch-time mutation itself (`WAL_JOB_DISPATCH`'s payload extended with the new `WAL_JOB_DISPATCH_HAS_START_TIME_FLAG` `_pad` bit, mirroring `WAL_JOB_FAIL_STAGE_FLAG` — NOT a length heuristic, since `JobAlloc`'s own variable length makes old-vs-new payload length ambiguous); `pidinfo` stays sourced from the worker's ack but its adoption is now unconditional like RUSAGE/CKPT/PREASON. RED confirmed by temp-reverting only the two defect lines (job.c's `start_time` stamp; persist.c's pidinfo `result==APPLIED` gate) and rebuilding — 5 genuine new failures at the exact production call sites (`dispatch_a_job_test`, `apply_committed_entry` seeded already-RUN). GREEN confirmed after restoring: "1198 Tests 3 Failures 0 Ignored" 3x direct runs, matching known baseline exactly.
+- **(2026-08-26, bug-1718, xce-test-rc-file task 3.1) A `python3 -` (or any "read program from stdin") invocation placed on the SAME command as a `<<HEREDOC` redirect while ALSO receiving piped input (`upstream | python3 - args <<EOF ... EOF`) silently loses the piped data, no error.** The heredoc redirect wins stdin over the pipe for that command; `python3 -` consumes the heredoc's content as its own program source, so by the time the script's own `sys.stdin.read()` runs, stdin is already at EOF — it just returns `""` with zero diagnostic. Caught by re-reading the draft before ever running it, not by a failing run. **Fix pattern: write the embedded program to its own temp file ONCE (`mktemp` + `cat >"$DRIVER" <<EOF...EOF`, cleaned via an `EXIT` trap), then invoke it as `upstream | python3 "$DRIVER" args...`** — a file argument doesn't compete with the pipe for stdin, unlike `-`. Applies to any interpreter's "read program from stdin" mode (`python3 -`, `perl -e -`, etc.) combined with a heredoc AND a real piped payload in the same pipeline.
+- **(2026-08-26, xce-test-rc-file task 3.1) `xce_test`'s `cmd_set(argc==1)` (`set` with no args) is the generic, no-live-cluster-required readback for ANY setting** (`test.c:3684`, format `"%-12s:%s default: %s\n"`, `KVAL()` prints `"<not set>"` for a NULL current value) — the right primitive for any integration test that needs to observe an rc-file/`set`-driven setting change (e.g. `masters`) without a live controller connection. Don't hardcode the exact column padding when grepping its output (fragile) — extract via `sed -E 's/^masters[[:space:]]*:(.*) default:.*/\1/'` (greedy `(.*)` correctly captures the embedded-space `"<not set>"` sentinel too, since there's only one literal `" default:"` substring per line to anchor against).
+- **(2026-08-26, xce-test-rc-file task 3.1) `xce_test` uses `libedit`'s `readline()`/`isatty(STDIN_FILENO)`-gated `g_interactive`, so a genuine interactive-mode integration test needs a REAL pty, not a piped stdin** — confirmed working via a small embedded Python `pty.fork()` driver (write commands to the master fd with a short settle delay after each, drain with `select()`, detect child exit via non-blocking `waitpid`, `os.waitstatus_to_exitcode()` for the real exit code). Same technique task 2.2's own implementer used for manual verification; this task turned it into a reusable scripted harness. The prompt itself (`fill_prompt`, top-level context) renders as literal `"xce>"` — grepping for that substring in the pty transcript is a good "this really was a TTY session, not a piped fallback" sanity check, distinct from the actual assertion under test.