docs(steering): sync steering + xce-audit-invariants skill after Jul 13-18 work

- ha-async-io.md: document the same-batch endpoint lifetime hazard
  (bug-1447) and its ref-whole-batch-before-dispatch fix pattern
- xce-audit-invariants SKILL.md: add object-lifetime-within-a-dispatch-batch
  as a 6th audit dimension, so a future pass checks other batch loops
  (vtaskmgr drain, peer-list iteration) for the same shape
- product.md: replace the stale "HA consistency family is the current
  focus" framing with the actual 2026-07-18 status (umbrella complete,
  9-spec validate-impl batch closed, job-array-single-wal-record shipped)
- roadmap.md: correct 5 entries that were stale relative to actual spec
  status (ha-consistency-invariants/ha-wal-replication/libxce-failover
  pulled out of "In Flight", ha-term-safety and raft-drive-async marked
  implemented+GO, raft-sim-conformance and job-array-single-wal-record
  given closure annotations) -- struck through, not deleted, per this
  file's own additive-update convention
- testing.md: document that meson's aggregate Fail:0 can mask Unity
  sub-group failures -- always check the standalone binary's own summary
- src/controller/CLAUDE.md: fix a stale line-citation for the votedFor
  atomicity invariant (had drifted onto unrelated post-election code)

.claude/skills/xce-audit-invariants/SKILL.md

@@ -66,6 +66,16 @@ have historically found real violations are:
    any ingest path (standby WAL receipt, catch-up, heartbeat)?
 5. **Error handling** — silent swallowed errors, missing `ReturnError`, or a
    fail-open where the invariant docs require fail-loud-and-fence.
+6. **Object lifetime within a dispatch batch** (new 2026-07-18, bug-1447) —
+   any loop that processes a batch of items fetched in one call (epoll
+   `events[]`, a vtask drain, a peer-list scan) where one item's handler can
+   free/unref an object that a LATER item in the SAME batch still references.
+   `iomgr_unmonitor()`/`endpoint_unref()` only affects *future* dispatch, not
+   an event already sitting in the current batch array — see `ha-async-io.md`
+   "Same-batch endpoint lifetime" for the reference shape (`io_worker_dispatch_batch`'s
+   ref-whole-batch-before-dispatch fix). Check other batch-processing loops
+   (vtaskmgr's drain, any peer-list iteration that can trigger eviction) for
+   the same shape, not just `io_worker`.
 
 Each subagent must report file:line evidence for every claimed violation —
 no speculative findings. Use the Agent tool directly (general-purpose or

.kiro/steering/ha-async-io.md

@@ -30,6 +30,8 @@ No thread that carries liveness signals (iomgr, delaymgr) may execute blocking w
 
 Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.
 
+**Same-batch endpoint lifetime (bug-1447, fixed 2026-07-18).** `io_worker`'s epoll loop processes every fd from ONE `epoll_wait()` batch (`events[MAX_EPOLL_EVENTS]`) in a single pass. A handler for one event in that batch (e.g. `on_standby_connected` evicting a stale duplicate outbound channel via `iomgr_unmonitor()`+`endpoint_unref()`) can free a DIFFERENT endpoint that ALSO has a ready event later in the SAME batch array — `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, it does not invalidate an event already sitting in the current batch. The later event then dereferences a freed `endpoint_t`/`SSL*` (heap-use-after-free). `INVOKE_MSG_HANDLER`'s ref/unref bracket only protects the endpoint a handler is CURRENTLY running on, not a different endpoint torn down as a side effect. Fix: `io_worker_dispatch_batch()` takes `endpoint_ref()` on every endpoint in the batch before dispatching any of them, and `endpoint_unref()`s each only after the whole batch completes — same lock-free bracket idiom as bug-1298, applied to the batch as a unit. When adding any handler that can tear down an endpoint other than its own, this batch-wide ref bracket is why that's safe — do not "optimize" it away as redundant with `INVOKE_MSG_HANDLER`'s per-call ref.
+
 ## Connection Ownership
 
 **Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

.kiro/steering/product.md

@@ -39,13 +39,26 @@ The project is in a **persistence and HA transformation** (branch `persistent_ha
 
 Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
 Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
-Active specs (15 total). The HA consistency family is the current focus:
-ha-consistency-invariants (umbrella contract) decomposed into ha-commit-engine,
-ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery.
-Supporting/foundational specs: ha-config-bootstrap, ha-heartbeat-liveness,
-ha-peer-connection-reuse, ha-repl-hardening, ha-wal-replication, raft-library,
-election-iomgr-refactor, scheduler-controller-msg-gap, worker-join-refactor,
-resval-id-decoupling. See `roadmap.md` for dependency order and status.
+
+**Status as of 2026-07-18**: the ha-consistency-invariants umbrella (ha-commit-engine,
+ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) is
+COMPLETE (closed 2026-07-13). The Raft TLA+ Conformance family (raft-library,
+raft-log-replication, raft-commit-safety, raft-sim-conformance) and its
+XCE-integration umbrella ha-term-safety are implemented + validate-impl GO.
+job-array-single-wal-record shipped complete (2026-07-17, including a
+user-directed residual-gap closure pass). A backlog sweep (2026-07-18) ran
+`/kiro-validate-impl` on 9 specs whose tasks were 100% done but had never
+been formally closed — raft-drive-async, ha-wal-replication,
+ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle,
+ha-entity-snapshots, ha-commit-durability, libxce-failover,
+raft-sim-conformance — all now GO. That sweep surfaced one real bug
+(bug-1447: same-batch endpoint use-after-free in `io_worker`'s epoll
+dispatch loop, see `ha-async-io.md`), root-caused and fixed the same day.
+
+Remaining backlog (requirements-only or design-pending, not yet implemented):
+admin-config-tool (spec finalized), scheduler-framework, config-upgrade-snapshot
+(blocked on entity-bearing snapshots). See `roadmap.md` for dependency order
+and status.
 
 Key design decisions already locked:
 - No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)

.kiro/steering/roadmap.md

@@ -7,9 +7,9 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
 ## In Flight
 
 - **scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`. Dependencies: none hard (builds on existing plugin framework + select expression evaluator). Next: `/kiro-spec-requirements scheduler-framework`.
-- **ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.
-- ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it
-- **libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:568-569`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).
+- ~~**ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.~~ **COMPLETE 2026-07-13** — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
+- ~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it~~ **task 5.4 CLOSED 2026-07-18** via `/kiro-validate-impl`'s live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
+- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:568-569`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this.
 
 ## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)
 
@@ -17,9 +17,9 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
 
 - [x] raft-log-replication — *(specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.)* Extend the library to OWN the replicated log. TLA+ actions: `ClientRequest`, `AppendEntries`, `HandleAppendEntriesRequest`, `UpdateTerm`-on-AppendEntries. Per-entry term in the library log model (closes **G1**); `AppendEntries` `logOk` = `prevLogTerm` match (**G2**); conflict branch deletes the divergent `(term,lsn)` suffix (**G3**); ANY higher-term RPC incl. AppendEntries/heartbeat advances `currentTerm` and steps down (**G5**). New pure callbacks: `on_persist_entry`, `on_get_entry_term(lsn)`, `on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx)`, `on_truncate_from(lsn)`; controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
 - [x] raft-commit-safety — *(specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.)* `AdvanceCommitIndex` with the Figure-8 current-term rule (commit `Max(agreeIndexes)` only if `log[Max].term == currentTerm`) (**G4**); `RequestVote`/PreVote freshness becomes `(lastLogTerm, lastLogIndex)` instead of LSN-only (**G6**). New callback `on_commit_advanced(lsn)` feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
-- [x] raft-sim-conformance — *(specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.)* Extend `raft_sim.c` into a conformance/verification harness driving the library through `raft.tla`-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of `raft.tla`. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety.
+- [x] raft-sim-conformance — *(specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.)* Extend `raft_sim.c` into a conformance/verification harness driving the library through `raft.tla`-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of `raft.tla`. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. **IMPLEMENTED, validate-impl GO 2026-07-18** (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — `raft_tla` — is named for).
 
-> **ha-term-safety** (requirements + design generated 2026-06-21; `.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. Next: `/kiro-spec-tasks ha-term-safety`.
+> **ha-term-safety** (`.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. **IMPLEMENTED, validate-impl GO 2026-07-17** (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).
 
 ## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`)
 
@@ -34,7 +34,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 
 ### Specs (dependency order — Raft Drive & Apply Async family)
 
-- [ ] **raft-drive-async** *(START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.**
+- [x] **raft-drive-async** — **IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in `io_worker`'s epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see `ha-async-io.md`; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. *(was: START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.**
 - [x] raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single `apply_committed_entry()` seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). **Contains P1 (commit-barrier async) as its keystone.** Commands (WAL_MODE_WAIT) apply-after-commit; **facts (WAL_MODE_RETAIN) routed through commit-then-apply too** — D1 RESOLVED (block-but-durable under quorum loss); formalized as `ha-consistency-invariants` criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: `.kiro/specs/raft-apply-decoupling/brief.md` + `facts-in-raft.md`. **IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO** — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on `persist_t`, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned `repl_submit_majority_commit()` was never built; apply pump wired via `commit_resolve_pending()` not `ha.c`'s `on_commit_advanced`; fact hooks collapse to generic `HOOK_DETAIL_JOB_UPDATE` not per-type); task 5.3 touched `ha.c` without a formal `_Boundary:_` exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous `jobsub` path (resembles but not confirmed as bug-1416's class), and `trim_wal_segments`'s retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
 - [ ] worker-rejoin-resync — **elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Already on the backlog below; pulled forward into this family. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards).
 - [ ] job-array-single-wal-record — **PROMOTED from backlog 2026-06-24.** Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: `.kiro/specs/job-array-single-wal-record/brief.md`.
@@ -65,7 +65,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
 - [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
 - [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) remains blocked on `entity-bearing-snapshots` (pre-existing). Next: `/kiro-spec-requirements admin-config-tool`.
 - [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
-- [→] job-array-single-wal-record — **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
+- [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
 - [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships.
 - [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
 - [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).

.kiro/steering/testing.md

@@ -75,6 +75,10 @@ extern void reset_ha_state(ha_state_t *s);
 
 **Add a test case in `testsuite/` before modifying any C implementation.**
 
+## Verifying a Test Run Actually Passed
+
+**Meson's aggregate `Ok:N Fail:0` is not sufficient evidence.** Meson can report a suite `Fail:0` while Unity sub-groups inside it fail — meson only tracks the process exit code of each *meson test unit*, not each Unity `RUN_TEST`. Always cross-check the **standalone test binary's own stdout** (`controller_unit_test`, `xcelib_test`, etc., run directly) for its authoritative `:FAIL:` count / `N Tests M Failures` summary before treating a run as green. This was the root cause of a masked-failure bug in the Unity harness itself (nested `UNITY_BEGIN()`/`UNITY_END()` across 83 `controller/testsuite/*.c` files swallowed the real exit code — fixed 2026-07-17) and is now the standard verification step for every `/kiro-validate-impl` pass.
+
 ## Coverage Targets
 
 - Go: 80% minimum on business logic (controllers, services, store)

src/controller/CLAUDE.md

@@ -95,7 +95,7 @@ The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogL
 ### Protocol Verification Checklist
 
 When modifying HA protocol code (election, replication, heartbeat), verify:
-- [ ] `votedFor` is updated atomically with term increment before granting any vote
+- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:1784-1824`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:477` → `raft_on_vote_request`. Corrected 2026-07-18 — the prior `ha.c:1218-1230` citation had drifted onto unrelated post-election code.)
 - [ ] `currentTerm` is monotonic — never decreases
 - [ ] `commitIndex` never decreases (monotonic)
 - [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)