myxce/.kiro/specs/ha-term-safety/tasks.md

# Implementation Plan

> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/controller/testsuite/` (e.g. `test_term_safety_adapter.c`) that fails before the controller change and passes after. Build/run via `/xce-build controller` (builds lib + controller + `controller_unit_test`, ASAN).
>
> **Spec boundary** (applies to ALL tasks): this spec is the **controller-side thin callback adapter + config marker + live regression** over the raft library (`src/lib/raft/raft.{c,h}`), consumed **verbatim**. It does NOT re-implement any Raft protocol decision (logOk, `(term,index)` truncation, current-term commit, freshness, UpdateTerm) — those are owned by `raft-log-replication` + `raft-commit-safety` (both landed, library-side). It does NOT change the commit-barrier / persist-before-ACK durability mechanism (`ha-commit-durability`) — `on_commit_advanced` only TRIGGERS it. The on-disk `wal_entry_hdr_t` (40 bytes) is unchanged — term is a value fix, not a layout change (R1.7). The full entity-bearing config-upgrade snapshot is OUT of scope (`config-upgrade-snapshot`).
>
> **Depends (HARD): `raft-log-replication` + `raft-commit-safety`** — needs the five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`), the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), `commit_index`/`next_index[]`/`match_index[]`, and the extended `(last_log_term, last_log_index)` vote signatures already present in `raft.{c,h}`.
>
> **Build-restoring note:** the upstream vote-signature extension currently breaks the controller build (`cluster.c` vote callers use the old arity). Task 1.2 restores the build; until 1.2 lands, `/xce-build controller` will not compile (`/xce-build lib` stays green).

## 1. Foundation: wire format, WAL term seam, recovery read-back

- [x] 1.1 Extend the AppendEntries replication envelope to carry the Raft term fields
  - Add `prev_log_index`, `prev_log_term`, and the entry's `term` to the AppendEntries-carrying repl envelope so a standby persists the leader's term, not a local substitute; regenerate the Cap'n Proto bindings
  - Observable completion: a replicated entry's envelope round-trips `(prev_log_index, prev_log_term, entry_term)` across send/decode (unit-asserted on a built envelope); existing replication messages still decode
  - _Requirements: 1.3, 1.4_
  - _Boundary: repl transport / typemap (Cap'n Proto)_

- [x] 1.2 Extend the RequestVote/PreVote wire with `(last_log_term, last_log_index)` and restore the controller vote call sites
  - Add `last_log_term` (and `last_log_index`) to the RequestVote/PreVote wire message; regenerate bindings; update the controller's vote and pre-vote intake to decode the candidate's tuple and pass it to the extended library `raft_on_vote_request`/`raft_on_pre_vote_request`, sourcing the voter's own tuple via the entry-term read-back at `log_len`
  - Observable completion: `/xce-build controller` COMPILES AND LINKS again (the upstream signature break is resolved); a vote intake decodes a candidate `last_log_term` and the library applies the logOk freshness gate
  - _Requirements: 2.3, 2.4, 2.6_
  - _Boundary: repl/vote wire (Cap'n Proto), controller vote intake_
  - _Depends: 1.1_

- [x] 1.3 Add the WAL term seam, recovery read-back, and the legacy-zero-term matching rule
  - Implement the `persist_set_term` seam (sets the controller's live election term) and a per-LSN recovered term index built during WAL replay, plus an accessor returning the recorded term for an LSN (0 when absent); preserve the 40-byte on-disk header (value fix only)
  - Pin the legacy-zero-term rule (REVIEW ISSUE 2): a pre-feature entry carrying term 0 maps to a single, precisely-defined "lowest-known term" for `(term,lsn)` matching so recovery never fails and a legacy entry is never spuriously treated as divergent from another legacy entry; document and test the exact mapping
  - Observable completion: after replaying a WAL with mixed real and legacy-0 terms, the accessor returns the recorded term for real entries and the defined lowest-known term for legacy-0 entries; header size unchanged
  - _Requirements: 1.1, 1.5, 1.6, 1.7, 1.8_
  - _Boundary: persist (WAL I/O + term value)_

## 2. Core: the five callback adapter bodies

- [x] 2.1 Implement `on_persist_entry` over the WAL write path
  - Stamp `hdr.term` from the library-supplied term (via `persist_set_term`) at the given 1-based index == LSN, through the existing async/replicated WAL write; preserve persist-before-ACK and the bug-850 contiguity guard beneath it
  - Observable completion: a library append/accept persists an entry whose recovered `hdr.term` equals the library term; persist failure propagates a non-zero rc and does not ACK
  - _Requirements: 1.1, 1.2, 1.3, 1.4_
  - _Depends: 1.3_

- [x] 2.2 Implement `on_get_entry_term` and `on_truncate_from`
  - `on_get_entry_term` answers from the recovered/live per-LSN term index (legacy-0 rule from 1.3); `on_truncate_from` deletes entries at index..end via the existing LSN truncate, replacing the committed-boundary heuristic
  - Observable completion: the library reads back a stamped entry term via the callback; a truncate-from call removes the local suffix at and above the index and leaves the prefix intact
  - _Requirements: 1.5, 1.8, 3.1, 3.2_
  - _Depends: 1.3_

- [x] 2.3 Implement `on_send_append_entries` over the repl envelope
  - Build and enqueue an AppendEntries on the repl transport carrying `prev_log_index`, `prev_log_term`, the entry's `term`, and `leader_commit = commit_index`; a heartbeat carries zero entries
  - Observable completion: emitting to a peer produces an envelope with the correct prev tuple and entry term; a 0-entry heartbeat is emitted when there is nothing to send
  - _Requirements: 1.3, 3.1_
  - _Depends: 1.1_

- [x] 2.4 Implement `on_commit_advanced` as a trigger into the existing barrier, and resolve the commit-gate ownership
  - Feed the library-supplied `commit_index` into the existing synchronous commit barrier (`commit_advance_locked`/`commit_set_writable_lsn`) as the commit SOURCE; the barrier mechanism (majority self-cap at durable floor, persist-before-ACK) is unchanged
  - Resolve double-gating (REVIEW ISSUE 3): state and implement whether the controller's existing own-term `WAL_NOOP` writable gate is now redundant with the library's current-term commit rule (and removed) or retained as a complementary durability gate; ensure the two cannot deadlock (commit held by both) or leave a gap
  - Observable completion: a strict library commit advance releases the barrier's WAIT only on durable majority; the term-blind safe-commit is no longer the commit source; a unit test pins the own-term-gate decision
  - _Requirements: 2.2, 4.1, 4.5, 5.4_
  - _Depends: 1.3_

- [x] 2.5 Wire the five callbacks into the controller callback set and the library init restore
  - Add the five callbacks to the controller's `s_real_cbs`; on `raft_init` restore `initial_log_len`, `initial_commit_index`, and the recovered last-log term/index from the WAL so the library boots with the durable floor
  - Observable completion: a controller node initializes the raft library with all required callbacks non-NULL and restored indices matching its recovered WAL; existing election callbacks remain wired
  - _Requirements: 1.5, 2.1_
  - _Depends: 2.1, 2.2, 2.3, 2.4_

## 3. Integration: drive the library entry points on the controller paths

- [x] 3.1 Drive `raft_log_append` on the primary append path
  - Route the primary's job-state append (and the own-term promotion no-op) through `raft_log_append` so the library stamps the term and assigns the LSN, firing `on_persist_entry`
  - Observable completion: a primary `jobsub`-driven append goes through the library and lands in the WAL with the live term stamped; the promotion no-op records the newly established term
  - _Requirements: 1.1, 1.2, 2.1_
  - _Depends: 2.5_

- [x] 3.2 Route heartbeat/replication intake through `raft_on_append_entries_request` (replaces term-blind intake + committed-boundary reconcile)
  - Replace the term-handling in `ha_consume_heartbeat_committed` and the committed-boundary truncate in `ha_rejoin_reconcile` with routing every incoming heartbeat/replication (0- or 1-entry AppendEntries) through the library so `UpdateTerm` (G5), logOk (G2), and `(term,lsn)` conflict-truncate (G3) happen via the callbacks; keep the loud non-promotable retry fence for an unreconcilable member
  - Observable completion: a higher-term heartbeat makes the node adopt the term and step down (term recorded); a divergent same-LSN tail is truncated and reconverged; an unreconcilable member stays non-promotable and keeps retrying (no false convergence)
  - _Requirements: 2.3, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
  - _Boundary: ha intake + ha_convergence_
  - _Depends: 2.5_

- [x] 3.3 Drive `raft_on_append_entries_reply` and `raft_advance_commit` on the ACK path
  - On a replication ACK, route it through `raft_on_append_entries_reply` to update `match_index[]`/`next_index[]`, then drive `raft_advance_commit` so the library recomputes the current-term-gated commit and may fire `on_commit_advanced`
  - Observable completion: a quorum ACK advances the library `commit_index` only when the agreed entry is current-term, which then triggers the barrier; a stale-term reply is dropped
  - _Requirements: 4.1, 4.5_
  - _Depends: 2.5, 3.2_

- [x] 3.4 Retire the term-0 `repl_broadcast` live send and drive the library send loop on the live primary (the deferred live-send cutover)
  - Make the LIVE primary replicate via the library send (`raft_send_append_entries`/the library's per-peer send) so `on_send_append_entries` (task 2.3) fires on the real replication path, carrying the leader's term; retire/replace the term-0 `repl_broadcast` (`persist.c:747`) as the live replicator so standbys no longer receive term-0 entries. Standbys must intake every replicated entry through `raft_on_append_entries_request` (task 3.2 route) and persist `hdr.term = leader_term` (not 0); retire the legacy direct-apply branch (`ha_apply_wal_catchup_entry` gated `entry->term != 0`) as the live intake. Preserve persist-before-ACK + the commit barrier + 848/849/850 + the bug-850 contiguity guard; keep `next_index[]`/`match_index[]` driven by the ACK path (3.3) so commit advances live.
  - Observable completion: on a live cluster a primary `jobsub`-driven append replicates a term-CARRYING AppendEntries; standbys persist the leader's real term (recovered `hdr.term == leader_term`, not 0); a clean failover converges to byte-identical `(term,lsn)` WALs (closes the bug-990 / R5.1 term-identity gap); the library `commit_index` advances from live quorum ACKs.
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.5, 4.1_
  - _Boundary: repl.c / persist.c / ha.c live replication path (send + intake cutover)_
  - _Depends: 3.1, 3.2, 3.3_

## 4. Configuration ordering as a unified `(term,lsn)` marker (R7)

- [x] 4.1 Add the `WAL_CONFIG_MARKER` entry type and reference payload
  - Add the new WAL entry type and its marker payload carrying only `{version_tag, sha256}` (no config bytes); no on-disk header layout change
  - Observable completion: a config marker can be encoded/decoded with its version tag and 32-byte hash and carries no configuration bytes
  - _Requirements: 7.2, 7.4_
  - _Boundary: wal_types / config_repl_

- [x] 4.2 Append config markers via the unified log and derive committed config from the latest committed marker
  - Append a `WAL_CONFIG_MARKER` through `raft_log_append` (inheriting term-stamp, commit, and `(term,lsn)` reconcile/truncate); derive the committed configuration reference from the latest committed marker within the committed prefix, retiring the separate `committed_config_tag` as the authoritative index
  - Observable completion: committing a config change advances the derived committed-config reference; a divergent config marker is truncated by reconcile exactly like a job entry
  - _Requirements: 7.1, 7.3, 7.6, 7.7, 7.11_
  - _Depends: 3.1_

- [x] 4.3 Deliver config bytes out-of-band, hash-verify before activate, fence on mismatch
  - Key the existing out-of-band config-push transport to the committed marker's `{version_tag, sha256}`; obtain bytes when not held locally, verify their hash against the committed marker before activating; keep a member non-serving under the new configuration (loud fence, keep retrying) on mismatch; on recovery activate the configuration referenced by the latest committed marker
  - Observable completion: a committed marker whose bytes match the hash activates; an injected hash mismatch keeps the member fenced and retrying (never activates mismatched config); config bytes never enter a WAL payload
  - _Requirements: 7.5, 7.8, 7.9, 7.10_
  - _Depends: 4.2_

## 5. Verification: clean-failover preservation, tooling, and the live chaotic regression

- [x] 5.1 Adapter callback unit/integration tests
  - Cover: term stamp + read-back incl. legacy-0 mapping; higher-term heartbeat adoption via the intake route; truncate-then-persist of a divergent suffix with committed prefix untouched; `on_commit_advanced` releasing the barrier only on durable majority; config marker append + committed-config derive + divergent-marker truncation
  - Observable completion: the adapter test binary exercises every callback and the marker path and passes under ASAN
  - _Requirements: 1.1, 1.5, 1.8, 2.3, 3.1, 3.2, 4.1, 7.1, 7.6, 7.7_
  - _Depends: 3.3, 4.2_

- [x] 5.2 Clean single-failover preservation integration test
  - Assert a single clean failover (one primary killed, one standby promoted, old primary restarted) converges all members to a byte-identical WAL including terms, preserving recovery-on-promotion + advertise-real-LSN (848/849) and the contiguity guard (850); no added steady-state ack latency
  - Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green
  - _Requirements: 5.1, 5.2, 5.3, 5.4_
  - _Depends: 3.3_
  - _Partially validated on a REAL fs ($HOME/tmp/xce-ha-deploy, ext4) 2026-06-22: bug-990 term propagation CONFIRMED LIVE — PAYLOAD run converged with ctrl-2/ctrl-3 persisting the leader's REAL term (2) for the committed prefix (pre-3.4 they were 0), walcompare OK over the committed prefix. BUT a NEW regression surfaced — see bug-991 below. Harness fixes landed: failover detection (match any post-kill CTRL_PRIMARY, not first_boot=0; 45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept, not commit, so the old harness raced)._
  - _Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept)._
  - _RESOLVED (commit 83fbed2): bug-991 (catch-up applied_lsn off-by-one → last-held) + **Option A** (commit 83fbed2): removed the 2.4 `min(durable-majority, raft_commit_index)` ceiling. Under single-writer the durable-majority watermark is ALREADY (term,lsn)-current-term-safe (term-carrying repl + reconcile-before-durable + own-term gate + truncate-to-safe), so the library commit_index was a redundant, starvation-prone gate. Durable-majority barrier + own-term gate are now authoritative; raft_commit_index is inert-advisory. Reviewer-APPROVED as safe (durable-majority is (term,lsn)-safe; Figure-8 holds via own-term gate, pinned by a new RED→GREEN test; bug-851 cannot recur). 177 unit tests. **LIVE CONFIRMED (commit 9e5e293, $HOME/tmp/xce-ha-deploy/final, ext4): `commit: committed_lsn 0→1→2→3→4→5` — the commit-stall is FIXED; the harness gate logs `jobs committed on a majority (committed_lsn=5 >= 5)`.** committed=5 is backed by a real durable majority (a standby fsync'd LSN 5)._
  - _NEW EXPOSED — bug-828/795 (pre-existing, DOCUMENTED, unfixed; NOT ha-term-safety; NOT introduced by this work): the live STRICT failover now FAILS at walcompare because the FAILOVER LOSES COMMITTED JOBS. Sequence: committed=5 (valid, durable majority), kill primary, a standby promotes at term 3 computing `safe_committed_lsn=1 (from 1 granted voter)` and TRUNCATES the committed jobs (LSN 2-5) → ctrl-2/ctrl-3 end at LSN 1 + term-3 NOOP while ctrl-1 (restarted) holds the committed jobs. This is a Leader-Completeness violation in the ELECTION/safe-commit-from-votes path (`ha_compute_safe_committed_lsn`), which this spec does NOT touch. It was INVISIBLE before because the commit-stall cascade (bug-990/991/2.4-ceiling) kept committed pinned at 1, so the failover never had committed jobs (LSN>1) to lose. Matches the project-memory "Failover job-loss root cause (bug-828/795, root-caused 2026-06-20, NOT fixed)". Evidence: /home/claude/tmp/xce-ha-deploy/final (kept). OWNER: ha-commit-durability / a failover-safety spec, NOT ha-term-safety. 5.2/5.4 cannot be GREEN until bug-828 is fixed._
  - _bug-828 PARTIAL FIX landed (commit 91b9a44, reviewer-verified SAFE, 177 unit tests): (part 1) `vote_reply_dispatch` was discarding the granting voter's decoded peerLastLogLsn (`(void)peer_lsn`), so `ha_compute_safe_committed_lsn` saw only the self-vote → truncated to self_lsn (the "from 1 granted voters" fingerprint). Now routes it into `ctx->votes[]` via a record-only `ha_record_vote_reply` (no double-promotion; raft_on_vote_reply stays the sole promotion driver). (part 2) candidate/voter advertise the real DURABLE (fsync'd) tip; `persist_wal_write_replicated` advances `durable_lsn` after its fail-loud inline fdatasync. Safe-commit is bounded by the durable majority (preserves more committed entries, never marks a non-majority-durable LSN safe). Fingerprint moved "from 1" → "from 2 granted voters"._
  - _REMAINING SLICE (precisely characterized, NOT yet fixed — the documented architectural core): the live clean-failover STILL loses committed jobs because the ELECTION WATERMARK TANGLE persists. A standby that caught up to applied_lsn/durable_lsn=5 during the run still resolves its OWN election position to the STALE BOOT `durable_floor=1` in the vote/PreVote freshness gate + advertise, so (a) it fails to out-rank a job-LESS candidate (freshness gate: job-less candidate's lastLogIndex=1 >= the job-having voter's mis-computed 1 → grants), and (b) a job-less node wins term 3, computes safe_committed=1, and `persist_truncate_wal_above(1)` destroys committed LSN 2-5. FIX DIRECTION: source the voter/candidate election position from the LIVE durable/applied tip (max(persist_durable_lsn/applied_lsn, last_known_committed_lsn)) NOT the boot durable_floor, consistently across ALL FOUR election paths — candidate self-advertise (`ha_candidate_advertise_lsn`), voter floor (cluster.c ~403/495 + what the controller passes as the voter's own last_log_index into `raft_on_vote_request`), the freshness comparison, and safe-commit. SAFETY GUARD: advertise/compare only DURABLE (fsync'd) positions — never a merely-received tip — or a node could win/keep an entry it isn't durable for. This is safety-critical consensus code (a wrong fix = data loss); needs a dedicated, careful, fresh-context effort. Owner: ha-commit-durability / failover-safety. Live evidence: /home/claude/tmp/xce-ha-deploy/{final,fix828,fix828b_r1} (kept)._
  - _**bug-828 FIXED (commit b77d412, reviewer-verified SAFE).** The earlier "watermark tangle" framing was WRONG — instrumentation proved the standbys DID fsync LSN 2-5 (durable_lsn=5; not a durability violation, not an advertisement-staleness bug). ACTUAL ROOT CAUSE: `ha_enter_real_election` (ha.c ~5187) did `persist_truncate_wal_above(&c->persist, ha->last_known_committed_lsn)` BEFORE the vote — a defensive heuristic copied from the RDHUP path that VIOLATES the documented rule "Never use last_known_committed_lsn for truncation — it may be stale". last_known_committed_lsn lags the true commit point (primary committed 2-5 on a durable majority then died before a heartbeat carried committed=5), so the cutoff was 1; the truncate's clamp_durable_lsn lowered durable_lsn 5→1, PHYSICALLY DESTROYING committed entries before the election computed the safe point → candidate advertised 1 → safe_committed=1 → post-election truncate kept only LSN 1. FIX: delete the premature truncate. Only truncates now are the authoritative votes-derived post-election persist_truncate_wal_above(safe_committed_lsn) + the library on_truncate_from (term,lsn) reconcile — both before any commit under the new leader. SAFE because a divergent uncommitted tail can only exist at a stale/lower term (single-writer), and the library's TERM-PRIMARY (lastLogTerm,lastLogIndex) freshness gate denies such a candidate — correct ONLY because task 3.4 fixed the term axis (the whole ha-term-safety cascade was the prerequisite that makes this fix safe). 177 unit tests + RED-proven regression test (`test_repl_catchup_durable_floor_survives_failover`). **LIVE: clean failover preserves committed jobs — walcompare committed-prefix IDENTICAL across all 3 members, both jobs on disk, across multiple runs.**_
  - _RESIDUAL (separate, pre-existing, NOT data-safety): the live harness still EXIT-1s at the POST-walcompare client `wait-primary`/`expect-job` step due to **bug-854** (libxce positional primary-discovery / dead-first-master: the client can't reach the live primary when the killed node is FIRST in the masters list — `ctrl-1,ctrl-2,ctrl-3` with ctrl-1 dead). This is a CLIENT-side discovery defect (libxce), orthogonal to the server-side committed-prefix safety which now PASSES. To get the harness fully GREEN: fix bug-854 (libxce client failover discovery) OR have the harness target a known-live node post-failover. 5.2's DATA-SAFETY assertion (walcompare identical + jobs on disk) now passes; the EXIT-0 is gated on bug-854._

- [x] 5.3 Cross-member `walcompare` divergence tool
  - Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder
  - Observable completion: the tool returns success on identical member WALs and fails (with the diverging LSN) on a hand-divergent pair
  - _Requirements: 6.1, 6.2, 6.3_
  - _Boundary: testsuite/walcompare_

- [x] 5.4 Live chaotic-multi-kill regression (R6 headline)
  - Drive a 3-node cluster through repeated primary kills interleaved with `jobsub` and config upgrades under deterministic WAL sync; after the run assert `(term,lsn,pcrc)` committed-prefix identity across all members via `walcompare`, that every acked-committed job is present on every surviving member, and that every member converges to the same committed config reference; confirm the script reproduces bug-851 on pre-fix code (term 0) and passes on the fixed code
  - Observable completion: the regression passes on the fixed code and fails on pre-fix code; no acked-committed job is absent; committed config reference identical across members
  - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 7.11_
  - _Depends: 5.2, 5.3, 4.3_
  - _Blocked (ENV-only — code gap CLOSED by 3.4): the R6 headline `(term,lsn,pcrc)` committed-prefix identity is now satisfiable (3.4 fixed bug-990). Remaining blocker is the same real-fs live-run env as 5.2 (bug-924 overlayfs can't run the chaotic-multi-kill harness here). Author/run on a real fs; pass an explicit `--committed N` to walcompare (5.3 advisory) + `expect-job`._

- [x] 5.5 Full controller build + suite green under ASAN
  - Run `/xce-build controller`; the controller compiles and links (build restored) and all controller unit/integration tests plus the adapter tests pass under ASAN with no regression in election/durability behavior
  - Observable completion: controller test binary exits 0; ASAN clean; clean-failover and durability suites still green
  - _Requirements: 1.1, 2.3, 3.1, 4.1, 5.1, 6.1, 7.1_
  - _Depends: 5.1, 5.2, 5.4_
  - _Blocked (live portion only — build+unit gate CONFIRMED GREEN): with all of 2.4–4.3 + 3.4 + 5.1 + 5.3 landed, `/xce-build controller` COMPILES + LINKS and the controller_unit_test binary EXITS 0 with **175 Tests 0 Failures** on the reliable gate (`rm -rf /tmp/xce_*` then direct binary, ASAN detect_leaks=0). The `test_wal_catchup_*` FAIL prints (forked-child overlayfs `fdatasync` errno=-11/5) + the meson `test_demotion_replay` flake are the documented bug-924 overlayfs artifacts, present since iter-1, NOT regressions. The ONLY remaining 5.5 acceptance is "clean-failover suite green", which is the LIVE 5.2 harness — env-blocked here (real fs needed). Re-run `/xce-build controller` + the 5.2/5.4 STRICT_TERM harness on a real fs to fully close 5.5._

## Implementation Notes

- **Iter 1 (1.1+1.2) — controller build RESTORED** (compile+link of xce_controller + controller_unit_test). Added RequestVote/PreVote `lastLogTerm@4/lastLogIndex@5` and WalCatchupEntry `term@4/prevLogIndex@5/prevLogTerm@6` (append-only, no field-number collision); fixed cluster.c vote callers to the new library arity; ha.c emits the candidate tuple.
- **5 STUB callbacks wired into s_real_cbs (ha.c) to satisfy raft_init's new non-NULL requirement — tasks 2.x MUST replace:** ha_cb_persist_entry_stub(→2.1), ha_cb_get_entry_term_stub(→2.2, returns 0), ha_cb_send_append_entries_stub(→2.3), ha_cb_truncate_from_stub(→2.4), ha_cb_commit_advanced_stub(→2.5). Freshness logOk is INERT until 1.3+2.2 (candidate last_log_term currently sourced from the unset G1 `persist.term`).
- **ENV BLOCKER (critical):** fresh `nix develop` shells fail — flake pins `c-capnproto` to a mutable `main.tar.gz` whose hash drifted (expected sha256-9dzSc3w+yxYzsF/rZe3qIarKOpjt2zvIyjuS5FXngIM=, got sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk=). Only the pre-cached main worktree builds; no worktree isolation / clean baseline possible until the flake input is repinned to an immutable commit/tag.
- **Pre-existing controller-test failures (NOT this task):** test_repl_unit.c `test_wal_catchup_batch_*` fail with -EAGAIN + ASAN leaks under overlayfs `/tmp` (bug-850 fsync env artifact). The diff touches none of repl.c/persist.c/test_repl_unit.c/ha_apply_wal_catchup_entry; the append-only capnp change cannot affect the @3 entries-list decode. The controller suite needs a real-fsync filesystem for clean validation.

- **CORRECTION (2026-06-22): env unblocked + iter-1 fully validated.** User refreshed the c-capnproto flake pin (bug-918 resolved). A CLEAN controller build now passes the full controller suite green (Ok:1 Fail:0, ~13s). The earlier `test_wal_catchup_*` failures were a STALE incremental-build artifact (iter-1 capnp schema compiled against the drifted toolchain) — NOT an overlayfs fsync issue and NOT a source regression. Validation gate for tasks 2.x+ is reliable: `/xce-build controller` (clean) or incremental `meson compile -C xce.build xce_controller controller_unit_test && meson test -C xce.build "controller test"`.

- **1.3 done.** persist_set_term + per-LSN in-memory term index (built in replay_wal, freed in persist_fini) + persist_entry_term accessor; legacy-0 -> WAL_LEGACY_TERM=1 at read-back only (raw 0 kept so present-legacy-0=1 vs absent=0). 40-byte header preserved. 5 new tests in test_term_safety_adapter.c.
  - **Task 2.2 MUST document (review MEDIUM):** in a mixed pre/post-feature cluster where the post-feature leader is in term 1, the (term,lsn) check cannot distinguish a legacy-0 entry from a real term-1 entry by term alone; the Raft log-matching invariant + the bounded upgrade window make this safe, but note it in the on_get_entry_term/conflict-truncate comment.
  - **Validation env (bug-924):** controller suite creates many /tmp/xce_* WAL dirs; overlayfs fdatasync degrades as they accumulate -> non-deterministic meson timeouts. Reliable gate: `rm -rf /tmp/xce_*` then run the `controller_unit_test` binary directly (all groups 0 failures). Pre-existing ASAN leak in test_commit_barrier_reply.c (not gated by meson detect_leaks=0).

- **2.1 done.** on_persist_entry body: persist_set_term(p,term) then role-routed write (primary persist_wal_async / standby persist_wal_write_replicated(p,index,...)); failure propagates non-zero, durable_lsn unchanged, persist-before-ACK + bug-850 guard preserved. Added in-flight `raft_entry_blob_t {type,payload_len}` envelope in wal_types.h (NEVER in the 40-byte header). 3 tests.
  - **DEFERRED to task 3.1 (review):** primary path uses persist_wal_async's `current_lsn`, NOT the library `index` — holds only while `s->log_len` and `p->current_lsn` stay co-tracked. Task 3.1 MUST retire/disable the hand-rolled primary append so no WAL append bypasses raft_log_append, else current_lsn diverges from s->log_len. Also: persist_set_term + write are two lock regions (safe today, no concurrent appender) — 3.1 should re-verify when real concurrent appends exist.
  - **DEFERRED to task 3.2 (review):** a heartbeat / 0-entry AppendEntries must NOT call on_persist_entry (entry==NULL,len==0 currently falls through to WAL_NOOP) — verify the library only calls on_persist_entry when there is an entry.
  - **3.1/3.2 producer format:** must pack `raft_entry_blob_t {type,payload_len}+payload` to match the on_persist_entry unpack (single shared def in wal_types.h).

- **2.2 done.** on_get_entry_term → persist_entry_term (legacy-0 caveat documented per 1.3 review); on_truncate_from → persist_truncate_wal_above(p,index-1) (delete LSN>=index), returns rc, non-promotable fence preserved. NEW `truncate_term_index_locked` (persist.c) shrinks the in-memory term index on BOTH truncate return paths (under p->lock, cutoff-aligned) so on_get_entry_term returns 0 for truncated LSNs — no stale-term hazard. 3 tests. Reviewer APPROVED.

- **2.3 done.** on_send_append_entries → repl_send_append_entries: builds a WalCatchupEntry AE to standbys[peer_idx].ep carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via ha_read_wal_entry_at); heartbeat = n_entries==0 (zero entries). Async (xce_wal_batch_req_raw), payload freed on all paths. 2 tests via repl_send_ae_capture_hook seam. Reviewer APPROVED.
  - **DEFERRED to task 3.2:** leader_commit is accepted+captured but NOT yet wire-encoded — WalCatchupEntry needs a `leaderCommit` capnp field + standby intake of it (3.2). Comment at repl.c corrected to state this.
  - **LOW (future refactor):** ha_read_wal_entry_at duplicates WAL segment-file parsing already in persist.c/config_repl.c/waldump.c — a persist_read_entry_at() accessor would consolidate; correct as written.

- **2.4 done.** on_commit_advanced (ha.c `ha_cb_commit_advanced_stub` body, still named `*_stub` per the 2.1–2.3 convention) routes the library `commit_index` into the barrier via NEW `commit_set_raft_commit_index(ce, commit_index)` (commit.c) → `ce->raft_commit_index` ceiling. `majority_candidate_locked` now caps `committed = min(durable-majority, raft_commit_index)` on BOTH the standalone (n_peers<=1) and multi-peer paths. Own-term WAL_NOOP writable gate **RETAINED** (design 193/201/228) — proven non-conflicting: `writable_lsn` is only a WAIT predicate (`wait_done_locked`), `raft_commit_index` only caps `committed_lsn` — disjoint quantities, no deadlock; `committed=min(...)` ⇒ no gap; ceiling only ever lowers `cap` (starts at durable floor) ⇒ never over-commits; `assert(cand<=persist_durable_lsn)` still holds. Monotonic: setter raises only on `commit_index > raft_commit_index`. 5 tests; reviewer APPROVED (stash-and-rebuild baseline proved test_repl_unit 6-fail + 1 timeout are pre-existing bug-924/working-dir env, not regressions).
  - **FEATURE GATE (critical for 3.x):** ceiling is INERT while `raft_commit_index == 0` (no on_commit_advanced fired). Lowest real LSN is 1 (commit_submit rejects lsn==0), so `0` unambiguously means "nothing committed" — no 0-vs-unset window. Tasks 3.1–3.3 wiring `raft_advance_commit` on the ACK path will fire the FIRST `on_commit_advanced(commit_index>=1)` and engage the library source; until then the autonomous barrier is preserved (no steady-state regression, R5.4).
  - **DEFERRED to 3.x (decommission the term-blind source):** `ha_compute_safe_committed_lsn` is no longer the commit source but is NOT yet removed — it still drives the promotion-time safe truncate (R5). 3.x must ensure no path still feeds it INTO the barrier as a commit source once `raft_advance_commit` is wired.
  - For-test seam: `ha_cb_commit_advanced_for_test` (ha.h, XCE_TESTING) + `commit_set_raft_commit_index` are the hooks 5.1 integration tests should reuse.

- **2.5 done.** Init restore on the REAL post-recovery boot path `ha_init_raft` (ha.c ~4472, runs after `persist_recover` per myapp.c:237/242) — NOT the pre-recovery `ha_init` placeholder (~1756, raft_c==NULL, left at zero by design). `rcfg.initial_log_len = persist_durable_floor(&c->persist)`; `rcfg.initial_commit_index = committed_lsn` clamped to log_len (preserves library `commit_index<=log_len`, raft.h:61). There is NO `initial_last_log_term` field — the library sources last-log term via the wired `on_get_entry_term(s->log_len)` (raft.c:161). All five callbacks + election callbacks (on_become_leader/on_persist) verified non-NULL + real bodies in s_real_cbs. 2 tests; reviewer APPROVED (independently reverted restore→`Expected 3 Was 0` RED; investigated commit_index concern vs literal §1.5).
  - **commit_index-at-recover = 0 is CORRECT-BY-DESIGN, not a gap:** `persist_recover` does NOT re-derive `committed_lsn` from disk (it stays 0 after persist_init); the committed position is re-established post-election from voter replies / via `on_commit_advanced` (tasks 3.x). §1.5/design Traceability row 1.5 mandate restoring `initial_log_len` + last-log-term ONLY. The clamp is just defensive wiring for any future path that sets committed_lsn pre-init. The 2.5 test sets committed_lsn=2 manually to exercise the clamp/pass-through (honestly commented); the load-bearing real-recover assertions are log_len==3 + last-log term==7.
  - **PRE-EXISTING STALE COMMENT (cleanup, not 2.5 scope):** ha.c ~1326-1334 still calls the five callbacks "stubs"/"placeholders" that "perform NO persistence" — stale since 2.1–2.4 gave them real bodies. Also the callbacks are still NAMED `ha_cb_*_stub` (real bodies). A rename + comment refresh is a good 5.5 / cleanup-pass item; cosmetic only.

- **3.1 done.** `raft_log_append` is now the SINGLE primary append entry point. NEW `ha_primary_log_append(c,type,payload,len)` (ha.c) packs the `raft_entry_blob_t{type,payload_len}+payload` envelope → `raft_log_append` → on_persist_entry → persist_wal_async (leaf). Hand-rolled bypass RETIRED: persist.c `wal_hook_append_classified` (~2332) + `persist_wal_sync_append` (~2047) route through `ha_primary_log_append` when `p->cluster != NULL` (standalone persist keeps direct path). `ha_arm_own_term_gate` NOOP also routes through it (stale current_lsn-1 LSN derivation deleted). Surviving direct persist_wal_async callers are ONLY: the leaf writer, on_persist_entry's library sink, and the `-EPERM` legacy fallback (stamps real `ha->term` via persist_set_term — never term 0, only the non-raft/engineless road). 3 tests; reviewer APPROVED (verified retire-not-parallel, no recursion [persist_wal_async is leaf], envelope parity, persist-before-act, stash-baseline proved bug-924 7-fail cluster pre-existing).
  - **Anti-divergence invariant now LIVE + tested:** `p->current_lsn == s->log_len + 1` (next-LSN co-tracking) asserted across consecutive appends. Seeded at boot by 2.5 restore (s->log_len = durable_floor = current_lsn-1); preserved because raft_log_append (+1 to log_len) and its on_persist_entry→persist_wal_async (+1 to current_lsn) move in lockstep.
  - **DEFERRED to 3.2:** heartbeat / 0-entry AppendEntries intake must route through `raft_on_append_entries_request` (UpdateTerm/logOk/(term,lsn) truncate via callbacks), replacing `ha_consume_heartbeat_committed` term handling + `ha_rejoin_reconcile` committed-boundary truncate; WalCatchupEntry still needs a `leaderCommit` capnp field + standby intake (deferred from 2.3).
  - **PROCESS NOTE:** implementer first emitted `STATUS: COMPLETE` (non-parseable); corrected to READY_FOR_REVIEW. (Continue an existing subagent with SendMessage, not a fresh Agent call — a fresh Agent has no context.)

- **3.2 done.** Heartbeat/replication intake routes through the library. NEW `ha_consume_append_entries` (ha.c) = library-route intake; `ha_consume_heartbeat_committed` (~5691) now sends a 0-entry AE through `raft_on_append_entries_request` for UpdateTerm (G5) BEFORE the monotonic committed-learn — old term-blind `if(hb->term<myterm)return` DELETED; reachable on the LIVE heartbeat path (cluster.c:326). `ha_apply_wal_catchup_entry` (~2034) routes term-carrying single-entry AE through the library (logOk G2 + (term,lsn) truncate G3 via on_truncate_from/on_persist_entry). `ha_rejoin_reconcile` (ha_convergence.c) RETIRED to no-op; dead `ha_rejoin_reconcile_fn`/`_ctx_t` removed (the watermark committed-boundary truncate was the bug-851-unsafe heuristic). Wire gap CLOSED: `WalCatchupEntry.leaderCommit @7` (controller.capnp, append-only, no collision w/ term@4/prevLogIndex@5/prevLogTerm@6) encoded (request.c/repl.c) + decoded into typemap `leader_commit` + passed to the library (library currently `(void)`s it — commit-advance is 3.3). 3 new intake tests + 3 rewritten rejoin tests (test_demotion_convergence.c — preserve safety asserts, pin the retired contract, NOT weakened). 156 Tests 0 Failures; reviewer APPROVED (verified live-reachability, replace-not-parallel, leaderCommit end-to-end, baseline-stash proved bug-924 cluster pre-existing).
  - **CRITICAL for 3.3 (the live-path gap):** live steady-state replication still uses TERM-0 `repl_broadcast` (legacy direct-apply in ha_apply_wal_catchup_entry, gated `entry->term != 0`). So the term-carrying single-entry AE RECEIVE route is UNIT-TEST-ONLY in production until 3.3 wires the library SEND loop (`raft_on_append_entries_reply` + `raft_advance_commit`) AND RETIRES the term-0 `repl_broadcast`. The higher-term-heartbeat UpdateTerm IS live (heartbeats are 0-entry → always go through the library route). Live divergent-tail convergence is validated in 5.2/5.4.
  - **LOW cleanup (5.5 / cleanup pass):** (1) ha_convergence.c ~292-297 doc-comment on `cluster_apply_ctrlhello_reply` still describes the old "clamped truncate + restart" — stale. (2) `ha.h:197 rejoin_reconciled` atomic is now a dead struct field (once-guard unused). Both harmless/cosmetic. Add to the growing 5.5 cleanup list alongside the `ha_cb_*_stub` rename + stale stub-header comment.

- **3.3 done (REJECTED round 1, fixed, APPROVED round 2).** ACK path: NEW `ha_raft_on_repl_ack(c, abs_peer_idx, acked_lsn, reply_term)` (ha.c ~2914) calls `raft_on_append_entries_reply` then `raft_advance_commit` → fires `on_commit_advanced` on a strict current-term advance (activates the 2.4 commit gate). Wired into the production walAck success branch (`repl_ack_decode_and_dispatch`, ha.c ~4327); `reply_term = war_raw.term`. §4.1/§4.5 current-term gate owned by library (raft.c G4); stale-term reply dropped library-first. 6 ACK tests; reviewer APPROVED round 2 after a full re-verify.
  - **bug-960 (CRITICAL, caught by review — logged in buglog.json):** `ha_raft_on_repl_ack` first applied the votes-index→absolute remap UNCONDITIONALLY, but the walAck call site fires for BOTH promoted-vote eps (`vctx->ctx != NULL`, relative peer_idx) AND heartbeat standby eps (`vctx->ctx == NULL`, ALREADY-absolute peer_idx — see authoritative ha.c ~6172-6189 / ~5808). Over-shift → OOB `match_index[]` write (raft.c:802 has NO bounds check) in a 3-node cluster / wrong-peer FALSE-QUORUM data-loss in larger. FIX: `ha_repl_ack_abs_peer_idx(ha, ep_is_vote, peer_idx)` resolver (vote→remap, heartbeat→passthrough) resolved at the call site via `vctx->ctx != NULL`; `ha_raft_on_repl_ack` now takes ABSOLUTE idx + fail-loud `[0,n_peers)` self-excluding bounds check (logs+returns, never crashes/self-terminates). Unit tests had MASKED it (only exercised votes_idx under my_priority==0). LESSON: any peer-index passed to a library `match_index[]`/`next_index[]` write MUST be resolved per endpoint-type (vote-relative vs heartbeat-absolute) — never blind-remap; test BOTH endpoint types at my_priority 0 AND non-0.
  - **DO-NOT-REPEAT (cerebrum):** an implementer ran `clang-format -i` (whole-file reformat) then `git checkout HEAD -- <files>` to undo it, which WIPED the uncommitted task diff (HEAD was at the prior task's commit). It reconstructed the diff from context and re-review confirmed fidelity, but: never `git checkout HEAD --` to undo a reformat on a file holding uncommitted work; reformat only the changed hunks.
  - **STILL DEFERRED to 5.2/5.4 (carried 3.2→3.3):** live steady-state send is still TERM-0 `repl_broadcast`; the library send loop (`on_send_append_entries`) is not yet the live sender, so the ACK→commit path is proven end-to-end in UNIT TESTS but the first LIVE `on_commit_advanced(>=1)` requires retiring term-0 `repl_broadcast` + driving the library send. Live chaotic convergence (and this retirement) is validated/owned by 5.2 (clean failover) + 5.4 (chaotic regression). Phase 3 (drive library entry points) is structurally complete; phase 4 (config marker) + phase 5 (verification) remain.

- **4.1 done.** `WAL_CONFIG_MARKER = 50` (wal_types.h, between WAL_NOOP=40/WAL_CHECKPOINT=100) + `wal_config_marker_t {uint64_t version_tag; uint8_t sha256[32];}` + `WAL_CONFIG_MARKER_PAYLOAD_LEN (8u+32u)`. Reference-only codec `config_marker_encode/decode` (config_repl.c/h): explicit BIG-ENDIAN version_tag + raw 32-byte sha256 memcpy; NO config-bytes parameter on any path (R7.4); `-EINVAL`/`-ENOSPC` guards, no OOB on truncated decode/short encode. 4 tests (test_config_repl.c). 40-byte hdr static_assert intact; src/lib/raft untouched. Reviewer APPROVED.
  - **For 4.2 (use, don't rediscover):** (1) the marker version_tag is consistent with the EXISTING config tag: `config_version_tag = be64toh(first 8 bytes of sha256)` (config_repl.c ~171-174/382-386), so a marker's BE version_tag bytes == sha256[0..7]. (2) `wal_classify` (persist.c ~2188) has `default: assert(0)` under XCE_TESTING and `replay_wal` (persist.c ~1499) has `default: break;` — 4.2 MUST add WAL_CONFIG_MARKER classification (and update the hand-maintained `test_classify_is_total_over_wal_entry_type` array in test_commit_barrier_rollout.c ~263-281) BEFORE any live path appends a type-50 entry, or the assert fires. The marker rides the `raft_entry_blob_t {type=WAL_CONFIG_MARKER, payload_len=40}` envelope through `ha_primary_log_append`→raft_log_append→on_persist_entry.

- **4.2 done (REJECTED round 1, fixed, APPROVED round 2).** LIVE wiring: `do_upgrade_config_with_bytes` step 7 (config_repl.c ~1396, reached via OP_upgradeConfig handler ~1468) calls NEW `config_commit_change` → `config_append_marker` → `ha_primary_log_append(WAL_CONFIG_MARKER)` at the live term, replacing the term-blind authoritative path. `committed_config_tag` DEMOTED to a non-authoritative cache (only advanced in push_config_to_standbys; remaining readers are status/heartbeat-propagation, NOT ordering authority); the authoritative committed-config reference is DERIVED via `config_committed_marker` (latest committed marker within the committed prefix). `wal_classify`+`replay_wal` give WAL_CONFIG_MARKER → `WAL_MODE_RETAIN` with NO entity restore (reference-only). 6 tests; reviewer APPROVED round 2 (RED reproduced by neutering; 162/0 ASAN).
  - **Round-1 REJECT lesson (design settles scope):** the live cutover (do_upgrade_config_with_bytes appends the marker + retire committed_config_tag as authoritative) is **4.2's** job per design §306-320 (Config marker component owns R7.1/7.3/7.6/7.7/7.11); task 4.3's component §336-364 is EXCLUSIVELY byte-delivery R7.5/7.8/7.9/7.10. Providing append/derive PRIMITIVES without wiring the live path = incomplete task (dead code). When scoping "is the live wiring this task or the next", read the design COMPONENT→requirement mapping, not just the task title.
  - **Compaction hazard FIXED (Finding 2):** `trim_wal_segments` can unlink a committed marker's segment. `persist_checkpoint` now calls `scan_latest_committed_config_ref` (committed-prefix, latest-wins) BEFORE write_snapshot+trim, stamps the 41-byte `{version_tag,sha256}` ref into an OPTIONAL snapshot-sentinel TAIL (backward-compat: load_snapshot needs only n>=16; tail gated on `n>=57 && header[16]!=0`, so OLD 16-byte sentinels still load — no durable-format regression). `config_committed_marker` falls back to the durable `persist.cfgref_*` ONLY on live-scan -ENOENT (newer live marker always wins; durable ref is the last-checkpoint FLOOR, monotonic, never a shadow). This is a 40-byte REFERENCE in the snapshot, NOT the out-of-scope full `config-upgrade-snapshot`.
  - **DEFERRED to 4.3:** `config_committed_marker` has no production CONSUMER yet (only tests) — activation/serve-fence/hash-verify-before-activate + the status-from-derived-value consumer is 4.3 (R7.5/7.8/7.9/7.10). 4.2 only requires the derive to ADVANCE on commit (proven). cb5 fixture in test_commit_barrier_rollout.c made persist-backed so the live marker append succeeds (assertions unchanged).

- **4.3 done (REJECTED ×2, APPROVED round 3). PHASE 4 COMPLETE.** Byte delivery + hash-verify-before-activate + serve-fence. NEW `config_activate_committed(c)` = the production consumer: reads `config_committed_marker`, hashes the locally-held out-of-band bytes (`local_config_sha256`), VERIFIES sha256 == committed marker BEFORE `load_primary_config`, else sets `ha.config_serve_fence` (loud, `-EAGAIN`, stay-alive). `config_serve_fenced` accessor. ENFORCED by `ha_dispatch_gate` (CTRL_PRIMARY denies config-dependent state-changing ops e.g. jobsub while fenced; admits replConfigPush/read/status — no un-fence deadlock) + `ha_config_promotion_fenced` (refuse promotion in raft_cb_become_leader, stay follower). Wired at: (A) `handle_repl_config_push` (save-then-activate, SOLE activation point — no pre-activate), (B) `myapp.c` boot step 7 (recovery activates latest committed marker; -ENOENT→load_primary_config fallback, NO false fence), (D) `ha_cb_commit_advanced_stub` schedules an OFF-IOMGR delaymgr retry. bug-981 logged. Reviewer APPROVED round 3.
  - **bug-981 (CRITICAL liveness, caught round 2):** push-before-commit on a 2nd+ upgrade — a standby that saves NEW bytes before its committed prefix advances past the NEW marker fences (config_committed_marker still returns the OLDER marker → hash mismatch), and NOTHING re-activated post-commit (on_commit_advanced is async-I/O-bound so can't do disk reads; primary's config-behind re-push defeated because conf_save_config already advanced the advertised tag) → PERMANENT fence until reboot, violating R7.9 "keep retrying". FIX: `ha_cb_commit_advanced_stub` (iomgr ACK thread) does ONLY an atomic `config_serve_fenced` read + non-blocking `config_schedule_activate_retry` (delay_task_ms→ordq_put, NO disk I/O); the disk-reading `config_activate_retry_fn` runs on the delaymgr worker (mirrors ha_convergence.c delay_task pattern); clears the fence once the NEW marker commits + bytes verify; no-op once unfenced (no busy-spin, single worker serializes, atomic fence).
  - **bug (verify-bypass, caught round 1 Finding A):** handle_repl_config_push originally called load_primary_config (activating pushed bytes) BEFORE the hash gate → mismatched config went live. Fixed: gate is the SOLE activation point.
  - **PATTERN (Do-Not-Repeat):** for a marker-keyed out-of-band-bytes consumer, (1) verify-before-activate must be the SOLE activation path (no earlier load); (2) the consumer must be wired into ALL production reach paths — byte-arrival AND boot-recovery AND commit-advance — not just one (dead-code-in-production = incomplete task, same trap as 4.2); (3) a fence flag must have a real production READER or it's not a fence; (4) marker (WAL) and bytes (out-of-band) commit on INDEPENDENT transports → always handle push-before-commit with a commit-advance retry. async-I/O: schedule disk work off the iomgr ACK thread via delaymgr.
  - **CONSOLIDATED 5.5/cleanup list:** (a) `ha_cb_*_stub` rename + stale stub-header comment ha.c ~1326; (b) ha_convergence.c ~292-297 stale cluster_apply_ctrlhello_reply doc-comment; (c) ha.h rejoin_reconciled dead field; (d) committed_config_tag now cache-only (audit remaining readers). All cosmetic.

- **5.1 done.** Coverage audit found all 5 scenarios + 5 callbacks already covered by per-task tests; the genuine gap was the task's literal observable ("exercises every callback AND the marker path"). Added ONE integration test `test_every_callback_and_marker_path_end_to_end` (test_term_safety_adapter.c, test-only, empty production diff) driving a leader-lifecycle flow through the REAL seams: ha_primary_log_append→raft_log_append→on_persist_entry (job@1, marker@2, job@3 term-stamped); on_get_entry_term read-back; on_send_append_entries captures the marker entry (lsn2/WAL_CONFIG_MARKER/payload40) via repl_send_ae_capture_hook; on_commit_advanced ceiling caps then real ha_raft_on_repl_ack→raft_advance_commit releases; on_truncate_from drops LSN3, committed job+marker prefix + derived config intact. RED proven by neuter-and-revert. 33 Tests 0 Failures under DEFAULT ASAN (leak-detect ON), no new leak. Reviewer APPROVED (independently rebuilt the coverage map; confirmed real seams + load-bearing assertions).
  - **NEXT (5.x live env):** 5.2 (clean single-failover integration) _Depends 3.3_ and 5.4 (live chaotic-multi-kill regression) _Depends 5.2,5.3,4.3_ require a real 3-node deploy (/xce-deploy-ha) on a real-fs (bug-924: overlayfs fdatasync degrades meson + would corrupt timing-sensitive live runs). 5.3 (walcompare tool) is build-only (no live env). The live-send retirement (term-0 repl_broadcast → library send loop) deferred from 3.2/3.3 lands/validates in 5.2/5.4.

- **5.3 done.** `walcompare` tool: `xce_test walcompare [--committed N] <memberA> <memberB> [...]` + reusable `walcompare()` (walcompare.c/h). Diffs members by `(lsn → term, pcrc)` over the committed prefix; pcrc = payload-only crc32 (NOT header) so same-LSN/diff-content divergences surface; legacy-0→WAL_LEGACY_TERM. Exit 0 identical / WALCMP_DIVERGENT(1) on committed-prefix divergence (reports diverging LSN + per-member term/pcrc/present-or-MISSING for ALL N members) / negative errno on load error. REUSES the canonical waldump decoder via a NEW `waldump_path_visit` per-entry visitor seam (`waldump_path` is now a thin wrapper — existing waldump behavior + tests unchanged, NO 5th WAL parser). 11 tests; reviewer APPROVED (RED empirically reproduced by neutering compare; CLI hand-verified).
  - **ADVISORY for 5.4 (reviewer MEDIUM):** the DEFAULT committed boundary = `min(recovered_tip)` across members. A member that diverged then has a CRC-broken/short tail carries a LOWER recovered_tip → lowers the shared boundary → could MASK a divergence above it. 5.4 MUST pass an EXPLICIT `--committed N` (highest acked LSN), and pair walcompare with `expect-job $jobid` (R6.4) which independently catches a lost committed job.

- **3.4 done — bug-990 CLOSED (the live-send cutover).** ROOT CAUSE: `repl_broadcast` (the LIVE replicator, persist.c:747) built `wal_catchup_entry_t` without setting `entry.term`, so the live broadcast wire-encoded term=0 even though primary `hdr.term=p->term`; standbys took the legacy term-0 direct-apply and persisted term=0. FIX (MINIMAL cutover, reviewer-justified over the fuller raft_send_append_entries rewrite — the requirement set is met by carrying the term so entries auto-route through the already-wired 3.2 intake + 3.3 ACK): `repl_broadcast` now takes `prev_log_term` + sets `entry.term=hdr->term`/`prev_log_index=lsn-1`/`prev_log_term`; `persist_wal_async` sources prev_log_term via NEW lock-free `entry_term_locked(p, lsn-1)` (predecessor term, under p->lock; lsn==1→(0,0)); catch-up streamer `ha_stream_wal_to_standby` (config_repl.c) carries term+prev tuple per entry (ascending (lsn,term) tracker + conservative segment-skip so the FIRST streamed entry's prev references what the standby already holds); `ha_apply_wal_catchup_entry` (ha.c) routes EVERY term≠0 entry (single + each batch sub-entry, in order, guarded raft_c==c) through `raft_on_append_entries_request`→on_persist_entry (leader term stamped) — retiring the term-0 direct-apply for live/catch-up; `wal_entry_to_wire` (request.c) encodes prevLog{Index,Term}+leaderCommit PER SUB-ENTRY. 0-entry heartbeat still routes through the library for UpdateTerm (persists nothing). bug-850 contiguity + persist-before-ACK + commit-barrier + bug-960 peer-resolution all preserved. Live commit chain now genuinely end-to-end (term-carrying send → standby on_persist_entry + ACK → ha_raft_on_repl_ack → raft_advance_commit → on_commit_advanced → barrier). 2 new tests (send: `Expected 7 Was 0`; receive via the REAL `ha_apply_wal_catchup_entry` entry point: `Expected 7 Was 3`), RED independently reproduced by the reviewer. 175 Tests 0 Failures. Reviewer APPROVED.
  - **REMAINING (ENV-only, NOT code):** the definitive LIVE proof — clean failover → byte-identical `(term,lsn)` WALs via `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` — could NOT run here: this overlayfs sandbox can't complete a failover on baseline OR the change (split-vote / `fdatasync errno=5`, bug-924). The fix is proven at the integration level (real receive path, real wire shape). 5.2/5.4 + the live half of 5.5 need a REAL (non-overlayfs) fs to run the harness; the harness + STRICT_TERM gate are ready and should flip GREEN there.

- **bug-1095 + bug-1096 ABSORBED (Requirement 1 / Objective "two leaders never share a term"; bug-851 family) — owned here, fixed cross-boundary in libxce `raft.c` with user authorization.** _Surfaced by the raft-sim-conformance oracle (task 4.2), absorbed into ha-term-safety because the defect is an Election-Safety violation in the controller's SOLE production promotion driver: `vote_reply_dispatch` (ha.c ~4336) → `raft_on_vote_reply(&ha->raft, idx, vote_granted, term)` (ha.c ~4361)._
  - **The defect:** `raft_on_vote_reply` tallied vote grants with a plain counter that ignored `from_idx` and had no stale-term drop. (1) **bug-1095** — a duplicated vote-response (legal `DuplicateMessage`) from ONE peer double-counted toward quorum, so two candidates could each cross majority and become **leader at the SAME term**. (2) **bug-1096** — a grant carrying `mterm < currentTerm` (a stale, prior-election reply) was counted toward the CURRENT election, again admitting a spurious quorum. Both break Election Safety (`raft.tla` `votesGranted` is a SET; `DropStaleResponse` discards `mterm < currentTerm`).
  - **The fix (commit `b5b7e01`, libxce `raft.c` only — NOT touched by this task):** added a per-peer `vote_replies[RAFT_MAX_PEERS]` idempotency set (a duplicate `from_idx` is dropped before the tally) + a `term < s->term` `DropStaleResponse` that discards the stale reply WITHOUT stepping the candidate down — mirroring the already-present pre-vote dedup idiom (`pre_vote_replies[]`).
  - **Regression coverage (BOTH levels):** protocol unit — libxce `test_vote_reply_duplicate_grant_not_double_counted` + `test_vote_reply_stale_term_dropped` (`src/lib/testsuite/test_raft.c`); controller level — NEW `test_vote_reply_duplicate_grant_does_not_promote` + `test_vote_reply_stale_term_does_not_count_or_promote` (`src/controller/testsuite/test_ha_election.c`). The controller tests drive the REAL `raft_on_vote_reply` against the controller's own `ha->raft` instance (re-inited to a 5-node cluster; `xce_vote_reply_hook` NULL, `raft_c` NULL so the callbacks take their test-mode paths and `raft_cb_become_leader` still sets `ha->role = CTRL_PRIMARY`) and assert on the resulting **role / promotion** (Election Safety), not an internal counter — a duplicate grant must NOT promote at 2-of-5, and a stale grant must NOT count or promote (and must NOT step the candidate down). Both PASS over the fixed library; they would FAIL without the dedup / stale-drop. `test_ha_election` group: 0 failures.