myxce/.kiro/specs/ha-term-safety/tasks.md

Implementation Plan

TDD-mandatory (testing.md): every implementation sub-task is preceded by a RED test in src/controller/testsuite/ (e.g. test_term_safety_adapter.c) that fails before the controller change and passes after. Build/run via /xce-build controller (builds lib + controller + controller_unit_test, ASAN).

Spec boundary (applies to ALL tasks): this spec is the controller-side thin callback adapter + config marker + live regression over the raft library (src/lib/raft/raft.{c,h}), consumed verbatim. It does NOT re-implement any Raft protocol decision (logOk, (term,index) truncation, current-term commit, freshness, UpdateTerm) — those are owned by raft-log-replication + raft-commit-safety (both landed, library-side). It does NOT change the commit-barrier / persist-before-ACK durability mechanism (ha-commit-durability) — on_commit_advanced only TRIGGERS it. The on-disk wal_entry_hdr_t (40 bytes) is unchanged — term is a value fix, not a layout change (R1.7). The full entity-bearing config-upgrade snapshot is OUT of scope (config-upgrade-snapshot).

Depends (HARD): raft-log-replication + raft-commit-safety — needs the five callbacks (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced), the entry points (raft_log_append, raft_on_append_entries_request/reply, raft_advance_commit), commit_index/next_index[]/match_index[], and the extended (last_log_term, last_log_index) vote signatures already present in raft.{c,h}.

Build-restoring note: the upstream vote-signature extension currently breaks the controller build (cluster.c vote callers use the old arity). Task 1.2 restores the build; until 1.2 lands, /xce-build controller will not compile (/xce-build lib stays green).

1. Foundation: wire format, WAL term seam, recovery read-back

• 1.1 Extend the AppendEntries replication envelope to carry the Raft term fields

  • Add prev_log_index, prev_log_term, and the entry's term to the AppendEntries-carrying repl envelope so a standby persists the leader's term, not a local substitute; regenerate the Cap'n Proto bindings
  • Observable completion: a replicated entry's envelope round-trips (prev_log_index, prev_log_term, entry_term) across send/decode (unit-asserted on a built envelope); existing replication messages still decode
  • Requirements: 1.3, 1.4
  • Boundary: repl transport / typemap (Cap'n Proto)

• 1.2 Extend the RequestVote/PreVote wire with (last_log_term, last_log_index) and restore the controller vote call sites

  • Add last_log_term (and last_log_index) to the RequestVote/PreVote wire message; regenerate bindings; update the controller's vote and pre-vote intake to decode the candidate's tuple and pass it to the extended library raft_on_vote_request/raft_on_pre_vote_request, sourcing the voter's own tuple via the entry-term read-back at log_len
  • Observable completion: /xce-build controller COMPILES AND LINKS again (the upstream signature break is resolved); a vote intake decodes a candidate last_log_term and the library applies the logOk freshness gate
  • Requirements: 2.3, 2.4, 2.6
  • Boundary: repl/vote wire (Cap'n Proto), controller vote intake
  • Depends: 1.1

• 1.3 Add the WAL term seam, recovery read-back, and the legacy-zero-term matching rule

  • Implement the persist_set_term seam (sets the controller's live election term) and a per-LSN recovered term index built during WAL replay, plus an accessor returning the recorded term for an LSN (0 when absent); preserve the 40-byte on-disk header (value fix only)
  • Pin the legacy-zero-term rule (REVIEW ISSUE 2): a pre-feature entry carrying term 0 maps to a single, precisely-defined "lowest-known term" for (term,lsn) matching so recovery never fails and a legacy entry is never spuriously treated as divergent from another legacy entry; document and test the exact mapping
  • Observable completion: after replaying a WAL with mixed real and legacy-0 terms, the accessor returns the recorded term for real entries and the defined lowest-known term for legacy-0 entries; header size unchanged
  • Requirements: 1.1, 1.5, 1.6, 1.7, 1.8
  • Boundary: persist (WAL I/O + term value)

2. Core: the five callback adapter bodies

• 2.1 Implement on_persist_entry over the WAL write path

  • Stamp hdr.term from the library-supplied term (via persist_set_term) at the given 1-based index == LSN, through the existing async/replicated WAL write; preserve persist-before-ACK and the bug-850 contiguity guard beneath it
  • Observable completion: a library append/accept persists an entry whose recovered hdr.term equals the library term; persist failure propagates a non-zero rc and does not ACK
  • Requirements: 1.1, 1.2, 1.3, 1.4
  • Depends: 1.3

• 2.2 Implement on_get_entry_term and on_truncate_from

  • on_get_entry_term answers from the recovered/live per-LSN term index (legacy-0 rule from 1.3); on_truncate_from deletes entries at index..end via the existing LSN truncate, replacing the committed-boundary heuristic
  • Observable completion: the library reads back a stamped entry term via the callback; a truncate-from call removes the local suffix at and above the index and leaves the prefix intact
  • Requirements: 1.5, 1.8, 3.1, 3.2
  • Depends: 1.3

• 2.3 Implement on_send_append_entries over the repl envelope

  • Build and enqueue an AppendEntries on the repl transport carrying prev_log_index, prev_log_term, the entry's term, and leader_commit = commit_index; a heartbeat carries zero entries
  • Observable completion: emitting to a peer produces an envelope with the correct prev tuple and entry term; a 0-entry heartbeat is emitted when there is nothing to send
  • Requirements: 1.3, 3.1
  • Depends: 1.1

• 2.4 Implement on_commit_advanced as a trigger into the existing barrier, and resolve the commit-gate ownership

  • Feed the library-supplied commit_index into the existing synchronous commit barrier (commit_advance_locked/commit_set_writable_lsn) as the commit SOURCE; the barrier mechanism (majority self-cap at durable floor, persist-before-ACK) is unchanged
  • Resolve double-gating (REVIEW ISSUE 3): state and implement whether the controller's existing own-term WAL_NOOP writable gate is now redundant with the library's current-term commit rule (and removed) or retained as a complementary durability gate; ensure the two cannot deadlock (commit held by both) or leave a gap
  • Observable completion: a strict library commit advance releases the barrier's WAIT only on durable majority; the term-blind safe-commit is no longer the commit source; a unit test pins the own-term-gate decision
  • Requirements: 2.2, 4.1, 4.5, 5.4
  • Depends: 1.3

• 2.5 Wire the five callbacks into the controller callback set and the library init restore

  • Add the five callbacks to the controller's s_real_cbs; on raft_init restore initial_log_len, initial_commit_index, and the recovered last-log term/index from the WAL so the library boots with the durable floor
  • Observable completion: a controller node initializes the raft library with all required callbacks non-NULL and restored indices matching its recovered WAL; existing election callbacks remain wired
  • Requirements: 1.5, 2.1
  • Depends: 2.1, 2.2, 2.3, 2.4

3. Integration: drive the library entry points on the controller paths

• 3.1 Drive raft_log_append on the primary append path

  • Route the primary's job-state append (and the own-term promotion no-op) through raft_log_append so the library stamps the term and assigns the LSN, firing on_persist_entry
  • Observable completion: a primary jobsub-driven append goes through the library and lands in the WAL with the live term stamped; the promotion no-op records the newly established term
  • Requirements: 1.1, 1.2, 2.1
  • Depends: 2.5

• 3.2 Route heartbeat/replication intake through raft_on_append_entries_request (replaces term-blind intake + committed-boundary reconcile)

  • Replace the term-handling in ha_consume_heartbeat_committed and the committed-boundary truncate in ha_rejoin_reconcile with routing every incoming heartbeat/replication (0- or 1-entry AppendEntries) through the library so UpdateTerm (G5), logOk (G2), and (term,lsn) conflict-truncate (G3) happen via the callbacks; keep the loud non-promotable retry fence for an unreconcilable member
  • Observable completion: a higher-term heartbeat makes the node adopt the term and step down (term recorded); a divergent same-LSN tail is truncated and reconverged; an unreconcilable member stays non-promotable and keeps retrying (no false convergence)
  • Requirements: 2.3, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6
  • Boundary: ha intake + ha_convergence
  • Depends: 2.5

• 3.3 Drive raft_on_append_entries_reply and raft_advance_commit on the ACK path

  • On a replication ACK, route it through raft_on_append_entries_reply to update match_index[]/next_index[], then drive raft_advance_commit so the library recomputes the current-term-gated commit and may fire on_commit_advanced
  • Observable completion: a quorum ACK advances the library commit_index only when the agreed entry is current-term, which then triggers the barrier; a stale-term reply is dropped
  • Requirements: 4.1, 4.5
  • Depends: 2.5, 3.2

• 3.4 Retire the term-0 repl_broadcast live send and drive the library send loop on the live primary (the deferred live-send cutover)

  • Make the LIVE primary replicate via the library send (raft_send_append_entries/the library's per-peer send) so on_send_append_entries (task 2.3) fires on the real replication path, carrying the leader's term; retire/replace the term-0 repl_broadcast (persist.c:747) as the live replicator so standbys no longer receive term-0 entries. Standbys must intake every replicated entry through raft_on_append_entries_request (task 3.2 route) and persist hdr.term = leader_term (not 0); retire the legacy direct-apply branch (ha_apply_wal_catchup_entry gated entry->term != 0) as the live intake. Preserve persist-before-ACK + the commit barrier + 848/849/850 + the bug-850 contiguity guard; keep next_index[]/match_index[] driven by the ACK path (3.3) so commit advances live.
  • Observable completion: on a live cluster a primary jobsub-driven append replicates a term-CARRYING AppendEntries; standbys persist the leader's real term (recovered hdr.term == leader_term, not 0); a clean failover converges to byte-identical (term,lsn) WALs (closes the bug-990 / R5.1 term-identity gap); the library commit_index advances from live quorum ACKs.
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.5, 4.1
  • Boundary: repl.c / persist.c / ha.c live replication path (send + intake cutover)
  • Depends: 3.1, 3.2, 3.3

4. Configuration ordering as a unified (term,lsn) marker (R7)

• 4.1 Add the WAL_CONFIG_MARKER entry type and reference payload

  • Add the new WAL entry type and its marker payload carrying only {version_tag, sha256} (no config bytes); no on-disk header layout change
  • Observable completion: a config marker can be encoded/decoded with its version tag and 32-byte hash and carries no configuration bytes
  • Requirements: 7.2, 7.4
  • Boundary: wal_types / config_repl

• 4.2 Append config markers via the unified log and derive committed config from the latest committed marker

  • Append a WAL_CONFIG_MARKER through raft_log_append (inheriting term-stamp, commit, and (term,lsn) reconcile/truncate); derive the committed configuration reference from the latest committed marker within the committed prefix, retiring the separate committed_config_tag as the authoritative index
  • Observable completion: committing a config change advances the derived committed-config reference; a divergent config marker is truncated by reconcile exactly like a job entry
  • Requirements: 7.1, 7.3, 7.6, 7.7, 7.11
  • Depends: 3.1

• 4.3 Deliver config bytes out-of-band, hash-verify before activate, fence on mismatch

  • Key the existing out-of-band config-push transport to the committed marker's {version_tag, sha256}; obtain bytes when not held locally, verify their hash against the committed marker before activating; keep a member non-serving under the new configuration (loud fence, keep retrying) on mismatch; on recovery activate the configuration referenced by the latest committed marker
  • Observable completion: a committed marker whose bytes match the hash activates; an injected hash mismatch keeps the member fenced and retrying (never activates mismatched config); config bytes never enter a WAL payload
  • Requirements: 7.5, 7.8, 7.9, 7.10
  • Depends: 4.2

5. Verification: clean-failover preservation, tooling, and the live chaotic regression

• 5.1 Adapter callback unit/integration tests

  • Cover: term stamp + read-back incl. legacy-0 mapping; higher-term heartbeat adoption via the intake route; truncate-then-persist of a divergent suffix with committed prefix untouched; on_commit_advanced releasing the barrier only on durable majority; config marker append + committed-config derive + divergent-marker truncation
  • Observable completion: the adapter test binary exercises every callback and the marker path and passes under ASAN
  • Requirements: 1.1, 1.5, 1.8, 2.3, 3.1, 3.2, 4.1, 7.1, 7.6, 7.7
  • Depends: 3.3, 4.2

• 5.2 Clean single-failover preservation integration test

  • Assert a single clean failover (one primary killed, one standby promoted, old primary restarted) converges all members to a byte-identical WAL including terms, preserving recovery-on-promotion + advertise-real-LSN (848/849) and the contiguity guard (850); no added steady-state ack latency
  • Observable completion: the clean-failover test converges to identical (term,lsn) WALs and the 848/849/850 behaviors remain green
  • Requirements: 5.1, 5.2, 5.3, 5.4
  • Depends: 3.3
  • Partially validated on a REAL fs ($HOME/tmp/xce-ha-deploy, ext4) 2026-06-22: bug-990 term propagation CONFIRMED LIVE — PAYLOAD run converged with ctrl-2/ctrl-3 persisting the leader's REAL term (2) for the committed prefix (pre-3.4 they were 0), walcompare OK over the committed prefix. BUT a NEW regression surfaced — see bug-991 below. Harness fixes landed: failover detection (match any post-kill CTRL_PRIMARY, not first_boot=0; 45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept, not commit, so the old harness raced).
  • Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept).
  • RESOLVED (commit 83fbed2): bug-991 (catch-up applied_lsn off-by-one → last-held) + Option A (commit 83fbed2): removed the 2.4 min(durable-majority, raft_commit_index) ceiling. Under single-writer the durable-majority watermark is ALREADY (term,lsn)-current-term-safe (term-carrying repl + reconcile-before-durable + own-term gate + truncate-to-safe), so the library commit_index was a redundant, starvation-prone gate. Durable-majority barrier + own-term gate are now authoritative; raft_commit_index is inert-advisory. Reviewer-APPROVED as safe (durable-majority is (term,lsn)-safe; Figure-8 holds via own-term gate, pinned by a new RED→GREEN test; bug-851 cannot recur). 177 unit tests. LIVE CONFIRMED (commit 9e5e293, $HOME/tmp/xce-ha-deploy/final, ext4): commit: committed_lsn 0→1→2→3→4→5 — the commit-stall is FIXED; the harness gate logs jobs committed on a majority (committed_lsn=5 >= 5). committed=5 is backed by a real durable majority (a standby fsync'd LSN 5).
  • NEW EXPOSED — bug-828/795 (pre-existing, DOCUMENTED, unfixed; NOT ha-term-safety; NOT introduced by this work): the live STRICT failover now FAILS at walcompare because the FAILOVER LOSES COMMITTED JOBS. Sequence: committed=5 (valid, durable majority), kill primary, a standby promotes at term 3 computing safe_committed_lsn=1 (from 1 granted voter) and TRUNCATES the committed jobs (LSN 2-5) → ctrl-2/ctrl-3 end at LSN 1 + term-3 NOOP while ctrl-1 (restarted) holds the committed jobs. This is a Leader-Completeness violation in the ELECTION/safe-commit-from-votes path (ha_compute_safe_committed_lsn), which this spec does NOT touch. It was INVISIBLE before because the commit-stall cascade (bug-990/991/2.4-ceiling) kept committed pinned at 1, so the failover never had committed jobs (LSN>1) to lose. Matches the project-memory "Failover job-loss root cause (bug-828/795, root-caused 2026-06-20, NOT fixed)". Evidence: /home/claude/tmp/xce-ha-deploy/final (kept). OWNER: ha-commit-durability / a failover-safety spec, NOT ha-term-safety. 5.2/5.4 cannot be GREEN until bug-828 is fixed.
  • bug-828 PARTIAL FIX landed (commit 91b9a44, reviewer-verified SAFE, 177 unit tests): (part 1) vote_reply_dispatch was discarding the granting voter's decoded peerLastLogLsn ((void)peer_lsn), so ha_compute_safe_committed_lsn saw only the self-vote → truncated to self_lsn (the "from 1 granted voters" fingerprint). Now routes it into ctx->votes[] via a record-only ha_record_vote_reply (no double-promotion; raft_on_vote_reply stays the sole promotion driver). (part 2) candidate/voter advertise the real DURABLE (fsync'd) tip; persist_wal_write_replicated advances durable_lsn after its fail-loud inline fdatasync. Safe-commit is bounded by the durable majority (preserves more committed entries, never marks a non-majority-durable LSN safe). Fingerprint moved "from 1" → "from 2 granted voters".
  • REMAINING SLICE (precisely characterized, NOT yet fixed — the documented architectural core): the live clean-failover STILL loses committed jobs because the ELECTION WATERMARK TANGLE persists. A standby that caught up to applied_lsn/durable_lsn=5 during the run still resolves its OWN election position to the STALE BOOT durable_floor=1 in the vote/PreVote freshness gate + advertise, so (a) it fails to out-rank a job-LESS candidate (freshness gate: job-less candidate's lastLogIndex=1 >= the job-having voter's mis-computed 1 → grants), and (b) a job-less node wins term 3, computes safe_committed=1, and persist_truncate_wal_above(1) destroys committed LSN 2-5. FIX DIRECTION: source the voter/candidate election position from the LIVE durable/applied tip (max(persist_durable_lsn/applied_lsn, last_known_committed_lsn)) NOT the boot durable_floor, consistently across ALL FOUR election paths — candidate self-advertise (ha_candidate_advertise_lsn), voter floor (cluster.c ~403/495 + what the controller passes as the voter's own last_log_index into raft_on_vote_request), the freshness comparison, and safe-commit. SAFETY GUARD: advertise/compare only DURABLE (fsync'd) positions — never a merely-received tip — or a node could win/keep an entry it isn't durable for. This is safety-critical consensus code (a wrong fix = data loss); needs a dedicated, careful, fresh-context effort. Owner: ha-commit-durability / failover-safety. Live evidence: /home/claude/tmp/xce-ha-deploy/{final,fix828,fix828b_r1} (kept).
  • bug-828 FIXED (commit b77d412, reviewer-verified SAFE). The earlier "watermark tangle" framing was WRONG — instrumentation proved the standbys DID fsync LSN 2-5 (durable_lsn=5; not a durability violation, not an advertisement-staleness bug). ACTUAL ROOT CAUSE: ha_enter_real_election (ha.c ~5187) did persist_truncate_wal_above(&c->persist, ha->last_known_committed_lsn) BEFORE the vote — a defensive heuristic copied from the RDHUP path that VIOLATES the documented rule "Never use last_known_committed_lsn for truncation — it may be stale". last_known_committed_lsn lags the true commit point (primary committed 2-5 on a durable majority then died before a heartbeat carried committed=5), so the cutoff was 1; the truncate's clamp_durable_lsn lowered durable_lsn 5→1, PHYSICALLY DESTROYING committed entries before the election computed the safe point → candidate advertised 1 → safe_committed=1 → post-election truncate kept only LSN 1. FIX: delete the premature truncate. Only truncates now are the authoritative votes-derived post-election persist_truncate_wal_above(safe_committed_lsn) + the library on_truncate_from (term,lsn) reconcile — both before any commit under the new leader. SAFE because a divergent uncommitted tail can only exist at a stale/lower term (single-writer), and the library's TERM-PRIMARY (lastLogTerm,lastLogIndex) freshness gate denies such a candidate — correct ONLY because task 3.4 fixed the term axis (the whole ha-term-safety cascade was the prerequisite that makes this fix safe). 177 unit tests + RED-proven regression test (test_repl_catchup_durable_floor_survives_failover). LIVE: clean failover preserves committed jobs — walcompare committed-prefix IDENTICAL across all 3 members, both jobs on disk, across multiple runs.
  • RESIDUAL (separate, pre-existing, NOT data-safety): the live harness still EXIT-1s at the POST-walcompare client wait-primary/expect-job step due to bug-854 (libxce positional primary-discovery / dead-first-master: the client can't reach the live primary when the killed node is FIRST in the masters list — ctrl-1,ctrl-2,ctrl-3 with ctrl-1 dead). This is a CLIENT-side discovery defect (libxce), orthogonal to the server-side committed-prefix safety which now PASSES. To get the harness fully GREEN: fix bug-854 (libxce client failover discovery) OR have the harness target a known-live node post-failover. 5.2's DATA-SAFETY assertion (walcompare identical + jobs on disk) now passes; the EXIT-0 is gated on bug-854.

• 5.3 Cross-member walcompare divergence tool

  • Build a tool that diffs member WALs by (lsn → term, pcrc) and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder
  • Observable completion: the tool returns success on identical member WALs and fails (with the diverging LSN) on a hand-divergent pair
  • Requirements: 6.1, 6.2, 6.3
  • Boundary: testsuite/walcompare

• 5.4 Live chaotic-multi-kill regression (R6 headline)

  • Drive a 3-node cluster through repeated primary kills interleaved with jobsub and config upgrades under deterministic WAL sync; after the run assert (term,lsn,pcrc) committed-prefix identity across all members via walcompare, that every acked-committed job is present on every surviving member, and that every member converges to the same committed config reference; confirm the script reproduces bug-851 on pre-fix code (term 0) and passes on the fixed code
  • Observable completion: the regression passes on the fixed code and fails on pre-fix code; no acked-committed job is absent; committed config reference identical across members
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 7.11
  • Depends: 5.2, 5.3, 4.3
  • Blocked (ENV-only — code gap CLOSED by 3.4): the R6 headline (term,lsn,pcrc) committed-prefix identity is now satisfiable (3.4 fixed bug-990). Remaining blocker is the same real-fs live-run env as 5.2 (bug-924 overlayfs can't run the chaotic-multi-kill harness here). Author/run on a real fs; pass an explicit --committed N to walcompare (5.3 advisory) + expect-job.

• 5.5 Full controller build + suite green under ASAN

  • Run /xce-build controller; the controller compiles and links (build restored) and all controller unit/integration tests plus the adapter tests pass under ASAN with no regression in election/durability behavior
  • Observable completion: controller test binary exits 0; ASAN clean; clean-failover and durability suites still green
  • Requirements: 1.1, 2.3, 3.1, 4.1, 5.1, 6.1, 7.1
  • Depends: 5.1, 5.2, 5.4
  • Blocked (live portion only — build+unit gate CONFIRMED GREEN): with all of 2.4–4.3 + 3.4 + 5.1 + 5.3 landed, /xce-build controller COMPILES + LINKS and the controller_unit_test binary EXITS 0 with 175 Tests 0 Failures on the reliable gate (rm -rf /tmp/xce_* then direct binary, ASAN detect_leaks=0). The test_wal_catchup_* FAIL prints (forked-child overlayfs fdatasync errno=-11/5) + the meson test_demotion_replay flake are the documented bug-924 overlayfs artifacts, present since iter-1, NOT regressions. The ONLY remaining 5.5 acceptance is "clean-failover suite green", which is the LIVE 5.2 harness — env-blocked here (real fs needed). Re-run /xce-build controller + the 5.2/5.4 STRICT_TERM harness on a real fs to fully close 5.5.

Implementation Notes

• Iter 1 (1.1+1.2) — controller build RESTORED (compile+link of xce_controller + controller_unit_test). Added RequestVote/PreVote lastLogTerm@4/lastLogIndex@5 and WalCatchupEntry term@4/prevLogIndex@5/prevLogTerm@6 (append-only, no field-number collision); fixed cluster.c vote callers to the new library arity; ha.c emits the candidate tuple.

• 5 STUB callbacks wired into s_real_cbs (ha.c) to satisfy raft_init's new non-NULL requirement — tasks 2.x MUST replace: ha_cb_persist_entry_stub(→2.1), ha_cb_get_entry_term_stub(→2.2, returns 0), ha_cb_send_append_entries_stub(→2.3), ha_cb_truncate_from_stub(→2.4), ha_cb_commit_advanced_stub(→2.5). Freshness logOk is INERT until 1.3+2.2 (candidate last_log_term currently sourced from the unset G1 persist.term).

• ENV BLOCKER (critical): fresh nix develop shells fail — flake pins c-capnproto to a mutable main.tar.gz whose hash drifted (expected sha256-9dzSc3w+yxYzsF/rZe3qIarKOpjt2zvIyjuS5FXngIM=, got sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk=). Only the pre-cached main worktree builds; no worktree isolation / clean baseline possible until the flake input is repinned to an immutable commit/tag.

• Pre-existing controller-test failures (NOT this task): test_repl_unit.c test_wal_catchup_batch_* fail with -EAGAIN + ASAN leaks under overlayfs /tmp (bug-850 fsync env artifact). The diff touches none of repl.c/persist.c/test_repl_unit.c/ha_apply_wal_catchup_entry; the append-only capnp change cannot affect the @3 entries-list decode. The controller suite needs a real-fsync filesystem for clean validation.

• CORRECTION (2026-06-22): env unblocked + iter-1 fully validated. User refreshed the c-capnproto flake pin (bug-918 resolved). A CLEAN controller build now passes the full controller suite green (Ok:1 Fail:0, ~13s). The earlier test_wal_catchup_* failures were a STALE incremental-build artifact (iter-1 capnp schema compiled against the drifted toolchain) — NOT an overlayfs fsync issue and NOT a source regression. Validation gate for tasks 2.x+ is reliable: /xce-build controller (clean) or incremental meson compile -C xce.build xce_controller controller_unit_test && meson test -C xce.build "controller test".

• 1.3 done. persist_set_term + per-LSN in-memory term index (built in replay_wal, freed in persist_fini) + persist_entry_term accessor; legacy-0 -> WAL_LEGACY_TERM=1 at read-back only (raw 0 kept so present-legacy-0=1 vs absent=0). 40-byte header preserved. 5 new tests in test_term_safety_adapter.c.

  • Task 2.2 MUST document (review MEDIUM): in a mixed pre/post-feature cluster where the post-feature leader is in term 1, the (term,lsn) check cannot distinguish a legacy-0 entry from a real term-1 entry by term alone; the Raft log-matching invariant + the bounded upgrade window make this safe, but note it in the on_get_entry_term/conflict-truncate comment.
  • Validation env (bug-924): controller suite creates many /tmp/xce_* WAL dirs; overlayfs fdatasync degrades as they accumulate -> non-deterministic meson timeouts. Reliable gate: rm -rf /tmp/xce_* then run the controller_unit_test binary directly (all groups 0 failures). Pre-existing ASAN leak in test_commit_barrier_reply.c (not gated by meson detect_leaks=0).

• 2.1 done. on_persist_entry body: persist_set_term(p,term) then role-routed write (primary persist_wal_async / standby persist_wal_write_replicated(p,index,...)); failure propagates non-zero, durable_lsn unchanged, persist-before-ACK + bug-850 guard preserved. Added in-flight raft_entry_blob_t {type,payload_len} envelope in wal_types.h (NEVER in the 40-byte header). 3 tests.

  • DEFERRED to task 3.1 (review): primary path uses persist_wal_async's current_lsn, NOT the library index — holds only while s->log_len and p->current_lsn stay co-tracked. Task 3.1 MUST retire/disable the hand-rolled primary append so no WAL append bypasses raft_log_append, else current_lsn diverges from s->log_len. Also: persist_set_term + write are two lock regions (safe today, no concurrent appender) — 3.1 should re-verify when real concurrent appends exist.
  • DEFERRED to task 3.2 (review): a heartbeat / 0-entry AppendEntries must NOT call on_persist_entry (entry==NULL,len==0 currently falls through to WAL_NOOP) — verify the library only calls on_persist_entry when there is an entry.
  • 3.1/3.2 producer format: must pack raft_entry_blob_t {type,payload_len}+payload to match the on_persist_entry unpack (single shared def in wal_types.h).

• 2.2 done. on_get_entry_term → persist_entry_term (legacy-0 caveat documented per 1.3 review); on_truncate_from → persist_truncate_wal_above(p,index-1) (delete LSN>=index), returns rc, non-promotable fence preserved. NEW truncate_term_index_locked (persist.c) shrinks the in-memory term index on BOTH truncate return paths (under p->lock, cutoff-aligned) so on_get_entry_term returns 0 for truncated LSNs — no stale-term hazard. 3 tests. Reviewer APPROVED.

• 2.3 done. on_send_append_entries → repl_send_append_entries: builds a WalCatchupEntry AE to standbys[peer_idx].ep carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via ha_read_wal_entry_at); heartbeat = n_entries==0 (zero entries). Async (xce_wal_batch_req_raw), payload freed on all paths. 2 tests via repl_send_ae_capture_hook seam. Reviewer APPROVED.

  • DEFERRED to task 3.2: leader_commit is accepted+captured but NOT yet wire-encoded — WalCatchupEntry needs a leaderCommit capnp field + standby intake of it (3.2). Comment at repl.c corrected to state this.
  • LOW (future refactor): ha_read_wal_entry_at duplicates WAL segment-file parsing already in persist.c/config_repl.c/waldump.c — a persist_read_entry_at() accessor would consolidate; correct as written.

• 2.4 done. on_commit_advanced (ha.c ha_cb_commit_advanced_stub body, still named *_stub per the 2.1–2.3 convention) routes the library commit_index into the barrier via NEW commit_set_raft_commit_index(ce, commit_index) (commit.c) → ce->raft_commit_index ceiling. majority_candidate_locked now caps committed = min(durable-majority, raft_commit_index) on BOTH the standalone (n_peers<=1) and multi-peer paths. Own-term WAL_NOOP writable gate RETAINED (design 193/201/228) — proven non-conflicting: writable_lsn is only a WAIT predicate (wait_done_locked), raft_commit_index only caps committed_lsn — disjoint quantities, no deadlock; committed=min(...) ⇒ no gap; ceiling only ever lowers cap (starts at durable floor) ⇒ never over-commits; assert(cand<=persist_durable_lsn) still holds. Monotonic: setter raises only on commit_index > raft_commit_index. 5 tests; reviewer APPROVED (stash-and-rebuild baseline proved test_repl_unit 6-fail + 1 timeout are pre-existing bug-924/working-dir env, not regressions).

  • FEATURE GATE (critical for 3.x): ceiling is INERT while raft_commit_index == 0 (no on_commit_advanced fired). Lowest real LSN is 1 (commit_submit rejects lsn==0), so 0 unambiguously means "nothing committed" — no 0-vs-unset window. Tasks 3.1–3.3 wiring raft_advance_commit on the ACK path will fire the FIRST on_commit_advanced(commit_index>=1) and engage the library source; until then the autonomous barrier is preserved (no steady-state regression, R5.4).
  • DEFERRED to 3.x (decommission the term-blind source): ha_compute_safe_committed_lsn is no longer the commit source but is NOT yet removed — it still drives the promotion-time safe truncate (R5). 3.x must ensure no path still feeds it INTO the barrier as a commit source once raft_advance_commit is wired.
  • For-test seam: ha_cb_commit_advanced_for_test (ha.h, XCE_TESTING) + commit_set_raft_commit_index are the hooks 5.1 integration tests should reuse.

• 2.5 done. Init restore on the REAL post-recovery boot path ha_init_raft (ha.c ~4472, runs after persist_recover per myapp.c:237/242) — NOT the pre-recovery ha_init placeholder (~1756, raft_c==NULL, left at zero by design). rcfg.initial_log_len = persist_durable_floor(&c->persist); rcfg.initial_commit_index = committed_lsn clamped to log_len (preserves library commit_index<=log_len, raft.h:61). There is NO initial_last_log_term field — the library sources last-log term via the wired on_get_entry_term(s->log_len) (raft.c:161). All five callbacks + election callbacks (on_become_leader/on_persist) verified non-NULL + real bodies in s_real_cbs. 2 tests; reviewer APPROVED (independently reverted restore→Expected 3 Was 0 RED; investigated commit_index concern vs literal §1.5).

  • commit_index-at-recover = 0 is CORRECT-BY-DESIGN, not a gap: persist_recover does NOT re-derive committed_lsn from disk (it stays 0 after persist_init); the committed position is re-established post-election from voter replies / via on_commit_advanced (tasks 3.x). §1.5/design Traceability row 1.5 mandate restoring initial_log_len + last-log-term ONLY. The clamp is just defensive wiring for any future path that sets committed_lsn pre-init. The 2.5 test sets committed_lsn=2 manually to exercise the clamp/pass-through (honestly commented); the load-bearing real-recover assertions are log_len==3 + last-log term==7.
  • PRE-EXISTING STALE COMMENT (cleanup, not 2.5 scope): ha.c ~1326-1334 still calls the five callbacks "stubs"/"placeholders" that "perform NO persistence" — stale since 2.1–2.4 gave them real bodies. Also the callbacks are still NAMED ha_cb_*_stub (real bodies). A rename + comment refresh is a good 5.5 / cleanup-pass item; cosmetic only.

• 3.1 done. raft_log_append is now the SINGLE primary append entry point. NEW ha_primary_log_append(c,type,payload,len) (ha.c) packs the raft_entry_blob_t{type,payload_len}+payload envelope → raft_log_append → on_persist_entry → persist_wal_async (leaf). Hand-rolled bypass RETIRED: persist.c wal_hook_append_classified (~2332) + persist_wal_sync_append (~2047) route through ha_primary_log_append when p->cluster != NULL (standalone persist keeps direct path). ha_arm_own_term_gate NOOP also routes through it (stale current_lsn-1 LSN derivation deleted). Surviving direct persist_wal_async callers are ONLY: the leaf writer, on_persist_entry's library sink, and the -EPERM legacy fallback (stamps real ha->term via persist_set_term — never term 0, only the non-raft/engineless road). 3 tests; reviewer APPROVED (verified retire-not-parallel, no recursion [persist_wal_async is leaf], envelope parity, persist-before-act, stash-baseline proved bug-924 7-fail cluster pre-existing).

  • Anti-divergence invariant now LIVE + tested: p->current_lsn == s->log_len + 1 (next-LSN co-tracking) asserted across consecutive appends. Seeded at boot by 2.5 restore (s->log_len = durable_floor = current_lsn-1); preserved because raft_log_append (+1 to log_len) and its on_persist_entry→persist_wal_async (+1 to current_lsn) move in lockstep.
  • DEFERRED to 3.2: heartbeat / 0-entry AppendEntries intake must route through raft_on_append_entries_request (UpdateTerm/logOk/(term,lsn) truncate via callbacks), replacing ha_consume_heartbeat_committed term handling + ha_rejoin_reconcile committed-boundary truncate; WalCatchupEntry still needs a leaderCommit capnp field + standby intake (deferred from 2.3).
  • PROCESS NOTE: implementer first emitted STATUS: COMPLETE (non-parseable); corrected to READY_FOR_REVIEW. (Continue an existing subagent with SendMessage, not a fresh Agent call — a fresh Agent has no context.)

• 3.2 done. Heartbeat/replication intake routes through the library. NEW ha_consume_append_entries (ha.c) = library-route intake; ha_consume_heartbeat_committed (~5691) now sends a 0-entry AE through raft_on_append_entries_request for UpdateTerm (G5) BEFORE the monotonic committed-learn — old term-blind if(hb->term<myterm)return DELETED; reachable on the LIVE heartbeat path (cluster.c:326). ha_apply_wal_catchup_entry (~2034) routes term-carrying single-entry AE through the library (logOk G2 + (term,lsn) truncate G3 via on_truncate_from/on_persist_entry). ha_rejoin_reconcile (ha_convergence.c) RETIRED to no-op; dead ha_rejoin_reconcile_fn/_ctx_t removed (the watermark committed-boundary truncate was the bug-851-unsafe heuristic). Wire gap CLOSED: WalCatchupEntry.leaderCommit @7 (controller.capnp, append-only, no collision w/ term@4/prevLogIndex@5/prevLogTerm@6) encoded (request.c/repl.c) + decoded into typemap leader_commit + passed to the library (library currently (void)s it — commit-advance is 3.3). 3 new intake tests + 3 rewritten rejoin tests (test_demotion_convergence.c — preserve safety asserts, pin the retired contract, NOT weakened). 156 Tests 0 Failures; reviewer APPROVED (verified live-reachability, replace-not-parallel, leaderCommit end-to-end, baseline-stash proved bug-924 cluster pre-existing).

  • CRITICAL for 3.3 (the live-path gap): live steady-state replication still uses TERM-0 repl_broadcast (legacy direct-apply in ha_apply_wal_catchup_entry, gated entry->term != 0). So the term-carrying single-entry AE RECEIVE route is UNIT-TEST-ONLY in production until 3.3 wires the library SEND loop (raft_on_append_entries_reply + raft_advance_commit) AND RETIRES the term-0 repl_broadcast. The higher-term-heartbeat UpdateTerm IS live (heartbeats are 0-entry → always go through the library route). Live divergent-tail convergence is validated in 5.2/5.4.
  • LOW cleanup (5.5 / cleanup pass): (1) ha_convergence.c ~292-297 doc-comment on cluster_apply_ctrlhello_reply still describes the old "clamped truncate + restart" — stale. (2) ha.h:197 rejoin_reconciled atomic is now a dead struct field (once-guard unused). Both harmless/cosmetic. Add to the growing 5.5 cleanup list alongside the ha_cb_*_stub rename + stale stub-header comment.

• 3.3 done (REJECTED round 1, fixed, APPROVED round 2). ACK path: NEW ha_raft_on_repl_ack(c, abs_peer_idx, acked_lsn, reply_term) (ha.c ~2914) calls raft_on_append_entries_reply then raft_advance_commit → fires on_commit_advanced on a strict current-term advance (activates the 2.4 commit gate). Wired into the production walAck success branch (repl_ack_decode_and_dispatch, ha.c ~4327); reply_term = war_raw.term. §4.1/§4.5 current-term gate owned by library (raft.c G4); stale-term reply dropped library-first. 6 ACK tests; reviewer APPROVED round 2 after a full re-verify.

  • bug-960 (CRITICAL, caught by review — logged in buglog.json): ha_raft_on_repl_ack first applied the votes-index→absolute remap UNCONDITIONALLY, but the walAck call site fires for BOTH promoted-vote eps (vctx->ctx != NULL, relative peer_idx) AND heartbeat standby eps (vctx->ctx == NULL, ALREADY-absolute peer_idx — see authoritative ha.c ~6172-6189 / ~5808). Over-shift → OOB match_index[] write (raft.c:802 has NO bounds check) in a 3-node cluster / wrong-peer FALSE-QUORUM data-loss in larger. FIX: ha_repl_ack_abs_peer_idx(ha, ep_is_vote, peer_idx) resolver (vote→remap, heartbeat→passthrough) resolved at the call site via vctx->ctx != NULL; ha_raft_on_repl_ack now takes ABSOLUTE idx + fail-loud [0,n_peers) self-excluding bounds check (logs+returns, never crashes/self-terminates). Unit tests had MASKED it (only exercised votes_idx under my_priority==0). LESSON: any peer-index passed to a library match_index[]/next_index[] write MUST be resolved per endpoint-type (vote-relative vs heartbeat-absolute) — never blind-remap; test BOTH endpoint types at my_priority 0 AND non-0.
  • DO-NOT-REPEAT (cerebrum): an implementer ran clang-format -i (whole-file reformat) then git checkout HEAD -- <files> to undo it, which WIPED the uncommitted task diff (HEAD was at the prior task's commit). It reconstructed the diff from context and re-review confirmed fidelity, but: never git checkout HEAD -- to undo a reformat on a file holding uncommitted work; reformat only the changed hunks.
  • STILL DEFERRED to 5.2/5.4 (carried 3.2→3.3): live steady-state send is still TERM-0 repl_broadcast; the library send loop (on_send_append_entries) is not yet the live sender, so the ACK→commit path is proven end-to-end in UNIT TESTS but the first LIVE on_commit_advanced(>=1) requires retiring term-0 repl_broadcast + driving the library send. Live chaotic convergence (and this retirement) is validated/owned by 5.2 (clean failover) + 5.4 (chaotic regression). Phase 3 (drive library entry points) is structurally complete; phase 4 (config marker) + phase 5 (verification) remain.

• 4.1 done. WAL_CONFIG_MARKER = 50 (wal_types.h, between WAL_NOOP=40/WAL_CHECKPOINT=100) + wal_config_marker_t {uint64_t version_tag; uint8_t sha256[32];} + WAL_CONFIG_MARKER_PAYLOAD_LEN (8u+32u). Reference-only codec config_marker_encode/decode (config_repl.c/h): explicit BIG-ENDIAN version_tag + raw 32-byte sha256 memcpy; NO config-bytes parameter on any path (R7.4); -EINVAL/-ENOSPC guards, no OOB on truncated decode/short encode. 4 tests (test_config_repl.c). 40-byte hdr static_assert intact; src/lib/raft untouched. Reviewer APPROVED.

  • For 4.2 (use, don't rediscover): (1) the marker version_tag is consistent with the EXISTING config tag: config_version_tag = be64toh(first 8 bytes of sha256) (config_repl.c ~171-174/382-386), so a marker's BE version_tag bytes == sha256[0..7]. (2) wal_classify (persist.c ~2188) has default: assert(0) under XCE_TESTING and replay_wal (persist.c ~1499) has default: break; — 4.2 MUST add WAL_CONFIG_MARKER classification (and update the hand-maintained test_classify_is_total_over_wal_entry_type array in test_commit_barrier_rollout.c ~263-281) BEFORE any live path appends a type-50 entry, or the assert fires. The marker rides the raft_entry_blob_t {type=WAL_CONFIG_MARKER, payload_len=40} envelope through ha_primary_log_append→raft_log_append→on_persist_entry.

• 4.2 done (REJECTED round 1, fixed, APPROVED round 2). LIVE wiring: do_upgrade_config_with_bytes step 7 (config_repl.c ~1396, reached via OP_upgradeConfig handler ~1468) calls NEW config_commit_change → config_append_marker → ha_primary_log_append(WAL_CONFIG_MARKER) at the live term, replacing the term-blind authoritative path. committed_config_tag DEMOTED to a non-authoritative cache (only advanced in push_config_to_standbys; remaining readers are status/heartbeat-propagation, NOT ordering authority); the authoritative committed-config reference is DERIVED via config_committed_marker (latest committed marker within the committed prefix). wal_classify+replay_wal give WAL_CONFIG_MARKER → WAL_MODE_RETAIN with NO entity restore (reference-only). 6 tests; reviewer APPROVED round 2 (RED reproduced by neutering; 162/0 ASAN).

  • Round-1 REJECT lesson (design settles scope): the live cutover (do_upgrade_config_with_bytes appends the marker + retire committed_config_tag as authoritative) is 4.2's job per design §306-320 (Config marker component owns R7.1/7.3/7.6/7.7/7.11); task 4.3's component §336-364 is EXCLUSIVELY byte-delivery R7.5/7.8/7.9/7.10. Providing append/derive PRIMITIVES without wiring the live path = incomplete task (dead code). When scoping "is the live wiring this task or the next", read the design COMPONENT→requirement mapping, not just the task title.
  • Compaction hazard FIXED (Finding 2): trim_wal_segments can unlink a committed marker's segment. persist_checkpoint now calls scan_latest_committed_config_ref (committed-prefix, latest-wins) BEFORE write_snapshot+trim, stamps the 41-byte {version_tag,sha256} ref into an OPTIONAL snapshot-sentinel TAIL (backward-compat: load_snapshot needs only n>=16; tail gated on n>=57 && header[16]!=0, so OLD 16-byte sentinels still load — no durable-format regression). config_committed_marker falls back to the durable persist.cfgref_* ONLY on live-scan -ENOENT (newer live marker always wins; durable ref is the last-checkpoint FLOOR, monotonic, never a shadow). This is a 40-byte REFERENCE in the snapshot, NOT the out-of-scope full config-upgrade-snapshot.
  • DEFERRED to 4.3: config_committed_marker has no production CONSUMER yet (only tests) — activation/serve-fence/hash-verify-before-activate + the status-from-derived-value consumer is 4.3 (R7.5/7.8/7.9/7.10). 4.2 only requires the derive to ADVANCE on commit (proven). cb5 fixture in test_commit_barrier_rollout.c made persist-backed so the live marker append succeeds (assertions unchanged).

• 4.3 done (REJECTED ×2, APPROVED round 3). PHASE 4 COMPLETE. Byte delivery + hash-verify-before-activate + serve-fence. NEW config_activate_committed(c) = the production consumer: reads config_committed_marker, hashes the locally-held out-of-band bytes (local_config_sha256), VERIFIES sha256 == committed marker BEFORE load_primary_config, else sets ha.config_serve_fence (loud, -EAGAIN, stay-alive). config_serve_fenced accessor. ENFORCED by ha_dispatch_gate (CTRL_PRIMARY denies config-dependent state-changing ops e.g. jobsub while fenced; admits replConfigPush/read/status — no un-fence deadlock) + ha_config_promotion_fenced (refuse promotion in raft_cb_become_leader, stay follower). Wired at: (A) handle_repl_config_push (save-then-activate, SOLE activation point — no pre-activate), (B) myapp.c boot step 7 (recovery activates latest committed marker; -ENOENT→load_primary_config fallback, NO false fence), (D) ha_cb_commit_advanced_stub schedules an OFF-IOMGR delaymgr retry. bug-981 logged. Reviewer APPROVED round 3.

  • bug-981 (CRITICAL liveness, caught round 2): push-before-commit on a 2nd+ upgrade — a standby that saves NEW bytes before its committed prefix advances past the NEW marker fences (config_committed_marker still returns the OLDER marker → hash mismatch), and NOTHING re-activated post-commit (on_commit_advanced is async-I/O-bound so can't do disk reads; primary's config-behind re-push defeated because conf_save_config already advanced the advertised tag) → PERMANENT fence until reboot, violating R7.9 "keep retrying". FIX: ha_cb_commit_advanced_stub (iomgr ACK thread) does ONLY an atomic config_serve_fenced read + non-blocking config_schedule_activate_retry (delay_task_ms→ordq_put, NO disk I/O); the disk-reading config_activate_retry_fn runs on the delaymgr worker (mirrors ha_convergence.c delay_task pattern); clears the fence once the NEW marker commits + bytes verify; no-op once unfenced (no busy-spin, single worker serializes, atomic fence).
  • bug (verify-bypass, caught round 1 Finding A): handle_repl_config_push originally called load_primary_config (activating pushed bytes) BEFORE the hash gate → mismatched config went live. Fixed: gate is the SOLE activation point.
  • PATTERN (Do-Not-Repeat): for a marker-keyed out-of-band-bytes consumer, (1) verify-before-activate must be the SOLE activation path (no earlier load); (2) the consumer must be wired into ALL production reach paths — byte-arrival AND boot-recovery AND commit-advance — not just one (dead-code-in-production = incomplete task, same trap as 4.2); (3) a fence flag must have a real production READER or it's not a fence; (4) marker (WAL) and bytes (out-of-band) commit on INDEPENDENT transports → always handle push-before-commit with a commit-advance retry. async-I/O: schedule disk work off the iomgr ACK thread via delaymgr.
  • CONSOLIDATED 5.5/cleanup list: (a) ha_cb_*_stub rename + stale stub-header comment ha.c ~1326; (b) ha_convergence.c ~292-297 stale cluster_apply_ctrlhello_reply doc-comment; (c) ha.h rejoin_reconciled dead field; (d) committed_config_tag now cache-only (audit remaining readers). All cosmetic.

• 5.1 done. Coverage audit found all 5 scenarios + 5 callbacks already covered by per-task tests; the genuine gap was the task's literal observable ("exercises every callback AND the marker path"). Added ONE integration test test_every_callback_and_marker_path_end_to_end (test_term_safety_adapter.c, test-only, empty production diff) driving a leader-lifecycle flow through the REAL seams: ha_primary_log_append→raft_log_append→on_persist_entry (job@1, marker@2, job@3 term-stamped); on_get_entry_term read-back; on_send_append_entries captures the marker entry (lsn2/WAL_CONFIG_MARKER/payload40) via repl_send_ae_capture_hook; on_commit_advanced ceiling caps then real ha_raft_on_repl_ack→raft_advance_commit releases; on_truncate_from drops LSN3, committed job+marker prefix + derived config intact. RED proven by neuter-and-revert. 33 Tests 0 Failures under DEFAULT ASAN (leak-detect ON), no new leak. Reviewer APPROVED (independently rebuilt the coverage map; confirmed real seams + load-bearing assertions).

  • NEXT (5.x live env): 5.2 (clean single-failover integration) Depends 3.3 and 5.4 (live chaotic-multi-kill regression) Depends 5.2,5.3,4.3 require a real 3-node deploy (/xce-deploy-ha) on a real-fs (bug-924: overlayfs fdatasync degrades meson + would corrupt timing-sensitive live runs). 5.3 (walcompare tool) is build-only (no live env). The live-send retirement (term-0 repl_broadcast → library send loop) deferred from 3.2/3.3 lands/validates in 5.2/5.4.

• 5.3 done. walcompare tool: xce_test walcompare [--committed N] <memberA> <memberB> [...] + reusable walcompare() (walcompare.c/h). Diffs members by (lsn → term, pcrc) over the committed prefix; pcrc = payload-only crc32 (NOT header) so same-LSN/diff-content divergences surface; legacy-0→WAL_LEGACY_TERM. Exit 0 identical / WALCMP_DIVERGENT(1) on committed-prefix divergence (reports diverging LSN + per-member term/pcrc/present-or-MISSING for ALL N members) / negative errno on load error. REUSES the canonical waldump decoder via a NEW waldump_path_visit per-entry visitor seam (waldump_path is now a thin wrapper — existing waldump behavior + tests unchanged, NO 5th WAL parser). 11 tests; reviewer APPROVED (RED empirically reproduced by neutering compare; CLI hand-verified).

  • ADVISORY for 5.4 (reviewer MEDIUM): the DEFAULT committed boundary = min(recovered_tip) across members. A member that diverged then has a CRC-broken/short tail carries a LOWER recovered_tip → lowers the shared boundary → could MASK a divergence above it. 5.4 MUST pass an EXPLICIT --committed N (highest acked LSN), and pair walcompare with expect-job $jobid (R6.4) which independently catches a lost committed job.

• 3.4 done — bug-990 CLOSED (the live-send cutover). ROOT CAUSE: repl_broadcast (the LIVE replicator, persist.c:747) built wal_catchup_entry_t without setting entry.term, so the live broadcast wire-encoded term=0 even though primary hdr.term=p->term; standbys took the legacy term-0 direct-apply and persisted term=0. FIX (MINIMAL cutover, reviewer-justified over the fuller raft_send_append_entries rewrite — the requirement set is met by carrying the term so entries auto-route through the already-wired 3.2 intake + 3.3 ACK): repl_broadcast now takes prev_log_term + sets entry.term=hdr->term/prev_log_index=lsn-1/prev_log_term; persist_wal_async sources prev_log_term via NEW lock-free entry_term_locked(p, lsn-1) (predecessor term, under p->lock; lsn==1→(0,0)); catch-up streamer ha_stream_wal_to_standby (config_repl.c) carries term+prev tuple per entry (ascending (lsn,term) tracker + conservative segment-skip so the FIRST streamed entry's prev references what the standby already holds); ha_apply_wal_catchup_entry (ha.c) routes EVERY term≠0 entry (single + each batch sub-entry, in order, guarded raft_c==c) through raft_on_append_entries_request→on_persist_entry (leader term stamped) — retiring the term-0 direct-apply for live/catch-up; wal_entry_to_wire (request.c) encodes prevLog{Index,Term}+leaderCommit PER SUB-ENTRY. 0-entry heartbeat still routes through the library for UpdateTerm (persists nothing). bug-850 contiguity + persist-before-ACK + commit-barrier + bug-960 peer-resolution all preserved. Live commit chain now genuinely end-to-end (term-carrying send → standby on_persist_entry + ACK → ha_raft_on_repl_ack → raft_advance_commit → on_commit_advanced → barrier). 2 new tests (send: Expected 7 Was 0; receive via the REAL ha_apply_wal_catchup_entry entry point: Expected 7 Was 3), RED independently reproduced by the reviewer. 175 Tests 0 Failures. Reviewer APPROVED.

  • REMAINING (ENV-only, NOT code): the definitive LIVE proof — clean failover → byte-identical (term,lsn) WALs via clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1 — could NOT run here: this overlayfs sandbox can't complete a failover on baseline OR the change (split-vote / fdatasync errno=5, bug-924). The fix is proven at the integration level (real receive path, real wire shape). 5.2/5.4 + the live half of 5.5 need a REAL (non-overlayfs) fs to run the harness; the harness + STRICT_TERM gate are ready and should flip GREEN there.

• bug-1095 + bug-1096 ABSORBED (Requirement 1 / Objective "two leaders never share a term"; bug-851 family) — owned here, fixed cross-boundary in libxce raft.c with user authorization. Surfaced by the raft-sim-conformance oracle (task 4.2), absorbed into ha-term-safety because the defect is an Election-Safety violation in the controller's SOLE production promotion driver: vote_reply_dispatch (ha.c ~4336) → raft_on_vote_reply(&ha->raft, idx, vote_granted, term) (ha.c ~4361).

  • The defect: raft_on_vote_reply tallied vote grants with a plain counter that ignored from_idx and had no stale-term drop. (1) bug-1095 — a duplicated vote-response (legal DuplicateMessage) from ONE peer double-counted toward quorum, so two candidates could each cross majority and become leader at the SAME term. (2) bug-1096 — a grant carrying mterm < currentTerm (a stale, prior-election reply) was counted toward the CURRENT election, again admitting a spurious quorum. Both break Election Safety (raft.tla votesGranted is a SET; DropStaleResponse discards mterm < currentTerm).
  • The fix (commit b5b7e01, libxce raft.c only — NOT touched by this task): added a per-peer vote_replies[RAFT_MAX_PEERS] idempotency set (a duplicate from_idx is dropped before the tally) + a term < s->term DropStaleResponse that discards the stale reply WITHOUT stepping the candidate down — mirroring the already-present pre-vote dedup idiom (pre_vote_replies[]).
  • Regression coverage (BOTH levels): protocol unit — libxce test_vote_reply_duplicate_grant_not_double_counted + test_vote_reply_stale_term_dropped (src/lib/testsuite/test_raft.c); controller level — NEW test_vote_reply_duplicate_grant_does_not_promote + test_vote_reply_stale_term_does_not_count_or_promote (src/controller/testsuite/test_ha_election.c). The controller tests drive the REAL raft_on_vote_reply against the controller's own ha->raft instance (re-inited to a 5-node cluster; xce_vote_reply_hook NULL, raft_c NULL so the callbacks take their test-mode paths and raft_cb_become_leader still sets ha->role = CTRL_PRIMARY) and assert on the resulting role / promotion (Election Safety), not an internal counter — a duplicate grant must NOT promote at 2-of-5, and a stale grant must NOT count or promote (and must NOT step the candidate down). Both PASS over the fixed library; they would FAIL without the dedup / stale-drop. test_ha_election group: 0 failures.