test(ha-term-safety): live-harness fixes + record bug-991 (commit stall via catch-up + 2.4 ceiling)

Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.

Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).

That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.

.kiro/specs/ha-term-safety/tasks.md

@@ -124,7 +124,8 @@
   - Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green
   - _Requirements: 5.1, 5.2, 5.3, 5.4_
   - _Depends: 3.3_
-  - _Blocked (ENV-only — code gap CLOSED): bug-990 (the term-0 live-send gap) is FIXED by task 3.4 — the live broadcast + catch-up now carry the leader's term and standbys persist it, proven at the integration level (real receive path). The remaining blocker is purely the EXECUTION ENV: this overlayfs sandbox cannot complete a live failover (split-vote / `fdatasync errno=5`, bug-924) on baseline OR the fixed code, so the definitive `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` byte-identical-(term,lsn) assertion can't be run here. RUN the committed harness on a REAL (non-overlayfs) fs to close: it should pass PAYLOAD (848/849/850 + no job loss) AND STRICT_TERM now that 3.4 landed._
+  - _Partially validated on a REAL fs ($HOME/tmp/xce-ha-deploy, ext4) 2026-06-22: bug-990 term propagation CONFIRMED LIVE — PAYLOAD run converged with ctrl-2/ctrl-3 persisting the leader's REAL term (2) for the committed prefix (pre-3.4 they were 0), walcompare OK over the committed prefix. BUT a NEW regression surfaced — see bug-991 below. Harness fixes landed: failover detection (match any post-kill CTRL_PRIMARY, not first_boot=0; 45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept, not commit, so the old harness raced)._
+  - _Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept)._
 
 - [x] 5.3 Cross-member `walcompare` divergence tool
   - Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder

src/controller/testsuite/clean_failover_preservation.sh

@@ -92,8 +92,12 @@ PEERS="$P1,$P2,$P3"
 
 N_JOBS="${XCE_FAILOVER_NJOBS:-2}"
 WAIT_PRIMARY_MS="${XCE_FAILOVER_WAIT_MS:-30000}"
-# Liveness timeout in the controller is ~10s; allow margin for failover.
-FAILOVER_SETTLE_S="${XCE_FAILOVER_SETTLE_S:-20}"
+# Liveness timeout in the controller is ~10s; allow generous margin for failover.
+# A standby that booted with no local config participates voter-only on the first
+# RDHUP (ha-config-bootstrap Req 2.8) and only self-promotes on a later failover
+# trigger once it holds config — so a clean-failover promotion can legitimately
+# take ~25-35s (RDHUP detect + PreVote stagger + retry). 45s gives margin.
+FAILOVER_SETTLE_S="${XCE_FAILOVER_SETTLE_S:-45}"
 CATCHUP_SETTLE_S="${XCE_FAILOVER_CATCHUP_S:-8}"
 
 STRICT_TERM="${XCE_FAILOVER_STRICT_TERM:-0}"
@@ -191,6 +195,31 @@ done
 [ -n "$PRIMARY" ] || fail "could not identify the elected primary from logs"
 log "primary is $PRIMARY"
 
+# ── 2b. Wait for the jobs to COMMIT on a majority BEFORE killing ──────────────
+# A clean-failover test must kill the primary AFTER the submitted jobs are
+# committed (durable on a majority). `jobsub` returns the JobId on acceptance,
+# NOT after commit, so killing immediately would discard the still-UNCOMMITTED
+# job entries on the new primary's promotion-truncate (correct Raft, NOT a job
+# loss) — making the "no committed-job lost" assertion a race (it passed/failed
+# nondeterministically on whether replication won the race with the kill). The
+# primary logs each standby's learned committed_lsn via ha_process_standby_state;
+# require it to reach the committed boundary (own-term noop LSN 1 + the committed
+# JOB_SUBMIT entries; bug-845 doubles each submit => 1 + 2*N_JOBS).
+COMMITTED_BOUNDARY="${XCE_FAILOVER_COMMITTED:-$((1 + 2 * N_JOBS))}"
+COMMIT_WAIT_S="${XCE_FAILOVER_COMMIT_WAIT_S:-40}"
+log "waiting up to ${COMMIT_WAIT_S}s for jobs to commit on a majority (committed_lsn >= $COMMITTED_BOUNDARY)"
+COMMIT_HIGH=0
+for _ in $(seq 1 "$COMMIT_WAIT_S"); do
+    COMMIT_HIGH="$(grep -oE 'committed_lsn=[0-9]+' "$TOPDIR/logs/${PRIMARY}.log" 2>/dev/null \
+                   | awk -F= '{print $2}' | sort -n | tail -1)"
+    COMMIT_HIGH="${COMMIT_HIGH:-0}"
+    [ "$COMMIT_HIGH" -ge "$COMMITTED_BOUNDARY" ] && break
+    sleep 1
+done
+[ "$COMMIT_HIGH" -ge "$COMMITTED_BOUNDARY" ] \
+    || fail "jobs did not commit on a majority within ${COMMIT_WAIT_S}s (max committed_lsn=$COMMIT_HIGH < $COMMITTED_BOUNDARY); killing now would test in-flight replication loss, not committed-job preservation (durability concern if the jobs truly never commit on a stable cluster)"
+log "jobs committed on a majority (committed_lsn=$COMMIT_HIGH >= $COMMITTED_BOUNDARY)"
+
 # ── 3. KILL the primary ──────────────────────────────────────────────────────
 PRIMARY_PID="$(node_pid "$PRIMARY")"
 [ -n "$PRIMARY_PID" ] || fail "could not resolve pid of primary $PRIMARY"
@@ -204,7 +233,12 @@ for _ in $(seq 1 "$FAILOVER_SETTLE_S"); do
     sleep 1
     for n in ctrl-1 ctrl-2 ctrl-3; do
         [ "$n" = "$PRIMARY" ] && continue
-        if grep -q 'raft_cb_become_leader.*CTRL_PRIMARY.*first_boot=0' "$TOPDIR/logs/${n}.log" 2>/dev/null; then
+        # A standby never logs become_leader before the kill, so ANY
+        # CTRL_PRIMARY promotion line on a non-original-primary node is the
+        # failover promotion (the first_boot flag varies with whether the
+        # standby recovered its WAL vs fell back to election-path recovery —
+        # it is NOT a reliable promotion discriminator).
+        if grep -q 'raft_cb_become_leader.*CTRL_PRIMARY' "$TOPDIR/logs/${n}.log" 2>/dev/null; then
             NEW_PRIMARY="$n"
             break
         fi