test(ha-term-safety): live-harness fixes + record bug-991 (commit stall via catch-up + 2.4 ceiling)

Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.

Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).

That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.
This commit is contained in:
Rongsong Shen 2026-06-22 21:10:48 +00:00
parent 5fa394f598
commit df4bc4f86a
2 changed files with 39 additions and 4 deletions

View file

@ -124,7 +124,8 @@
- Observable completion: the clean-failover test converges to identical `(term,lsn)` WALs and the 848/849/850 behaviors remain green
- _Requirements: 5.1, 5.2, 5.3, 5.4_
- _Depends: 3.3_
- _Blocked (ENV-only — code gap CLOSED): bug-990 (the term-0 live-send gap) is FIXED by task 3.4 — the live broadcast + catch-up now carry the leader's term and standbys persist it, proven at the integration level (real receive path). The remaining blocker is purely the EXECUTION ENV: this overlayfs sandbox cannot complete a live failover (split-vote / `fdatasync errno=5`, bug-924) on baseline OR the fixed code, so the definitive `clean_failover_preservation.sh XCE_FAILOVER_STRICT_TERM=1` byte-identical-(term,lsn) assertion can't be run here. RUN the committed harness on a REAL (non-overlayfs) fs to close: it should pass PAYLOAD (848/849/850 + no job loss) AND STRICT_TERM now that 3.4 landed._
- _Partially validated on a REAL fs ($HOME/tmp/xce-ha-deploy, ext4) 2026-06-22: bug-990 term propagation CONFIRMED LIVE — PAYLOAD run converged with ctrl-2/ctrl-3 persisting the leader's REAL term (2) for the committed prefix (pre-3.4 they were 0), walcompare OK over the committed prefix. BUT a NEW regression surfaced — see bug-991 below. Harness fixes landed: failover detection (match any post-kill CTRL_PRIMARY, not first_boot=0; 45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept, not commit, so the old harness raced)._
- _Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept)._
- [x] 5.3 Cross-member `walcompare` divergence tool
- Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder