From 14c41cb2983ebf7419f291d421d6759b0225ba3f Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 23 Jun 2026 00:29:06 +0000 Subject: [PATCH] docs(ha-term-safety): record Option A resolution + newly-exposed bug-828 failover job-loss --- .kiro/specs/ha-term-safety/tasks.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.kiro/specs/ha-term-safety/tasks.md b/.kiro/specs/ha-term-safety/tasks.md index c69555e5..3ed20427 100644 --- a/.kiro/specs/ha-term-safety/tasks.md +++ b/.kiro/specs/ha-term-safety/tasks.md @@ -126,6 +126,8 @@ - _Depends: 3.3_ - _Partially validated on a REAL fs ($HOME/tmp/xce-ha-deploy, ext4) 2026-06-22: bug-990 term propagation CONFIRMED LIVE — PAYLOAD run converged with ctrl-2/ctrl-3 persisting the leader's REAL term (2) for the committed prefix (pre-3.4 they were 0), walcompare OK over the committed prefix. BUT a NEW regression surfaced — see bug-991 below. Harness fixes landed: failover detection (match any post-kill CTRL_PRIMARY, not first_boot=0; 45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept, not commit, so the old harness raced)._ - _Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept)._ + - _RESOLVED (commit 83fbed2): bug-991 (catch-up applied_lsn off-by-one → last-held) + **Option A** (commit 83fbed2): removed the 2.4 `min(durable-majority, raft_commit_index)` ceiling. Under single-writer the durable-majority watermark is ALREADY (term,lsn)-current-term-safe (term-carrying repl + reconcile-before-durable + own-term gate + truncate-to-safe), so the library commit_index was a redundant, starvation-prone gate. Durable-majority barrier + own-term gate are now authoritative; raft_commit_index is inert-advisory. Reviewer-APPROVED as safe (durable-majority is (term,lsn)-safe; Figure-8 holds via own-term gate, pinned by a new RED→GREEN test; bug-851 cannot recur). 177 unit tests. **LIVE CONFIRMED (commit 9e5e293, $HOME/tmp/xce-ha-deploy/final, ext4): `commit: committed_lsn 0→1→2→3→4→5` — the commit-stall is FIXED; the harness gate logs `jobs committed on a majority (committed_lsn=5 >= 5)`.** committed=5 is backed by a real durable majority (a standby fsync'd LSN 5)._ + - _NEW EXPOSED — bug-828/795 (pre-existing, DOCUMENTED, unfixed; NOT ha-term-safety; NOT introduced by this work): the live STRICT failover now FAILS at walcompare because the FAILOVER LOSES COMMITTED JOBS. Sequence: committed=5 (valid, durable majority), kill primary, a standby promotes at term 3 computing `safe_committed_lsn=1 (from 1 granted voter)` and TRUNCATES the committed jobs (LSN 2-5) → ctrl-2/ctrl-3 end at LSN 1 + term-3 NOOP while ctrl-1 (restarted) holds the committed jobs. This is a Leader-Completeness violation in the ELECTION/safe-commit-from-votes path (`ha_compute_safe_committed_lsn`), which this spec does NOT touch. It was INVISIBLE before because the commit-stall cascade (bug-990/991/2.4-ceiling) kept committed pinned at 1, so the failover never had committed jobs (LSN>1) to lose. Matches the project-memory "Failover job-loss root cause (bug-828/795, root-caused 2026-06-20, NOT fixed)". Evidence: /home/claude/tmp/xce-ha-deploy/final (kept). OWNER: ha-commit-durability / a failover-safety spec, NOT ha-term-safety. 5.2/5.4 cannot be GREEN until bug-828 is fixed._ - [x] 5.3 Cross-member `walcompare` divergence tool - Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder