diff --git a/.kiro/specs/ha-term-safety/tasks.md b/.kiro/specs/ha-term-safety/tasks.md index 3ed20427..b9053429 100644 --- a/.kiro/specs/ha-term-safety/tasks.md +++ b/.kiro/specs/ha-term-safety/tasks.md @@ -128,6 +128,8 @@ - _Blocked: bug-991 (commit stall via catch-up + the task-2.4 ceiling). On a STABLE 3-node cluster (no kill, 40s) submitted jobs replicate to ALL members (durable LSN 5) but committed_lsn PINS at 1 (the own-term NOOP) — jobs acked to client, never committed. ROOT CAUSE (strong hypothesis): when entries are delivered via the CATCH-UP path (ha_stream_wal_to_standby), the library commit_index does not advance past the own-term NOOP (catch-up walAck reply_term drop in raft_on_append_entries_reply, and/or match_index not advanced for catch-up delivery), so task-2.4's ceiling committed=min(durable-majority=5, raft_commit_index=1) PINS commit at 1. Pre-2.4 the durable-majority barrier would have committed 5. The PAYLOAD run committed only because delivery went via STEADY-STATE broadcast (whose ACKs do drive raft_advance_commit). This is a REAL regression from the commit-source switch (2.4 ceiling + 3.3 library-driven commit), invisible to the unit tests (clean steady-state ACKs only). Needs a fix: the 2.4 ceiling must NOT starve commit below the durable majority when the library commit_index lags (e.g. drive raft_advance_commit correctly on the catch-up ACK path incl. the standby's real reply_term, or rethink the ceiling so a lagging raft_commit_index does not pin commit below a durable-committed-safe majority). Evidence: /home/claude/tmp/xce-ha-deploy/run-strict2 (kept)._ - _RESOLVED (commit 83fbed2): bug-991 (catch-up applied_lsn off-by-one → last-held) + **Option A** (commit 83fbed2): removed the 2.4 `min(durable-majority, raft_commit_index)` ceiling. Under single-writer the durable-majority watermark is ALREADY (term,lsn)-current-term-safe (term-carrying repl + reconcile-before-durable + own-term gate + truncate-to-safe), so the library commit_index was a redundant, starvation-prone gate. Durable-majority barrier + own-term gate are now authoritative; raft_commit_index is inert-advisory. Reviewer-APPROVED as safe (durable-majority is (term,lsn)-safe; Figure-8 holds via own-term gate, pinned by a new RED→GREEN test; bug-851 cannot recur). 177 unit tests. **LIVE CONFIRMED (commit 9e5e293, $HOME/tmp/xce-ha-deploy/final, ext4): `commit: committed_lsn 0→1→2→3→4→5` — the commit-stall is FIXED; the harness gate logs `jobs committed on a majority (committed_lsn=5 >= 5)`.** committed=5 is backed by a real durable majority (a standby fsync'd LSN 5)._ - _NEW EXPOSED — bug-828/795 (pre-existing, DOCUMENTED, unfixed; NOT ha-term-safety; NOT introduced by this work): the live STRICT failover now FAILS at walcompare because the FAILOVER LOSES COMMITTED JOBS. Sequence: committed=5 (valid, durable majority), kill primary, a standby promotes at term 3 computing `safe_committed_lsn=1 (from 1 granted voter)` and TRUNCATES the committed jobs (LSN 2-5) → ctrl-2/ctrl-3 end at LSN 1 + term-3 NOOP while ctrl-1 (restarted) holds the committed jobs. This is a Leader-Completeness violation in the ELECTION/safe-commit-from-votes path (`ha_compute_safe_committed_lsn`), which this spec does NOT touch. It was INVISIBLE before because the commit-stall cascade (bug-990/991/2.4-ceiling) kept committed pinned at 1, so the failover never had committed jobs (LSN>1) to lose. Matches the project-memory "Failover job-loss root cause (bug-828/795, root-caused 2026-06-20, NOT fixed)". Evidence: /home/claude/tmp/xce-ha-deploy/final (kept). OWNER: ha-commit-durability / a failover-safety spec, NOT ha-term-safety. 5.2/5.4 cannot be GREEN until bug-828 is fixed._ + - _bug-828 PARTIAL FIX landed (commit 91b9a44, reviewer-verified SAFE, 177 unit tests): (part 1) `vote_reply_dispatch` was discarding the granting voter's decoded peerLastLogLsn (`(void)peer_lsn`), so `ha_compute_safe_committed_lsn` saw only the self-vote → truncated to self_lsn (the "from 1 granted voters" fingerprint). Now routes it into `ctx->votes[]` via a record-only `ha_record_vote_reply` (no double-promotion; raft_on_vote_reply stays the sole promotion driver). (part 2) candidate/voter advertise the real DURABLE (fsync'd) tip; `persist_wal_write_replicated` advances `durable_lsn` after its fail-loud inline fdatasync. Safe-commit is bounded by the durable majority (preserves more committed entries, never marks a non-majority-durable LSN safe). Fingerprint moved "from 1" → "from 2 granted voters"._ + - _REMAINING SLICE (precisely characterized, NOT yet fixed — the documented architectural core): the live clean-failover STILL loses committed jobs because the ELECTION WATERMARK TANGLE persists. A standby that caught up to applied_lsn/durable_lsn=5 during the run still resolves its OWN election position to the STALE BOOT `durable_floor=1` in the vote/PreVote freshness gate + advertise, so (a) it fails to out-rank a job-LESS candidate (freshness gate: job-less candidate's lastLogIndex=1 >= the job-having voter's mis-computed 1 → grants), and (b) a job-less node wins term 3, computes safe_committed=1, and `persist_truncate_wal_above(1)` destroys committed LSN 2-5. FIX DIRECTION: source the voter/candidate election position from the LIVE durable/applied tip (max(persist_durable_lsn/applied_lsn, last_known_committed_lsn)) NOT the boot durable_floor, consistently across ALL FOUR election paths — candidate self-advertise (`ha_candidate_advertise_lsn`), voter floor (cluster.c ~403/495 + what the controller passes as the voter's own last_log_index into `raft_on_vote_request`), the freshness comparison, and safe-commit. SAFETY GUARD: advertise/compare only DURABLE (fsync'd) positions — never a merely-received tip — or a node could win/keep an entry it isn't durable for. This is safety-critical consensus code (a wrong fix = data loss); needs a dedicated, careful, fresh-context effort. Owner: ha-commit-durability / failover-safety. Live evidence: /home/claude/tmp/xce-ha-deploy/{final,fix828,fix828b_r1} (kept)._ - [x] 5.3 Cross-member `walcompare` divergence tool - Build a tool that diffs member WALs by `(lsn → term, pcrc)` and exits non-zero on any committed-prefix divergence, reporting the diverging LSN and per-member entry identities; reuse the existing WAL decoder