docs(raft-ha-core-safety): 5.5 advertised-term fix verified as progress; add task 5.6 (dueling-candidate split-vote)

5.5's real root cause was a stale advertised lastLogTerm (c->persist.term vs
persist_entry_term), not the last_activity hypothesis — fixed in b8453b2 and
VERIFIED: with it, the freshest survivor now wins PreVote and enters the real
election (it could not before). But round 4 still does not complete: a deeper
layer (task 5.6) — under the dead-primary RDHUP cadence (~15s) the two survivors
duel in lockstep and the term escalates unbounded (21->25->28->29...) without
reaching majority. Adds task 5.6 (randomized failover backoff to desync the duel
/ don't re-trigger failover mid-real-election). Consensus SAFETY intact; this is
election LIVENESS under sustained churn. 4 layers of the round-4 stall now peeled
(deadlock, segfault, advertised-term fixed; dueling-candidate remains).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -569,3 +569,41 @@ the freshest survivor is electable.
 bug-1282 (ABBA iomgr deadlock) and bug-1286 (round-4 restart SIGSEGV, NULL ep->owner) are FIXED and
 VERIFIED (no crash / no deadlock / no fdatasync across round-4 churn). The remaining 7.2 blocker is
 this election-liveness cross-deny - the hard core of Phase 5, no longer masked.
+
+---
+
+# Task 5.5 fix VERIFIED as progress; round-4 now reaches a DEEPER layer (dueling-candidate term escalation)
+
+Landed the 5.5 fix (commit b8453b2: advertise persist_entry_term(p, log_len) as lastLogTerm instead
+of the stale c->persist.term) and re-ran the 6-round soak. Round-4 frozen state: ctrl-2 tip 15 @
+term 5 (freshest), ctrl-3 tip 11 @ term 4 (staler).
+
+## 5.5 is correct and made real progress
+BEFORE the fix the freshest survivor advertised a too-low lastLogTerm and EVERY voter's logOk
+rejected it — it could not even win a PreVote. AFTER the fix ctrl-2 WINS PreVote (pre_vote_won) AND
+enters the real election ("PreVote round WON — entering real election"). The advertised-term
+asymmetry is fixed; the fresher candidate is now electable in principle.
+
+## The deeper layer it exposed (next task)
+ctrl-2 wins PreVote -> real election (term N) -> RequestVote -> replies but NO became_leader ->
+retries at term N+1, N+2 ... prospective term climbing 21->25->28->29 unbounded. Mechanism: ctrl-2's
+primary (dead ctrl-1) keeps RDHUP'ing ~every 15s; 5.2 does NOT suppress (primary genuinely dead),
+so each RDHUP re-triggers ha_begin_failover -> fresh PreVote -> real election -> term++. ctrl-2 and
+ctrl-3 duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term),
+and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally
+resolves a split vote. Neither reaches majority; the term escalates without converging.
+
+Classic Raft split-vote / dueling-candidates liveness problem, amplified by the RDHUP failover
+re-trigger. SEPARATE layer from 5.5. Next: break the duel (per-node randomized failover backoff /
+election jitter that desynchronizes the two survivors; and/or do not re-trigger a fresh failover
+via RDHUP while a real election the node just entered is still within its timeout window).
+
+## Session net (4 layers peeled on the round-4 stall)
+1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d).
+2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a).
+3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified: the fresher survivor now
+   wins PreVote + enters the real election (could not before).
+4. REMAINING: dueling-candidate split-vote + unbounded term escalation under the RDHUP failover
+   cadence — the round-4 election round still does not complete. New follow-up.
+Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is
+purely election LIVENESS under sustained churn.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -135,6 +135,15 @@
   - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
   - _Boundary: Election Liveness_
   - _Depends: 5.2, 5.3_
+  - _ACTUAL root cause (NOT the last_activity hypothesis) + PARTIAL FIX (commit b8453b2): the cross-deny was a stale **advertised lastLogTerm** — the candidate advertised `c->persist.term` (the persist running term, stale/0 after restart) instead of `persist_entry_term(p, log_len)` (the term of the last WAL entry, the same source voters use via on_get_entry_term). The freshest survivor advertised a too-low term → every voter's logOk rejected it. bug-851 family. FIX VERIFIED AS PROGRESS: the fresher survivor now WINS PreVote + enters the real election (could not before). But the round still does not COMPLETE → a deeper layer remains (task 5.6). Detail in `research.md` §"Task 5.5 fix VERIFIED"._
+
+- [ ] 5.6 Break the dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence (NEW — the layer under 5.5)
+  - _Root cause (`research.md` §"Task 5.5 fix VERIFIED"): with 5.5's advertised-term fix in, round-4 ctrl-2 (freshest) wins PreVote and enters the real election, but never reaches majority — it retries at ever-higher terms (21→25→28→29…). Mechanism: the dead primary keeps RDHUP'ing ~every 15s; 5.2 correctly does not suppress (primary genuinely dead), so each RDHUP re-triggers `ha_begin_failover` → fresh PreVote → real election → term++. The two survivors duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally resolves a split vote._
+  - Fix direction: introduce/strengthen per-node randomized failover backoff (election jitter) so two surviving candidates DESYNCHRONIZE and one wins; and/or do not re-trigger a fresh failover via RDHUP while a real election the node just entered is still within its election-timeout window (let the in-progress round complete or time out before re-arming). Preserve genuine-loss liveness (R6), spurious-failover suppression (5.2), and Election Safety. RED: two divergent-freshness survivors + a periodic RDHUP cadence; assert a primary is elected within a bounded number of rounds (term does not escalate unbounded).
+  - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no unbounded term escalation / split-vote stall) with all prior fixes in place — the green 7.2 gate
+  - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
+  - _Boundary: Election Liveness_
+  - _Depends: 5.5_
 
 - [ ] 6. Integration: single-seam invariant end-to-end
 - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
@@ -162,7 +171,7 @@
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
   - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
-  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). That is the remaining 7.2 blocker. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._
+  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); the remaining 7.2 blocker is now **task 5.6** — dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure