docs(raft-ha-core-safety): 5.6 verified (unit) + reduces churn; add task 5.7 (vote-reply term-lag escalation)

5.6's RDHUP-no-preempt guard (81cb09a) is unit-verified and slows the round-4 term
churn, but the soak still stalls — a 5th layer (task 5.7). The freshest survivor
ctrl-2 (15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it
(vote_granted) — but ctrl-2 never tallies a majority: every grant arrives as
vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term
escalates via repeated election timeouts faster than ctrl-3 (cold peer
connection, fresh SSL connect per vote, plus dead-ctrl-1 connect timeouts) can
reply, so each grant is one term stale -> dropped -> timeout -> term++.
Self-perpetuating; DropStaleResponse itself is correct Raft.

Adds task 5.7 (don't re-escalate the term while a vote RTT is outstanding /
warm-reuse the peer vote connection / wider jitter). 5 layers of the round-4
stall now peeled, 4 fixed (deadlock, segfault, advertised-term, RDHUP-preempt);
the cold-connection term-lag remains. Consensus SAFETY intact throughout.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -607,3 +607,43 @@ via RDHUP while a real election the node just entered is still within its timeou
    cadence — the round-4 election round still does not complete. New follow-up.
 Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is
 purely election LIVENESS under sustained churn.
+
+---
+
+# Task 5.6 fix VERIFIED (unit) + reduces churn; round-4 exposes a 5th layer (vote-reply term-lag escalation)
+
+Landed 5.6 (commit 81cb09a: ha_on_primary_ctrl_fd_rdhup drops an RDHUP while role==CTRL_ELECTING
+or pre_vote_in_progress, instead of restarting failover). Unit RED→GREEN (test_ha_core_safety 22
+Tests / 0 Failures); the soak's term churn is slower (peaked ~24-31 vs ~28-29+ before at the same
+elapsed time) — the fix does prevent RDHUP from preempting an in-flight election.
+
+## The 5th layer (next task 5.7)
+Round 4 still stalls. The freshest survivor ctrl-2 (15@t5) repeatedly wins PreVote, enters the real
+election, AND ctrl-3 GRANTS it (vote_granted) — ctrl-3 is voting FOR ctrl-2, not leapfrogging
+(ctrl-3 wins only 1 prevote total). But ctrl-2 never tallies a majority: its log shows every grant
+as **vote_reply_stale_drop** — the reply's term is LOWER than ctrl-2's current election term.
+Mechanism: ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can process
+the request and reply, so ctrl-3's grant (stamped at ctrl-3's lower term) is always one term behind
+ctrl-2's current term → DropStaleResponse → not counted → election times out → term++ → the grant
+for the new term is again stale. Self-perpetuating term escalation; ctrl-2 never wins despite a
+willing grantor.
+
+election_timeout_ms = 5000, jitter = 200ms (HA_DEFAULT_*). The 5s window should suffice for a warm
+RTT, so the lag points at: a just-restarted ctrl-2 with COLD peer connections (each real-vote opens
+a fresh endpoint_connect + SSL handshake; raft_cb_send_vote_request), plus parallel connect attempts
+to the dead ctrl-1 (DEFAULT_CONNECT_TIMEOUT_MS) — the grant from ctrl-3 lands after ctrl-2 has moved
+on. Fix directions (5.7): (a) don't bump the election term while a vote round-trip for the current
+term is still outstanding (give the grant time to land before re-escalating); (b) warm/reuse the
+peer vote connection so the grant RTT < election_timeout; (c) larger election-timeout jitter so the
+two survivors desynchronize and one's term stops racing the other's reply. Preserve Election Safety
+(DropStaleResponse itself is correct Raft — the bug is the escalation cadence, not the drop).
+
+## Session net (5 layers peeled on the round-4 stall; 4 fixed)
+1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d).
+2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a).
+3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified prevote-win.
+4. 5.6 RDHUP preempts in-flight election — FIXED (81cb09a), unit-verified, reduces churn.
+5. REMAINING (task 5.7): vote-reply term-lag / unbounded term escalation under cold-connection churn
+   — the freshest survivor is granted but its term out-races the grant. New follow-up.
+Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every
+completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -144,6 +144,15 @@
   - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
   - _Boundary: Election Liveness_
   - _Depends: 5.5_
+  - _PARTIAL FIX (commit 81cb09a): the RDHUP-entry now drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress (don't preempt an in-flight election). Unit RED→GREEN (22 Tests/0 Failures); reduces the term churn. But the soak still stalls at round 4 → a deeper 5th layer (task 5.7): the freshest survivor is GRANTED but its term out-races the grant (vote_reply_stale_drop). Detail in `research.md` §"Task 5.6 fix VERIFIED"._
+
+- [ ] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6)
+  - _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. A just-restarted ctrl-2 has COLD peer connections (each real-vote opens a fresh endpoint_connect+SSL) and races connect attempts to the dead ctrl-1, so the grant RTT exceeds the window before ctrl-2 re-escalates. election_timeout_ms=5000, jitter=200ms._
+  - Fix direction: (a) do not bump the election term while a vote round-trip for the current term is still outstanding (let the grant land before re-escalating); and/or (b) warm/reuse the peer vote connection so the grant RTT < election_timeout (avoid per-vote connection setup); and/or (c) widen election-timeout jitter so the two survivors desynchronize. Preserve Election Safety (DropStaleResponse is correct Raft — fix the escalation cadence, not the drop), genuine-loss liveness, 5.2 suppression, and the 5.6 no-preempt guard. RED: a just-restarted freshest survivor + a willing grantor with a cold connection; assert it wins within a bounded number of election terms (term does not escalate unbounded).
+  - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate
+  - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
+  - _Boundary: Election Liveness_
+  - _Depends: 5.6_
 
 - [ ] 6. Integration: single-seam invariant end-to-end
 - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
@@ -171,7 +180,7 @@
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
   - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
-  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); the remaining 7.2 blocker is now **task 5.6** — dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._
+  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); 5.6 (don't preempt an in-flight election) is FIXED (81cb09a, unit-verified, reduces churn); the remaining 7.2 blocker is now **task 5.7** — vote-reply term-lag / unbounded term escalation under cold-connection churn (the freshest survivor is granted but its term out-races the grant). Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure