From 2b182ecde1fac8dd5d7db15a1fb062ac4acacaa2 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 11:58:43 +0000 Subject: [PATCH] docs(raft-ha-core-safety): 5.6 verified (unit) + reduces churn; add task 5.7 (vote-reply term-lag escalation) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 5.6's RDHUP-no-preempt guard (81cb09a) is unit-verified and slows the round-4 term churn, but the soak still stalls — a 5th layer (task 5.7). The freshest survivor ctrl-2 (15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — but ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 (cold peer connection, fresh SSL connect per vote, plus dead-ctrl-1 connect timeouts) can reply, so each grant is one term stale -> dropped -> timeout -> term++. Self-perpetuating; DropStaleResponse itself is correct Raft. Adds task 5.7 (don't re-escalate the term while a vote RTT is outstanding / warm-reuse the peer vote connection / wider jitter). 5 layers of the round-4 stall now peeled, 4 fixed (deadlock, segfault, advertised-term, RDHUP-preempt); the cold-connection term-lag remains. Consensus SAFETY intact throughout. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC --- .kiro/specs/raft-ha-core-safety/research.md | 40 +++++++++++++++++++++ .kiro/specs/raft-ha-core-safety/tasks.md | 11 +++++- 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index b096734..1df6699 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -607,3 +607,43 @@ via RDHUP while a real election the node just entered is still within its timeou cadence — the round-4 election round still does not complete. New follow-up. Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is purely election LIVENESS under sustained churn. + +--- + +# Task 5.6 fix VERIFIED (unit) + reduces churn; round-4 exposes a 5th layer (vote-reply term-lag escalation) + +Landed 5.6 (commit 81cb09a: ha_on_primary_ctrl_fd_rdhup drops an RDHUP while role==CTRL_ELECTING +or pre_vote_in_progress, instead of restarting failover). Unit RED→GREEN (test_ha_core_safety 22 +Tests / 0 Failures); the soak's term churn is slower (peaked ~24-31 vs ~28-29+ before at the same +elapsed time) — the fix does prevent RDHUP from preempting an in-flight election. + +## The 5th layer (next task 5.7) +Round 4 still stalls. The freshest survivor ctrl-2 (15@t5) repeatedly wins PreVote, enters the real +election, AND ctrl-3 GRANTS it (vote_granted) — ctrl-3 is voting FOR ctrl-2, not leapfrogging +(ctrl-3 wins only 1 prevote total). But ctrl-2 never tallies a majority: its log shows every grant +as **vote_reply_stale_drop** — the reply's term is LOWER than ctrl-2's current election term. +Mechanism: ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can process +the request and reply, so ctrl-3's grant (stamped at ctrl-3's lower term) is always one term behind +ctrl-2's current term → DropStaleResponse → not counted → election times out → term++ → the grant +for the new term is again stale. Self-perpetuating term escalation; ctrl-2 never wins despite a +willing grantor. + +election_timeout_ms = 5000, jitter = 200ms (HA_DEFAULT_*). The 5s window should suffice for a warm +RTT, so the lag points at: a just-restarted ctrl-2 with COLD peer connections (each real-vote opens +a fresh endpoint_connect + SSL handshake; raft_cb_send_vote_request), plus parallel connect attempts +to the dead ctrl-1 (DEFAULT_CONNECT_TIMEOUT_MS) — the grant from ctrl-3 lands after ctrl-2 has moved +on. Fix directions (5.7): (a) don't bump the election term while a vote round-trip for the current +term is still outstanding (give the grant time to land before re-escalating); (b) warm/reuse the +peer vote connection so the grant RTT < election_timeout; (c) larger election-timeout jitter so the +two survivors desynchronize and one's term stops racing the other's reply. Preserve Election Safety +(DropStaleResponse itself is correct Raft — the bug is the escalation cadence, not the drop). + +## Session net (5 layers peeled on the round-4 stall; 4 fixed) +1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d). +2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a). +3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified prevote-win. +4. 5.6 RDHUP preempts in-flight election — FIXED (81cb09a), unit-verified, reduces churn. +5. REMAINING (task 5.7): vote-reply term-lag / unbounded term escalation under cold-connection churn + — the freshest survivor is granted but its term out-races the grant. New follow-up. +Consensus SAFETY intact throughout (no two leaders; DropStaleResponse correct; zero job loss every +completed round). Purely election LIVENESS convergence under sustained overlapping-kill churn. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 836630a..64b58d7 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -144,6 +144,15 @@ - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.5_ + - _PARTIAL FIX (commit 81cb09a): the RDHUP-entry now drops an RDHUP while role==CTRL_ELECTING or pre_vote_in_progress (don't preempt an in-flight election). Unit RED→GREEN (22 Tests/0 Failures); reduces the term churn. But the soak still stalls at round 4 → a deeper 5th layer (task 5.7): the freshest survivor is GRANTED but its term out-races the grant (vote_reply_stale_drop). Detail in `research.md` §"Task 5.6 fix VERIFIED"._ + +- [ ] 5.7 Stop the vote-reply term-lag / unbounded term escalation under cold-connection churn (NEW — the 5th layer under 5.6) + - _Root cause (`research.md` §"Task 5.6 fix VERIFIED"): with 5.6 in, round-4 ctrl-2 (freshest, 15@t5) wins PreVote, enters the real election, and ctrl-3 GRANTS it (vote_granted) — ctrl-3 votes FOR ctrl-2, not leapfrogging. But ctrl-2 never tallies a majority: every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's current term). ctrl-2's term escalates via repeated election timeouts faster than ctrl-3 can reply, so each grant is one term stale → dropped → timeout → term++ → self-perpetuating. A just-restarted ctrl-2 has COLD peer connections (each real-vote opens a fresh endpoint_connect+SSL) and races connect attempts to the dead ctrl-1, so the grant RTT exceeds the window before ctrl-2 re-escalates. election_timeout_ms=5000, jitter=200ms._ + - Fix direction: (a) do not bump the election term while a vote round-trip for the current term is still outstanding (let the grant land before re-escalating); and/or (b) warm/reuse the peer vote connection so the grant RTT < election_timeout (avoid per-vote connection setup); and/or (c) widen election-timeout jitter so the two survivors desynchronize. Preserve Election Safety (DropStaleResponse is correct Raft — fix the escalation cadence, not the drop), genuine-loss liveness, 5.2 suppression, and the 5.6 no-preempt guard. RED: a just-restarted freshest survivor + a willing grantor with a cold connection; assert it wins within a bounded number of election terms (term does not escalate unbounded). + - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (bounded term, no escalation stall) with all prior fixes in place — the green 7.2 gate + - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ + - _Boundary: Election Liveness_ + - _Depends: 5.6_ - [ ] 6. Integration: single-seam invariant end-to-end - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end @@ -171,7 +180,7 @@ - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); the remaining 7.2 blocker is now **task 5.6** — dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._ + - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); 5.6 (don't preempt an in-flight election) is FIXED (81cb09a, unit-verified, reduces churn); the remaining 7.2 blocker is now **task 5.7** — vote-reply term-lag / unbounded term escalation under cold-connection churn (the freshest survivor is granted but its term out-races the grant). Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure