lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(raft-ha-core-safety): 5.10b deeper — RDHUP failover storm (48 begin_failovers) defeats 5.6 point-in-time suppression

Browse source 
The round-4 term ladder is driven by repeated RDHUP->begin_failover->
PreVote-won->real-election(term++) cycles from the dead primary's socket, not
a behind-node vote loop. 5.6's instantaneous 'suppress while electing' guard is
insufficient under a sustained RDHUP cadence. Fix direction: per-failover
cooldown/debounce + hold the failover latch across the full prevote->election->
settle window. Needs a deterministic dead-primary-cadence repro.
This commit is contained in:
Rongsong Shen 2026-07-01 01:21:48 +00:00
parent 9944c689db
commit cc80e65288
1 changed files with 26 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

26
.kiro/specs/raft-ha-core-safety/research.md
View file

@ -885,3 +885,29 @@ primary killed; assert the cluster elects within a BOUNDED number of terms, no u
freshness-deny escalation) before any fix — speculative tuning here risks regressing 5.5/5.6/5.7.
Stochastic: this tail appeared in 1 of 2 ten-round soaks (the other laddered single-increment
terms 4→13). Tracked as task 5.10(b).
### 5.10(b) DEEPER (2026-06-30) — it's an RDHUP-triggered failover STORM, not a lone behind-node loop
Log forensics on the escalating survivor (ctrl-2, PID 511891) over the round-4 window:
`begin_failover` ×48, `pre_vote_won` ×24, `event=election_start` ×24, `rdhup` ×27. So the
term ladder is driven by REPEATED `ha_on_primary_ctrl_fd_rdhup → ha_begin_failover →
ha_start_prevote_async → PreVote WON → enter real election (term++)` cycles — roughly one term
bump per RDHUP from the dead primary's socket. PreVote WINS each time (so PreVote's freshness
gate is not the lever here) and the real election then churns; the driver is at the
FAILOVER-TRIGGER level.
Task 5.6 added "drop an RDHUP while role==CTRL_ELECTING || pre_vote_in_progress" (commit
81cb09a) to stop exactly this. Under the round-4 overlap it is INSUFFICIENT: 48 begin_failovers
got through, so the node must be repeatedly OUTSIDE the electing/prevote window when an RDHUP
lands (e.g. brief returns to CTRL_STANDBY between a lost real election and the next trigger, or
the RDHUP arrives on a re-established fd the guard doesn't cover). Net: a sustained RDHUP cadence
from a dead primary defeats the point-in-time suppression and ladders the term.
Fix direction (5.6-family, NOT a quick patch): make the RDHUP→failover suppression robust to a
SUSTAINED cadence — e.g. a per-failover cooldown / debounce so a fresh RDHUP cannot re-trigger a
new PreVote+election until the prior attempt has fully resolved OR a minimum interval elapsed;
and/or hold the "failover in progress" latch across the whole prevote→real-election→settle
window (not just the instantaneous role check). Must preserve genuine-loss liveness (R6) and not
re-introduce the stall 5.2/5.5/5.6/5.7 fixed. Needs a deterministic repro: a dead-primary RDHUP
cadence + 2 survivors → assert bounded term growth (≤ K bumps) to convergence. Stochastic in the
soak (1 of 2 ten-round runs). Tracked as 5.10(b).