diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index f0f2287d..69adb0a7 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -885,3 +885,29 @@ primary killed; assert the cluster elects within a BOUNDED number of terms, no u freshness-deny escalation) before any fix — speculative tuning here risks regressing 5.5/5.6/5.7. Stochastic: this tail appeared in 1 of 2 ten-round soaks (the other laddered single-increment terms 4→13). Tracked as task 5.10(b). + +### 5.10(b) DEEPER (2026-06-30) — it's an RDHUP-triggered failover STORM, not a lone behind-node loop + +Log forensics on the escalating survivor (ctrl-2, PID 511891) over the round-4 window: +`begin_failover` ×48, `pre_vote_won` ×24, `event=election_start` ×24, `rdhup` ×27. So the +term ladder is driven by REPEATED `ha_on_primary_ctrl_fd_rdhup → ha_begin_failover → +ha_start_prevote_async → PreVote WON → enter real election (term++)` cycles — roughly one term +bump per RDHUP from the dead primary's socket. PreVote WINS each time (so PreVote's freshness +gate is not the lever here) and the real election then churns; the driver is at the +FAILOVER-TRIGGER level. + +Task 5.6 added "drop an RDHUP while role==CTRL_ELECTING || pre_vote_in_progress" (commit +81cb09a) to stop exactly this. Under the round-4 overlap it is INSUFFICIENT: 48 begin_failovers +got through, so the node must be repeatedly OUTSIDE the electing/prevote window when an RDHUP +lands (e.g. brief returns to CTRL_STANDBY between a lost real election and the next trigger, or +the RDHUP arrives on a re-established fd the guard doesn't cover). Net: a sustained RDHUP cadence +from a dead primary defeats the point-in-time suppression and ladders the term. + +Fix direction (5.6-family, NOT a quick patch): make the RDHUP→failover suppression robust to a +SUSTAINED cadence — e.g. a per-failover cooldown / debounce so a fresh RDHUP cannot re-trigger a +new PreVote+election until the prior attempt has fully resolved OR a minimum interval elapsed; +and/or hold the "failover in progress" latch across the whole prevote→real-election→settle +window (not just the instantaneous role check). Must preserve genuine-loss liveness (R6) and not +re-introduce the stall 5.2/5.5/5.6/5.7 fixed. Needs a deterministic repro: a dead-primary RDHUP +cadence + 2 survivors → assert bounded term growth (≤ K bumps) to convergence. Stochastic in the +soak (1 of 2 ten-round runs). Tracked as 5.10(b).