fix(controller): debounce RDHUP-triggered failover — stop the term-escalation storm (bug-1310, task 5.10b)
A dead primary's control fd RDHUPs faster than an election settles. The 5.6 guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/ prevote; an RDHUP landing in the gap after a lost election (role briefly back to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real election (term++). One bump per RDHUP -> unbounded escalation under a sustained cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the harness window). Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the last trigger (closes the dead fd, returns) so the election's own jittered retry (5.7) runs without the external RDHUP cadence adding term bumps. First trigger never debounced; genuine loss still fails over, at most once per window. Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary (last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect, terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no regression. The stochastic storm did not recur this run (debounce fired 0x), so soak-reproduction of the storm is still pending; the mechanism rests on the unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0, election 28/0, failover_prevote 6/0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
This commit is contained in:
parent
cc80e65288
commit
0d70d5e412
5 changed files with 175 additions and 1 deletions
|
|
@ -172,7 +172,8 @@
|
|||
- _Boundary: Commit durability (raft-apply-decoupling item #1). Full canonical inversion (async keystone + apply_committed_entry seam + facts + restart validation) = the follow-on ha spec._
|
||||
- _Depends: 5.8_
|
||||
|
||||
- [ ] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag + election-escalation tail
|
||||
- [ ] 5.10 Residual soak FAILs after 5.9 (NOT job loss) — divergence lag [5.10a, OPEN] + election-escalation tail [5.10b, FIX LANDED]
|
||||
- _5.10b DONE (bug-1310, commit pending): RDHUP failover DEBOUNCE. Root cause = a dead primary's fd RDHUPs faster than an election settles; the 5.6 point-in-time guard misses RDHUPs that land after a lost election (role briefly back to STANDBY) → term++ per RDHUP (48 begin_failovers, terms 6→24). Fix: ha_state_t.last_failover_ms stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2×election_timeout of the last trigger. Unit-proven (test_rdhup_within_cooldown_is_debounced); soak 10/10 elect, terms 4→14 bounded, begin_failover 14/14/12 (was 48), no regression — but the stochastic storm did not recur this run (debounce fired 0×), so soak-reproduction of the storm is still pending. 5.10a (divergence catch-up lag) remains OPEN below._
|
||||
- _(a) committed_divergence>0 from the just-restarted node's catch-up LAG (walcompare `present=2`: entry on the 2 live nodes, not yet backfilled on the restarted one). Transient, settles; NOT lost data. Closed by entity-bearing snapshots / the full raft-apply-decoupling apply seam (EXTERNAL — ha-commit-durability / ha-convergence-recovery). (b) Election-escalation TAIL: under an adversarial overlap (kill inside a split-vote window) a round can still stall + ladder terms (observed round-4 term 6→25 before electing) despite 5.7's jitter — liveness held (elected next round) but degraded. Harden: cap/dampen term escalation or widen the jitter under repeated same-round failovers._
|
||||
- Observable completion: a 10-round chaotic soak verdict is committed_divergence=0 with no per-round election stall — the green 7.2 gate
|
||||
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue