fix(controller): debounce RDHUP-triggered failover — stop the term-escalation storm (bug-1310, task 5.10b)
A dead primary's control fd RDHUPs faster than an election settles. The 5.6 guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/ prevote; an RDHUP landing in the gap after a lost election (role briefly back to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real election (term++). One bump per RDHUP -> unbounded escalation under a sustained cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the harness window). Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the last trigger (closes the dead fd, returns) so the election's own jittered retry (5.7) runs without the external RDHUP cadence adding term bumps. First trigger never debounced; genuine loss still fails over, at most once per window. Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary (last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect, terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no regression. The stochastic storm did not recur this run (debounce fired 0x), so soak-reproduction of the storm is still pending; the mechanism rests on the unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0, election 28/0, failover_prevote 6/0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
This commit is contained in:
parent
cc80e65288
commit
0d70d5e412
5 changed files with 175 additions and 1 deletions
|
|
@ -21809,6 +21809,76 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-01T00:42:13.591Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1314",
|
||||
"timestamp": "2026-07-01T01:24:37.398Z",
|
||||
"error_message": "Missing error handling in unknown",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
|
||||
"fix": "Added try/catch block",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"error-handling",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-01T01:24:37.398Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1315",
|
||||
"timestamp": "2026-07-01T01:26:38.341Z",
|
||||
"error_message": "Wrong reference: unistd should be time",
|
||||
"file": "src/controller/testsuite/test_ha_rdhup_integration.c",
|
||||
"root_cause": "Used \"unistd\" instead of \"time\"",
|
||||
"fix": "Changed unistd → time",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-reference",
|
||||
"c"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-01T01:26:38.341Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1310",
|
||||
"timestamp": "2026-06-30T18:40:00Z",
|
||||
"error_message": "chaotic soak round-4 election-escalation storm: terms laddered 6->24 (~18 bumps), 48 begin_failovers, round overran the harness window",
|
||||
"file": "src/controller/ha.c (ha_on_primary_ctrl_fd_rdhup)",
|
||||
"root_cause": "A dead primary's control fd RDHUPs faster than an election settles. The 5.6 guard only drops an RDHUP while role is INSTANTANEOUSLY CTRL_ELECTING/prevote; an RDHUP landing in the gap after a lost election (role briefly back to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote(won) -> real election(term++). One term bump per RDHUP -> unbounded escalation under a sustained cadence. NOT RNG (seed is /dev/urandom).",
|
||||
"fix": "5.10b RDHUP failover DEBOUNCE: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover; ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the last trigger (closes the dead fd, returns) so the election's own jittered retry (5.7) runs without the external RDHUP cadence adding term bumps. First trigger never debounced; genuine loss still fails over, at most once per window. Unit-proven: test_rdhup_within_cooldown_is_debounced (recent last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary (last_failover=0 -> CTRL_PRIMARY) = differential proof. Soak: 10/10 elect, terms 4->14 bounded, begin_failover 14/14/12 (was 48), no regression — but the stochastic storm did NOT recur this run (debounce fired 0x), so end-to-end storm suppression is unit-proven, not yet soak-reproduced.",
|
||||
"tags": [
|
||||
"election-liveness",
|
||||
"rdhup",
|
||||
"term-escalation",
|
||||
"debounce",
|
||||
"5.10b",
|
||||
"raft"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1296",
|
||||
"bug-1306"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T18:40:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1317",
|
||||
"timestamp": "2026-07-01T01:32:08.631Z",
|
||||
"error_message": "Missing error handling in unknown",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
|
||||
"fix": "Added try/catch block",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"error-handling",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-01T01:32:08.631Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue