myxce/.kiro/specs
Rongsong Shen 0d70d5e412 fix(controller): debounce RDHUP-triggered failover — stop the term-escalation storm (bug-1310, task 5.10b)
A dead primary's control fd RDHUPs faster than an election settles. The 5.6
guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/
prevote; an RDHUP landing in the gap after a lost election (role briefly back
to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real
election (term++). One bump per RDHUP -> unbounded escalation under a sustained
cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the
harness window).

Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover;
ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the
last trigger (closes the dead fd, returns) so the election's own jittered retry
(5.7) runs without the external RDHUP cadence adding term bumps. First trigger
never debounced; genuine loss still fails over, at most once per window.

Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent
last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary
(last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect,
terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no
regression. The stochastic storm did not recur this run (debounce fired 0x), so
soak-reproduction of the storm is still pending; the mechanism rests on the
unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0,
election 28/0, failover_prevote 6/0).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
2026-07-01 01:32:27 +00:00
..
config-upgrade-snapshot docs(specs): gap re-validation 2026-06-24 (5 HA/raft specs) 2026-06-24 10:04:06 +00:00
controller-status checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
election-iomgr-refactor checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-commit-barrier-rollout checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-commit-durability docs(ha-commit-durability): sharpen bug-1306 root cause — xce_run_hooks clobbers commit -ETIMEDOUT 2026-07-01 00:21:23 +00:00
ha-commit-engine checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-config-bootstrap checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-consistency-invariants checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-convergence-recovery test(ha-convergence-recovery): close Task 8 — live /xce-test-ha 8-round PASS 2026-06-24 10:17:11 +00:00
ha-elect-ctx-lifecycle checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-heartbeat-committed-propagation docs(ha-heartbeat-committed-propagation): mark 4.1 pending live-cluster validation (manual env) 2026-06-20 12:32:34 +00:00
ha-heartbeat-liveness checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-peer-connection-reuse checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-prevote checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-repl-hardening docs(specs): gap re-validation 2026-06-24 (5 HA/raft specs) 2026-06-24 10:04:06 +00:00
ha-single-leader-role checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
ha-term-safety chore(ha-term-safety): mark tasks 5.2/5.4/5.5 complete — live verification done 2026-06-29 08:19:14 +00:00
ha-wal-replication checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
libxce-failover checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
raft-commit-safety docs(specs): gap re-validation 2026-06-24 (5 HA/raft specs) 2026-06-24 10:04:06 +00:00
raft-drive-async chore(raft-drive-async): mark task 4.3 complete — thread-role compliance verified 2026-06-29 04:39:50 +00:00
raft-ha-core-safety fix(controller): debounce RDHUP-triggered failover — stop the term-escalation storm (bug-1310, task 5.10b) 2026-07-01 01:32:27 +00:00
raft-library checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
raft-log-replication test(raft-log-replication): close Req 8.3 node-restart with sim restart primitive 2026-06-21 21:49:20 +00:00
raft-sim-conformance feat(raft-sim-conformance): optional TLC-consumable trace export, default-off (task 8.1) 2026-06-24 16:49:36 +00:00
resval-id-decoupling checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
scheduler-controller-msg-gap checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
worker-join-refactor checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
xce-test-e2e-primitives checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
.gitignore chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family 2026-06-11 13:21:50 +00:00