lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): suppress spurious PreVote failover on transient RDHUP (task 5.2)

Browse source 
A recovered standby whose control channel flaps (transient RDHUP) while a
current-term primary is alive treated each flap as "primary died" and launched
an unwinnable PreVote failover (peers see the live primary and deny, the round
times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun
forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5").

Add a freshness guard at the RDHUP failover-initiation path
(ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh,
stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the
primary's reconnect instead of campaigning — the same guard
ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine
loss (stale heartbeat) falls through to failover unchanged; no term bump / no
role change, so Election Safety is untouched.

Keeping the standby attached lets catch-up complete, which clears in_catch_up
and lets votes flow again — so this one guard also resolves the downstream
catching-up-node vote-deny deadlock (obs 1240).

Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures
(spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 →
1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 02:03:50 +00:00
parent 806257afcf
commit 01419e1310
2 changed files with 35 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

32
src/controller/ha.c
View file

@ -6722,6 +6722,38 @@ void ha_on_primary_ctrl_fd_rdhup(ha_state_t *ha, cluster_t *c)
ha->primary_ctrl_fd = -1;
}
/* raft-ha-core-safety task 5.2: SPURIOUS-FAILOVER SUPPRESSION.
*
* A TRANSIENT control-channel RDHUP while the current-term primary is still
* alive (a heartbeat arrived within heartbeat_timeout_ms) must NOT launch an
* unwinnable PreVote failover. The live stall (research.md §"Phase 4/5"): a
* recovered standby whose channel to the live primary flaps treats each
* RDHUP as "primary died" → PreVote → peers see the primary alive and deny →
* round times out → reconnect → repeat ~15s, so the node prevote-spins
* forever instead of rejoining. Mirror the guard ha_failover_stagger_fn
* already applies on the DEFERRED path (a flap = the primary re-established
* the channel, so ha_heartbeat_is_fresh is the right liveness signal —
* the same one handle_request_vote uses to deny same-term challengers).
*
* Stay CTRL_STANDBY and await the primary's reconnect (handle_ctrlhello);
* re-arm the liveness watchdog so a GENUINE subsequent loss is still caught
* (the watchdog resets last_activity to now, granting a fresh grace window —
* if the primary truly never heartbeats again it fires a real failover after
* heartbeat_timeout_ms). GENUINE loss (stale heartbeat) falls through to the
* failover path below unchanged; Election Safety is unaffected (no term bump,
* no role change here). */
ctrl_role_t rdhup_role = (ctrl_role_t)atomic_load_explicit(&ha->role,
memory_order_acquire);
if (rdhup_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha)) {
INFO("HA: primary_ctrl_fd RDHUP but the primary is alive (fresh "
"heartbeat) — suppressing PreVote failover, staying CTRL_STANDBY "
"and awaiting reconnect (spurious-failover suppression, "
"raft-ha-core-safety 5.2)");
if (c)
ha_arm_heartbeat_timeout_vtask(ha, c);
return;
}
/* ha-prevote task 4.2: the failover trigger NO LONGER transitions to
* CTRL_ELECTING or truncates the WAL inline. It starts a PreVote round and
* STAYS CTRL_STANDBY; the destructive truncate + CTRL_ELECTING sequence is