lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): catching-up survivor must grant PreVote when primary gone (task 5.3)

Browse source 
The round-4 election stall surfaced by the 7.2 soak (bug-1272): after several
overlapping kills the only fully-up-to-date node is killed and the two survivors
are both restarted-and-lagging (in_catch_up=1) with no live primary left to catch
up FROM — so the flag never clears. Each holds the full committed prefix and is a
SAFE candidate, yet raft_cb_vote_eligible denied ALL PreVotes while in_catch_up
UNCONDITIONALLY, so they mutually denied and the election deadlocked forever.

Gate the in_catch_up deny on a LIVE primary (primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh), mirroring the heartbeat-fresh deny one block up. The
deny's purpose is to avoid disrupting a primary that is actively replicating to
us; once the primary is unreachable, raft_log_ok (already applied in Step 2 of
raft_on_pre_vote_request, before this callback) guarantees the candidate's log
>= ours, so granting a fresher candidate is safe (Leader Completeness preserved).

Adds 3 cases to test_ha_core_safety.c driving the real raft_on_pre_vote_request:
CORE RED (catching-up survivor + primary gone must GRANT a fresh candidate),
PIN (still DENY while a live primary feeds it), PIN (STALE candidate still DENIED
by logOk). test_ha_core_safety 20 Tests / 0 Failures; 5.1/5.2 unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 02:44:45 +00:00
parent da25ad53e1
commit bf88681fb0
2 changed files with 169 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

21
src/controller/ha.c
View file

@ -364,8 +364,25 @@ static int raft_cb_vote_eligible(void *ctx, int from_idx, uint64_t term)
ha_heartbeat_is_fresh(ha))
return 0;
/* In catch-up → deny */
if (atomic_load_explicit(&ha->in_catch_up, memory_order_acquire))
/* In catch-up WHILE a live primary is feeding us → deny.
*
* raft-ha-core-safety task 5.3 (bug-1272): this deny used to be
* UNCONDITIONAL, which deadlocks the election under overlapping churn — after
* several rapid kills the only fully-up-to-date node is killed and the two
* survivors are both restarted-and-lagging (in_catch_up=1) with NO live
* primary left to catch up FROM (so the flag never clears). Each holds the
* full committed prefix and is a SAFE candidate, yet they mutually deny every
* PreVote → permanent stall (round-N election stall, 7.2 soak).
*
* Gate the deny on a LIVE primary, mirroring the heartbeat-fresh deny above:
* the purpose of the in_catch_up deny is to avoid disrupting a primary that is
* ACTIVELY replicating to us — not to fence a survivor whose primary is gone.
* Once the primary is unreachable, raft_log_ok (already applied in Step 2 of
* raft_on_pre_vote_request / raft_on_vote_request, BEFORE this callback)
* guarantees the candidate's (lastLogTerm,lastLogIndex) >= ours, so granting a
* fresher candidate is safe (Leader Completeness preserved). */
if (atomic_load_explicit(&ha->in_catch_up, memory_order_acquire) &&
ha->primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha))
return 0;
/* Freshness gate requires knowing the candidate's config_tag/lsn.