lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): catching-up survivor must grant PreVote when primary gone (task 5.3)

Browse source 
The round-4 election stall surfaced by the 7.2 soak (bug-1272): after several
overlapping kills the only fully-up-to-date node is killed and the two survivors
are both restarted-and-lagging (in_catch_up=1) with no live primary left to catch
up FROM — so the flag never clears. Each holds the full committed prefix and is a
SAFE candidate, yet raft_cb_vote_eligible denied ALL PreVotes while in_catch_up
UNCONDITIONALLY, so they mutually denied and the election deadlocked forever.

Gate the in_catch_up deny on a LIVE primary (primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh), mirroring the heartbeat-fresh deny one block up. The
deny's purpose is to avoid disrupting a primary that is actively replicating to
us; once the primary is unreachable, raft_log_ok (already applied in Step 2 of
raft_on_pre_vote_request, before this callback) guarantees the candidate's log
>= ours, so granting a fresher candidate is safe (Leader Completeness preserved).

Adds 3 cases to test_ha_core_safety.c driving the real raft_on_pre_vote_request:
CORE RED (catching-up survivor + primary gone must GRANT a fresh candidate),
PIN (still DENY while a live primary feeds it), PIN (STALE candidate still DENIED
by logOk). test_ha_core_safety 20 Tests / 0 Failures; 5.1/5.2 unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 02:44:45 +00:00
parent da25ad53e1
commit bf88681fb0
2 changed files with 169 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

21
src/controller/ha.c
View file

@ -364,8 +364,25 @@ static int raft_cb_vote_eligible(void *ctx, int from_idx, uint64_t term)
ha_heartbeat_is_fresh(ha))
return 0;
/* In catch-up → deny */
if (atomic_load_explicit(&ha->in_catch_up, memory_order_acquire))
/* In catch-up WHILE a live primary is feeding us → deny.
*
* raft-ha-core-safety task 5.3 (bug-1272): this deny used to be
* UNCONDITIONAL, which deadlocks the election under overlapping churn — after
* several rapid kills the only fully-up-to-date node is killed and the two
* survivors are both restarted-and-lagging (in_catch_up=1) with NO live
* primary left to catch up FROM (so the flag never clears). Each holds the
* full committed prefix and is a SAFE candidate, yet they mutually deny every
* PreVote → permanent stall (round-N election stall, 7.2 soak).
*
* Gate the deny on a LIVE primary, mirroring the heartbeat-fresh deny above:
* the purpose of the in_catch_up deny is to avoid disrupting a primary that is
* ACTIVELY replicating to us — not to fence a survivor whose primary is gone.
* Once the primary is unreachable, raft_log_ok (already applied in Step 2 of
* raft_on_pre_vote_request / raft_on_vote_request, BEFORE this callback)
* guarantees the candidate's (lastLogTerm,lastLogIndex) >= ours, so granting a
* fresher candidate is safe (Leader Completeness preserved). */
if (atomic_load_explicit(&ha->in_catch_up, memory_order_acquire) &&
ha->primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha))
return 0;
/* Freshness gate requires knowing the candidate's config_tag/lsn.

150
src/controller/testsuite/test_ha_core_safety.c
View file

@ -1597,6 +1597,147 @@ static void test_5_1_election_safety_one_primary_per_term(void)
printf("test_5_1_election_safety_one_primary_per_term: done\n");
}
/* ═══════════════════════════════════════════════════════════════════════════
* TASK 5.3 — lagging-but-safe survivor must be electable under overlapping churn
*
* Surfaced by the 7.2 acceptance soak (research.md §"Task 7.2"; bug-1272), a
* DISTINCT defect from the 5.2 flap. After ≥3 overlapping kills the only fully-
* up-to-date node is killed and the two survivors are both restarted-and-lagging
* (e.g. tips 15 and 11 after the leader at 19 dies), both flagged in_catch_up.
* Each holds the FULL committed prefix (Leader Completeness satisfied) and is a
* SAFE candidate — yet neither can win a PreVote, so the election deadlocks and
* no primary is elected.
*
* Root cause: raft_cb_vote_eligible (ha.c) denies ALL PreVotes while in_catch_up
* UNCONDITIONALLY — unlike the heartbeat-fresh deny one block up, which correctly
* requires a LIVE primary (primary_ctrl_fd>=0). Two lagging survivors whose
* primary just died are both in_catch_up with no primary left to catch up FROM
* (so the flag never clears) → they mutually deny → permanent stall.
*
* These cases drive the REAL library PreVote-answer path
* (raft_on_pre_vote_request → raft_cb_vote_eligible, the same path cluster.c
* uses) over the wired hcs4 cluster fixture:
*
* (CORE RED) test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone:
* a catching-up survivor with NO live primary must GRANT a sufficiently-
* fresh PreVote. FAILs on baseline (the unconditional in_catch_up deny);
* GREEN when the deny is gated on a live primary.
*
* (PIN) test_5_3_catchup_denies_while_live_primary_feeds: while a LIVE primary
* feeds this catching-up node (fd>=0 + fresh heartbeat) the PreVote is still
* DENIED — the fix lifts the deny ONLY once the primary is gone.
*
* (PIN) test_5_3_stale_candidate_denied_even_with_primary_gone: even with the
* primary gone, a STALE candidate (tip below the voter) is DENIED by logOk
* — lifting the in_catch_up deny must NOT weaken Leader Completeness.
*
* Requirements: 4.1, 4.3, 6.1, 6.2, 6.3.
* ═══════════════════════════════════════════════════════════════════════════ */
/* Build a recovered LAGGING survivor (the round-4 ctrl-2/ctrl-3 shape): committed
* log [1..tip] @ a single old term, raft log_len=tip, in_catch_up=1 (restarted,
* was catching up), and NO live primary (primary_ctrl_fd=-1, no fresh heartbeat —
* the primary it caught up FROM just died). */
static cluster_t* hcs5_lagging_survivor_up(const char* suffix, uint64_t voter_term,
uint64_t log_term, uint64_t tip)
{
cluster_t* c = hcs4_cluster_up(suffix);
if (!c)
return NULL;
hcs4_make_follower(c, voter_term, /*n_peers*/3);
uint8_t full[HCS_FULL_LEN];
hcs_fill_body(full);
uint8_t blob[sizeof(raft_entry_blob_t) + HCS_FULL_LEN];
for (uint64_t lsn = 1; lsn <= tip; lsn++) {
size_t n = hcs4_pack_entry(blob, WAL_JOB_SUBMIT, full, HCS_FULL_LEN);
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
ha_cb_persist_entry_for_test(&c->ha, lsn, log_term, blob, n),
"seed write must persist (real fs required for inline fdatasync — bug-924)");
}
c->ha.raft.log_len = tip;
atomic_store_explicit(&c->ha.in_catch_up, 1, memory_order_release);
c->ha.primary_ctrl_fd = -1; /* the primary is DEAD */
c->ha.last_activity = 0; /* no fresh heartbeat */
return c;
}
/* CORE RED — a catching-up survivor with no live primary must GRANT a fresh PreVote. */
static void test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone(void)
{
/* Voter = ctrl-3 shape: committed [1..11] @ term 4, in_catch_up, primary dead. */
cluster_t* c = hcs5_lagging_survivor_up("s53core", /*voter_term*/4,
/*log_term*/4, /*tip*/11);
TEST_ASSERT_NOT_NULL(c);
/* A FRESHER survivor (ctrl-2 shape, tip 15 @ term 4) asks for a PreVote at a
* higher prospective term. logOk passes (15 >= 11 at the same term). */
int rc = raft_on_pre_vote_request(&c->ha.raft, /*from_idx*/1,
/*cand_term*/5, /*cand_last_log_term*/4,
/*cand_last_log_index*/15);
TEST_ASSERT_EQUAL_INT_MESSAGE(RAFT_GRANT, rc,
"RED: a catching-up survivor with NO live primary must GRANT a "
"sufficiently-fresh PreVote (logOk already enforces Leader Completeness). "
"The unconditional in_catch_up deny (ha.c raft_cb_vote_eligible) refuses "
"it, so two lagging survivors whose primary just died mutually deny and "
"the round-4 election deadlocks (task 5.3 / bug-1272). The fix gates the "
"in_catch_up deny on a LIVE primary, mirroring the heartbeat-fresh deny.");
hcs4_cluster_down(c);
printf("test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone: done\n");
}
/* PIN — while a LIVE primary feeds the catching-up node, the PreVote is still denied. */
static void test_5_3_catchup_denies_while_live_primary_feeds(void)
{
cluster_t* c = hcs5_lagging_survivor_up("s53live", 4, 4, 11);
TEST_ASSERT_NOT_NULL(c);
/* A LIVE primary is actively feeding this catching-up node: fresh heartbeat
* AND a live control fd. The in_catch_up deny MUST still hold — do not disrupt
* active replication with a competing election (the deny's original purpose). */
c->ha.config.heartbeat_timeout_ms = 6000;
struct timespec now;
clock_gettime(CLOCK_MONOTONIC, &now);
c->ha.last_activity = now.tv_sec; /* fresh heartbeat */
c->ha.primary_ctrl_fd = 999; /* a live control fd (>=0; never closed here) */
TEST_ASSERT_TRUE_MESSAGE(ha_heartbeat_is_fresh(&c->ha),
"precondition: the primary reads as alive (fresh heartbeat)");
int rc = raft_on_pre_vote_request(&c->ha.raft, 1, 5, 4, 15);
TEST_ASSERT_EQUAL_INT_MESSAGE(RAFT_DENY, rc,
"PIN: while a LIVE primary feeds this catching-up node (fd>=0 + fresh "
"heartbeat) a PreVote must still be DENIED — the fix lifts the in_catch_up "
"deny ONLY once the primary is gone (don't disrupt active replication)");
c->ha.primary_ctrl_fd = -1; /* defensive: never let teardown touch fd 999 */
hcs4_cluster_down(c);
printf("test_5_3_catchup_denies_while_live_primary_feeds: PASS (pin)\n");
}
/* PIN — even with the primary gone, a STALE candidate is denied (Leader Completeness). */
static void test_5_3_stale_candidate_denied_even_with_primary_gone(void)
{
cluster_t* c = hcs5_lagging_survivor_up("s53stale", 4, 4, 11);
TEST_ASSERT_NOT_NULL(c);
/* Primary gone (the in_catch_up deny is lifted by the fix), but the candidate
* is STALE — its tip (8) is BELOW the voter's (11). logOk (raft_log_ok, Step 2
* of raft_on_pre_vote_request) must still DENY: the fix must never let a behind
* candidate win. */
int rc = raft_on_pre_vote_request(&c->ha.raft, 1, /*cand_term*/5,
/*cand_last_log_term*/4,
/*cand_last_log_index*/8);
TEST_ASSERT_EQUAL_INT_MESSAGE(RAFT_DENY, rc,
"PIN: even with the primary gone, a STALE candidate (tip 8 < voter 11) "
"must be DENIED by logOk — lifting the in_catch_up deny must NOT weaken "
"Leader Completeness");
hcs4_cluster_down(c);
printf("test_5_3_stale_candidate_denied_even_with_primary_gone: PASS (pin)\n");
}
/* ═══════════════════════════════════════════════════════════════════════════
* SCAFFOLDING (PASS) — prove each shared seam is usable by later phases.
* ═══════════════════════════════════════════════════════════════════════════ */
@ -1753,6 +1894,15 @@ void test_ha_core_safety(void)
RUN_TEST(test_5_1_spurious_failover_suppressed_while_primary_alive);
RUN_TEST(test_5_1_genuine_loss_elects_and_advances_term);
RUN_TEST(test_5_1_election_safety_one_primary_per_term);
/* Task 5.3 lagging-but-safe survivor electability under overlapping churn:
* CORE RED — a catching-up survivor with NO live primary must GRANT a
* sufficiently-fresh PreVote (FAILs on baseline: unconditional
* in_catch_up deny deadlocks two lagging survivors — bug-1272).
* PIN — while a LIVE primary feeds it, the PreVote is still denied.
* PIN — a STALE candidate is denied by logOk even with primary gone. */
RUN_TEST(test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone);
RUN_TEST(test_5_3_catchup_denies_while_live_primary_feeds);
RUN_TEST(test_5_3_stale_candidate_denied_even_with_primary_gone);
/* Scaffolding: each shared seam is usable by 2.3 / 4.1 / 5.1. */
RUN_TEST(test_seam_wal_construct_and_probe);
RUN_TEST(test_seam_divergence_is_observable);