From bf88681fb03825914a15b481a4e94a7634ce36f0 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 02:44:45 +0000 Subject: [PATCH] fix(raft-ha-core-safety): catching-up survivor must grant PreVote when primary gone (task 5.3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-4 election stall surfaced by the 7.2 soak (bug-1272): after several overlapping kills the only fully-up-to-date node is killed and the two survivors are both restarted-and-lagging (in_catch_up=1) with no live primary left to catch up FROM — so the flag never clears. Each holds the full committed prefix and is a SAFE candidate, yet raft_cb_vote_eligible denied ALL PreVotes while in_catch_up UNCONDITIONALLY, so they mutually denied and the election deadlocked forever. Gate the in_catch_up deny on a LIVE primary (primary_ctrl_fd>=0 && ha_heartbeat_is_fresh), mirroring the heartbeat-fresh deny one block up. The deny's purpose is to avoid disrupting a primary that is actively replicating to us; once the primary is unreachable, raft_log_ok (already applied in Step 2 of raft_on_pre_vote_request, before this callback) guarantees the candidate's log >= ours, so granting a fresher candidate is safe (Leader Completeness preserved). Adds 3 cases to test_ha_core_safety.c driving the real raft_on_pre_vote_request: CORE RED (catching-up survivor + primary gone must GRANT a fresh candidate), PIN (still DENY while a live primary feeds it), PIN (STALE candidate still DENIED by logOk). test_ha_core_safety 20 Tests / 0 Failures; 5.1/5.2 unaffected. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC --- src/controller/ha.c | 21 ++- .../testsuite/test_ha_core_safety.c | 150 ++++++++++++++++++ 2 files changed, 169 insertions(+), 2 deletions(-) diff --git a/src/controller/ha.c b/src/controller/ha.c index 8acc889..9230b40 100644 --- a/src/controller/ha.c +++ b/src/controller/ha.c @@ -364,8 +364,25 @@ static int raft_cb_vote_eligible(void *ctx, int from_idx, uint64_t term) ha_heartbeat_is_fresh(ha)) return 0; - /* In catch-up → deny */ - if (atomic_load_explicit(&ha->in_catch_up, memory_order_acquire)) + /* In catch-up WHILE a live primary is feeding us → deny. + * + * raft-ha-core-safety task 5.3 (bug-1272): this deny used to be + * UNCONDITIONAL, which deadlocks the election under overlapping churn — after + * several rapid kills the only fully-up-to-date node is killed and the two + * survivors are both restarted-and-lagging (in_catch_up=1) with NO live + * primary left to catch up FROM (so the flag never clears). Each holds the + * full committed prefix and is a SAFE candidate, yet they mutually deny every + * PreVote → permanent stall (round-N election stall, 7.2 soak). + * + * Gate the deny on a LIVE primary, mirroring the heartbeat-fresh deny above: + * the purpose of the in_catch_up deny is to avoid disrupting a primary that is + * ACTIVELY replicating to us — not to fence a survivor whose primary is gone. + * Once the primary is unreachable, raft_log_ok (already applied in Step 2 of + * raft_on_pre_vote_request / raft_on_vote_request, BEFORE this callback) + * guarantees the candidate's (lastLogTerm,lastLogIndex) >= ours, so granting a + * fresher candidate is safe (Leader Completeness preserved). */ + if (atomic_load_explicit(&ha->in_catch_up, memory_order_acquire) && + ha->primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha)) return 0; /* Freshness gate requires knowing the candidate's config_tag/lsn. diff --git a/src/controller/testsuite/test_ha_core_safety.c b/src/controller/testsuite/test_ha_core_safety.c index 23814e0..84d7e8b 100644 --- a/src/controller/testsuite/test_ha_core_safety.c +++ b/src/controller/testsuite/test_ha_core_safety.c @@ -1597,6 +1597,147 @@ static void test_5_1_election_safety_one_primary_per_term(void) printf("test_5_1_election_safety_one_primary_per_term: done\n"); } +/* ═══════════════════════════════════════════════════════════════════════════ + * TASK 5.3 — lagging-but-safe survivor must be electable under overlapping churn + * + * Surfaced by the 7.2 acceptance soak (research.md §"Task 7.2"; bug-1272), a + * DISTINCT defect from the 5.2 flap. After ≥3 overlapping kills the only fully- + * up-to-date node is killed and the two survivors are both restarted-and-lagging + * (e.g. tips 15 and 11 after the leader at 19 dies), both flagged in_catch_up. + * Each holds the FULL committed prefix (Leader Completeness satisfied) and is a + * SAFE candidate — yet neither can win a PreVote, so the election deadlocks and + * no primary is elected. + * + * Root cause: raft_cb_vote_eligible (ha.c) denies ALL PreVotes while in_catch_up + * UNCONDITIONALLY — unlike the heartbeat-fresh deny one block up, which correctly + * requires a LIVE primary (primary_ctrl_fd>=0). Two lagging survivors whose + * primary just died are both in_catch_up with no primary left to catch up FROM + * (so the flag never clears) → they mutually deny → permanent stall. + * + * These cases drive the REAL library PreVote-answer path + * (raft_on_pre_vote_request → raft_cb_vote_eligible, the same path cluster.c + * uses) over the wired hcs4 cluster fixture: + * + * (CORE RED) test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone: + * a catching-up survivor with NO live primary must GRANT a sufficiently- + * fresh PreVote. FAILs on baseline (the unconditional in_catch_up deny); + * GREEN when the deny is gated on a live primary. + * + * (PIN) test_5_3_catchup_denies_while_live_primary_feeds: while a LIVE primary + * feeds this catching-up node (fd>=0 + fresh heartbeat) the PreVote is still + * DENIED — the fix lifts the deny ONLY once the primary is gone. + * + * (PIN) test_5_3_stale_candidate_denied_even_with_primary_gone: even with the + * primary gone, a STALE candidate (tip below the voter) is DENIED by logOk + * — lifting the in_catch_up deny must NOT weaken Leader Completeness. + * + * Requirements: 4.1, 4.3, 6.1, 6.2, 6.3. + * ═══════════════════════════════════════════════════════════════════════════ */ + +/* Build a recovered LAGGING survivor (the round-4 ctrl-2/ctrl-3 shape): committed + * log [1..tip] @ a single old term, raft log_len=tip, in_catch_up=1 (restarted, + * was catching up), and NO live primary (primary_ctrl_fd=-1, no fresh heartbeat — + * the primary it caught up FROM just died). */ +static cluster_t* hcs5_lagging_survivor_up(const char* suffix, uint64_t voter_term, + uint64_t log_term, uint64_t tip) +{ + cluster_t* c = hcs4_cluster_up(suffix); + if (!c) + return NULL; + hcs4_make_follower(c, voter_term, /*n_peers*/3); + + uint8_t full[HCS_FULL_LEN]; + hcs_fill_body(full); + uint8_t blob[sizeof(raft_entry_blob_t) + HCS_FULL_LEN]; + for (uint64_t lsn = 1; lsn <= tip; lsn++) { + size_t n = hcs4_pack_entry(blob, WAL_JOB_SUBMIT, full, HCS_FULL_LEN); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, + ha_cb_persist_entry_for_test(&c->ha, lsn, log_term, blob, n), + "seed write must persist (real fs required for inline fdatasync — bug-924)"); + } + c->ha.raft.log_len = tip; + atomic_store_explicit(&c->ha.in_catch_up, 1, memory_order_release); + c->ha.primary_ctrl_fd = -1; /* the primary is DEAD */ + c->ha.last_activity = 0; /* no fresh heartbeat */ + return c; +} + +/* CORE RED — a catching-up survivor with no live primary must GRANT a fresh PreVote. */ +static void test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone(void) +{ + /* Voter = ctrl-3 shape: committed [1..11] @ term 4, in_catch_up, primary dead. */ + cluster_t* c = hcs5_lagging_survivor_up("s53core", /*voter_term*/4, + /*log_term*/4, /*tip*/11); + TEST_ASSERT_NOT_NULL(c); + + /* A FRESHER survivor (ctrl-2 shape, tip 15 @ term 4) asks for a PreVote at a + * higher prospective term. logOk passes (15 >= 11 at the same term). */ + int rc = raft_on_pre_vote_request(&c->ha.raft, /*from_idx*/1, + /*cand_term*/5, /*cand_last_log_term*/4, + /*cand_last_log_index*/15); + + TEST_ASSERT_EQUAL_INT_MESSAGE(RAFT_GRANT, rc, + "RED: a catching-up survivor with NO live primary must GRANT a " + "sufficiently-fresh PreVote (logOk already enforces Leader Completeness). " + "The unconditional in_catch_up deny (ha.c raft_cb_vote_eligible) refuses " + "it, so two lagging survivors whose primary just died mutually deny and " + "the round-4 election deadlocks (task 5.3 / bug-1272). The fix gates the " + "in_catch_up deny on a LIVE primary, mirroring the heartbeat-fresh deny."); + + hcs4_cluster_down(c); + printf("test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone: done\n"); +} + +/* PIN — while a LIVE primary feeds the catching-up node, the PreVote is still denied. */ +static void test_5_3_catchup_denies_while_live_primary_feeds(void) +{ + cluster_t* c = hcs5_lagging_survivor_up("s53live", 4, 4, 11); + TEST_ASSERT_NOT_NULL(c); + + /* A LIVE primary is actively feeding this catching-up node: fresh heartbeat + * AND a live control fd. The in_catch_up deny MUST still hold — do not disrupt + * active replication with a competing election (the deny's original purpose). */ + c->ha.config.heartbeat_timeout_ms = 6000; + struct timespec now; + clock_gettime(CLOCK_MONOTONIC, &now); + c->ha.last_activity = now.tv_sec; /* fresh heartbeat */ + c->ha.primary_ctrl_fd = 999; /* a live control fd (>=0; never closed here) */ + TEST_ASSERT_TRUE_MESSAGE(ha_heartbeat_is_fresh(&c->ha), + "precondition: the primary reads as alive (fresh heartbeat)"); + + int rc = raft_on_pre_vote_request(&c->ha.raft, 1, 5, 4, 15); + TEST_ASSERT_EQUAL_INT_MESSAGE(RAFT_DENY, rc, + "PIN: while a LIVE primary feeds this catching-up node (fd>=0 + fresh " + "heartbeat) a PreVote must still be DENIED — the fix lifts the in_catch_up " + "deny ONLY once the primary is gone (don't disrupt active replication)"); + + c->ha.primary_ctrl_fd = -1; /* defensive: never let teardown touch fd 999 */ + hcs4_cluster_down(c); + printf("test_5_3_catchup_denies_while_live_primary_feeds: PASS (pin)\n"); +} + +/* PIN — even with the primary gone, a STALE candidate is denied (Leader Completeness). */ +static void test_5_3_stale_candidate_denied_even_with_primary_gone(void) +{ + cluster_t* c = hcs5_lagging_survivor_up("s53stale", 4, 4, 11); + TEST_ASSERT_NOT_NULL(c); + + /* Primary gone (the in_catch_up deny is lifted by the fix), but the candidate + * is STALE — its tip (8) is BELOW the voter's (11). logOk (raft_log_ok, Step 2 + * of raft_on_pre_vote_request) must still DENY: the fix must never let a behind + * candidate win. */ + int rc = raft_on_pre_vote_request(&c->ha.raft, 1, /*cand_term*/5, + /*cand_last_log_term*/4, + /*cand_last_log_index*/8); + TEST_ASSERT_EQUAL_INT_MESSAGE(RAFT_DENY, rc, + "PIN: even with the primary gone, a STALE candidate (tip 8 < voter 11) " + "must be DENIED by logOk — lifting the in_catch_up deny must NOT weaken " + "Leader Completeness"); + + hcs4_cluster_down(c); + printf("test_5_3_stale_candidate_denied_even_with_primary_gone: PASS (pin)\n"); +} + /* ═══════════════════════════════════════════════════════════════════════════ * SCAFFOLDING (PASS) — prove each shared seam is usable by later phases. * ═══════════════════════════════════════════════════════════════════════════ */ @@ -1753,6 +1894,15 @@ void test_ha_core_safety(void) RUN_TEST(test_5_1_spurious_failover_suppressed_while_primary_alive); RUN_TEST(test_5_1_genuine_loss_elects_and_advances_term); RUN_TEST(test_5_1_election_safety_one_primary_per_term); + /* Task 5.3 lagging-but-safe survivor electability under overlapping churn: + * CORE RED — a catching-up survivor with NO live primary must GRANT a + * sufficiently-fresh PreVote (FAILs on baseline: unconditional + * in_catch_up deny deadlocks two lagging survivors — bug-1272). + * PIN — while a LIVE primary feeds it, the PreVote is still denied. + * PIN — a STALE candidate is denied by logOk even with primary gone. */ + RUN_TEST(test_5_3_catchup_survivor_grants_fresh_candidate_when_primary_gone); + RUN_TEST(test_5_3_catchup_denies_while_live_primary_feeds); + RUN_TEST(test_5_3_stale_candidate_denied_even_with_primary_gone); /* Scaffolding: each shared seam is usable by 2.3 / 4.1 / 5.1. */ RUN_TEST(test_seam_wal_construct_and_probe); RUN_TEST(test_seam_divergence_is_observable);