From 806257afcf8b3bb9bdd4bf2cc8802790e0b621a4 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 00:22:33 +0000 Subject: [PATCH] test(raft-ha-core-safety): RED tests for election liveness & safety under churn (task 5.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover root cause (research.md §"Phase 4/5"): - CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a control-channel flap (transient RDHUP) while a current-term primary is alive (fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline (Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup -> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path. GREEN when task 5.2 adds that guard. - LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale heartbeat) still launches a real failover; a failed round re-arms; rapid rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3). - SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest the same term; the real handle_request_vote grants at most one vote per term (A=3/B=0), so at most one committing primary per term. Falsifiable Election Safety check mapped to raft/raft.tla (Req 4.1/4.3). Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC --- .kiro/specs/raft-ha-core-safety/tasks.md | 4 +- .../testsuite/test_ha_core_safety.c | 278 ++++++++++++++++++ 2 files changed, 281 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 36ea2b3c..9ab67bbd 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -97,7 +97,7 @@ - [ ] 5. Election liveness and safety under churn - _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._ -- [ ] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests +- [x] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests - Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`) - Observable completion: the spurious-failover case FAILS on the baseline (unbounded prevote-timeout loop while a live primary exists); the liveness cases pass/fail per the stall; the Election Safety assertion runs and reports per-term primary counts - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_ @@ -147,4 +147,6 @@ - **Task 2.2 — payload repair must be TIP-ONLY (consensus-safety, caught in review round 1).** The `(term,lsn,payload-crc)`-aware guard repairs an empty-over-real entry by truncate-to-`lsn-1`+re-append, but this is safe ONLY when `lsn == current_lsn-1` (the conflict is at the tip with nothing committed above). For a BELOW-tip conflict (`lsn < current_lsn-1` — the actual live LSN-9 case with 10/11/12 above), truncating the tail would drop committed entries and roll `committed_lsn` backward (State Machine Safety violation). Below-tip is handled as a loud non-promotable fence (reuses `recovery_unrecoverable` sticky flag), never mutating any boundary; the real reconvergence comes from the 2.4 streamer fix + re-stream. The tip path snapshots/restores `committed_lsn` under `p->lock` so the boundary is never observably lowered. Any future in-WAL repair must preserve: never drop a committed entry above the conflict, never lower `committed_lsn`. +- **Task 5.1 — RED lands (3 cases in `test_ha_core_safety.c`).** CORE RED `test_5_1_spurious_failover_suppressed_while_primary_alive` FAILs on baseline (`Expected 0 Was 5`): it drives the REAL `ha_on_primary_ctrl_fd_rdhup` (unit-test `c==NULL` path) 5× with a FRESH heartbeat (`ha_heartbeat_is_fresh()==1`) and counts PreVote launches — baseline launches one unwinnable round per flap. **Root-cause asymmetry the fix targets:** the DEFERRED PreVote path (`ha_failover_stagger_fn`, ha.c ~6664) ALREADY guards with `ha_heartbeat_is_fresh` and skips, but the RDHUP-INITIATION path (`ha_on_primary_ctrl_fd_rdhup`→`ha_begin_failover`→`ha_start_prevote_async` synchronous/`c==NULL` branch) has NO such guard. **Task 5.2 GREEN = add the `ha_heartbeat_is_fresh` guard at the RDHUP initiation path** (reconnect+rejoin instead of campaigning) WITHOUT regressing the two PASSing pins: `test_5_1_genuine_loss_elects_and_advances_term` (genuine loss — stale heartbeat — must STILL launch; failed round re-arms; rapid rounds advance term) guards against over-suppression, and `test_5_1_election_safety_one_primary_per_term` (vote-once via the real `handle_request_vote`, A=3/B=0 → ≤1 primary/term) is the falsifiable Election Safety check mapped to raft.tla. Note: `handle_request_vote` already denies same-term challengers when `primary_ctrl_fd>=0 && ha_heartbeat_is_fresh` — the VOTE path is safe; only failover-INITIATION is not. + - **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2. diff --git a/src/controller/testsuite/test_ha_core_safety.c b/src/controller/testsuite/test_ha_core_safety.c index 55a8fb6f..23814e01 100644 --- a/src/controller/testsuite/test_ha_core_safety.c +++ b/src/controller/testsuite/test_ha_core_safety.c @@ -48,6 +48,7 @@ #include #include #include +#include /* clock_gettime/CLOCK_MONOTONIC (task 5.1) */ #include @@ -1330,6 +1331,272 @@ static void test_4_1_apply_gap_blocks_converged_until_applied(void) printf("test_4_1_apply_gap_blocks_converged_until_applied: PASS (pin)\n"); } +/* ═══════════════════════════════════════════════════════════════════════════ + * TASK 5.1 — election liveness & safety under churn (RED + liveness + safety) + * + * Live stall root cause (research.md §"Phase 4/5"): the residual chaotic-soak + * wedge is NOT a missing election-restart (the retry machinery works). It is a + * recovered standby treating a TRANSIENT control-channel RDHUP as "primary + * died" and launching an UNWINNABLE PreVote-failover storm while a current-term + * primary is alive and quorum-reachable (peers deny/ignore, the round times out, + * reconnect, repeat ~15s). The node retries an election it can never win instead + * of rejoining the live primary. + * + * The asymmetry the RED exploits: the DEFERRED PreVote path + * (ha_failover_stagger_fn, ha.c) ALREADY guards with ha_heartbeat_is_fresh and + * skips a PreVote when a primary is alive — but the RDHUP failover-INITIATION + * path (ha_on_primary_ctrl_fd_rdhup → ha_begin_failover → ha_start_prevote_async, + * the synchronous / c==NULL branch) has NO such guard, so every flap launches a + * fresh unwinnable round. handle_request_vote already denies same-term + * challengers while a live primary connection is fresh — the VOTE path is safe; + * only the failover-initiation path is not. + * + * (CORE RED) test_5_1_spurious_failover_suppressed_while_primary_alive: a + * recovered standby whose control channel flaps while a current-term + * primary is alive (fresh heartbeat) must NOT launch a PreVote failover — + * it reconnects + rejoins as follower. FAILS on the baseline (one + * unwinnable PreVote per flap); GREEN when task 5.2 adds the + * ha_heartbeat_is_fresh guard the staggered path already has. + * + * (LIVENESS) test_5_1_genuine_loss_elects_and_advances_term: GENUINE primary + * loss (stale heartbeat) still launches a real failover (the suppression + * must fire ONLY for a live primary, never a dead one); a scripted failed + * round re-arms and another election can start; rapid kill/restart rounds + * each elect at a strictly higher term. Guards 5.2 against OVER-suppression + * reintroducing the round-N stall (Req 6.1/6.2/6.3). + * + * (SAFETY) test_5_1_election_safety_one_primary_per_term: across an + * overlapping-kill window two candidates contest the SAME term; each voter + * grants at most one vote per term (votedFor), so the two can never BOTH + * reach a majority — at most one committing primary per term. Falsifiable + * verification mapped to raft/raft.tla Election Safety (Req 4.1/4.3): it + * runs and reports the per-term primary count; a failure would disprove the + * single-leader-per-term assumption and promote a concrete adapter guard to + * required work. + * + * Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3. + * ═══════════════════════════════════════════════════════════════════════════ */ + +/* XCE_TESTING PreVote orchestration seams declared in ha.c (same externs the + * test_ha_prevote.c suite uses). ha_on_primary_ctrl_fd_rdhup, ha_do_demote, + * handle_request_vote and ha_heartbeat_is_fresh are public (ha.h). */ +extern void ha_start_prevote_async(ha_state_t* ha, cluster_t* c); +extern void pre_vote_reply_dispatch_for_test(ha_state_t* ha, cluster_t* c, + int peer_idx, int granted); +extern int (*ha_prevote_rearm_liveness_hook)(ha_state_t* ha, cluster_t* c); + +/* Make ha->raft a 3-node standby (self=0) so a PreVote tally needs a peer grant + * to win (quorum=2; self-grant=1) — mirrors make_three_node in test_ha_prevote.c. + * ha_init seeds only a 1-node placeholder, so widen the topology directly. */ +static void hcs5_make_standby(ha_state_t* ha) +{ + TEST_ASSERT_EQUAL_INT(0, ha_init(ha, &g_hcs_cfg, 0)); + ha_do_demote(ha); /* role = CTRL_STANDBY */ + ha->raft.n_peers = 3; + ha->raft.self_idx = 0; + ha->raft.role = RAFT_FOLLOWER; + ha->n_peers = 3; +} + +/* Mark the primary as alive NOW (a heartbeat just arrived) so + * ha_heartbeat_is_fresh()==1 — the "current-term primary is alive and + * quorum-reachable" condition the spurious-failover suppression must honor. */ +static void hcs5_mark_primary_fresh(ha_state_t* ha) +{ + struct timespec now; + clock_gettime(CLOCK_MONOTONIC, &now); + ha->last_activity = now.tv_sec; /* elapsed ~0ms < heartbeat_timeout_ms */ +} + +/* Mark the primary as genuinely gone: the last heartbeat is older than the + * timeout (NOT a transient flap) so ha_heartbeat_is_fresh()==0. */ +static void hcs5_mark_primary_dead(ha_state_t* ha) +{ + struct timespec now; + clock_gettime(CLOCK_MONOTONIC, &now); + ha->last_activity = + now.tv_sec - (time_t)(g_hcs_cfg.heartbeat_timeout_ms / 1000) - 5; +} + +/* Re-arm counter for the LOST-round liveness assertion (Req 6.3). */ +static int g_hcs5_rearm_calls; +static int hcs5_stub_rearm(ha_state_t* ha, cluster_t* c) +{ + (void)ha; + (void)c; + g_hcs5_rearm_calls++; + return 0; +} + +/* CORE RED — a flap while a live current-term primary exists must NOT fail over. */ +static void test_5_1_spurious_failover_suppressed_while_primary_alive(void) +{ + ha_state_t ha; + hcs5_make_standby(&ha); + + enum { CURRENT_TERM = 6, FLAPS = 5 }; + atomic_store_explicit(&ha.term, (uint64_t)CURRENT_TERM, memory_order_release); + + /* The primary is alive at the current term: a heartbeat just arrived. */ + hcs5_mark_primary_fresh(&ha); + TEST_ASSERT_TRUE_MESSAGE(ha_heartbeat_is_fresh(&ha), + "precondition: the primary must read as alive (fresh heartbeat) — this is " + "the same predicate the vote-deny gate and the staggered-failover guard " + "already consult"); + + uint64_t term_before = atomic_load_explicit(&ha.term, memory_order_acquire); + + /* The control channel flaps repeatedly (transient RDHUP). Each flap must NOT + * launch an unwinnable PreVote failover while the primary is alive — the node + * should reconnect + rejoin as follower. Count the launches per flap. */ + int launches = 0; + for (int flap = 0; flap < FLAPS; flap++) { + ha.raft.pre_vote_in_progress = 0; /* observe THIS flap's decision */ + hcs5_mark_primary_fresh(&ha); /* a heartbeat precedes each live flap */ + ha_on_primary_ctrl_fd_rdhup(&ha, NULL); /* unit-test-mode failover entry */ + if (ha.raft.pre_vote_in_progress) + launches++; + } + + /* CORE RED: on the baseline every RDHUP unconditionally reaches + * ha_begin_failover → ha_start_prevote_async (the synchronous / c==NULL path + * has NO ha_heartbeat_is_fresh guard, unlike ha_failover_stagger_fn), so + * launches == FLAPS — an unbounded PreVote storm while a live primary exists. + * Task 5.2 must add the freshness guard at the RDHUP initiation path so a + * flap reconnects instead of campaigning. */ + TEST_ASSERT_EQUAL_INT_MESSAGE(0, launches, + "RED: a control-channel flap while a current-term primary is alive must " + "NOT launch a PreVote failover — the node must reconnect + rejoin. The " + "baseline launches one unwinnable PreVote per flap (the spurious-failover " + "storm); task 5.2 adds the ha_heartbeat_is_fresh guard the staggered path " + "already has."); + + /* The node never campaigned: still CTRL_STANDBY at the unchanged term, no + * round left in progress. */ + TEST_ASSERT_EQUAL_INT_MESSAGE(CTRL_STANDBY, + (int)atomic_load_explicit(&ha.role, memory_order_acquire), + "a suppressed flap leaves the node CTRL_STANDBY (rejoins the live primary)"); + TEST_ASSERT_EQUAL_UINT64_MESSAGE(term_before, + atomic_load_explicit(&ha.term, memory_order_acquire), + "a suppressed flap must not bump the term"); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, ha.raft.pre_vote_in_progress, + "no PreVote round may be left in progress after the flaps"); + + ha_fini(&ha); + printf("test_5_1_spurious_failover_suppressed_while_primary_alive: done " + "(launches=%d/%d)\n", launches, (int)FLAPS); +} + +/* LIVENESS — genuine loss still elects; failed round re-arms; term advances. */ +static void test_5_1_genuine_loss_elects_and_advances_term(void) +{ + ha_state_t ha; + hcs5_make_standby(&ha); + + enum { START_TERM = 4, ROUNDS = 4 }; + atomic_store_explicit(&ha.term, (uint64_t)START_TERM, memory_order_release); + + /* (a) GENUINE primary loss (stale heartbeat, NOT a transient flap) MUST still + * launch a real failover — the suppression of task 5.2 fires ONLY for a live + * primary; over-suppressing a dead one would re-introduce the round-N stall. */ + hcs5_mark_primary_dead(&ha); + TEST_ASSERT_FALSE_MESSAGE(ha_heartbeat_is_fresh(&ha), + "precondition: genuine loss — the heartbeat is stale (primary gone)"); + ha.raft.pre_vote_in_progress = 0; + ha_on_primary_ctrl_fd_rdhup(&ha, NULL); + TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress, + "Req 6.1: genuine primary loss must launch a real PreVote failover — the " + "suppression must NEVER fire for a dead primary (no over-suppression)"); + + /* (b) a scripted FAILED round (both peers deny) re-arms liveness, and a later + * genuine failure can start a fresh round (Req 6.3 — never park without a + * primary while quorum is reachable). */ + g_hcs5_rearm_calls = 0; + ha_prevote_rearm_liveness_hook = hcs5_stub_rearm; + pre_vote_reply_dispatch_for_test(&ha, NULL, 1, 0 /*deny*/); + pre_vote_reply_dispatch_for_test(&ha, NULL, 2, 0 /*deny*/); + TEST_ASSERT_GREATER_OR_EQUAL_INT_MESSAGE(1, g_hcs5_rearm_calls, + "Req 6.3: a failed round must re-arm liveness so another election can start"); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, ha.raft.pre_vote_in_progress, + "the lost round is over (in-progress cleared)"); + ha_start_prevote_async(&ha, NULL); /* a later genuine failure → fresh round */ + TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress, + "Req 6.3: a fresh round arms after an earlier failed one (failover is not " + "wedged)"); + ha_prevote_rearm_liveness_hook = NULL; + + /* (c) liveness under churn: each completed kill/restart round elects at a + * STRICTLY HIGHER term and a primary emerges — modeled deterministically via + * the election-injection seam (a chaotic round, no sleeps). */ + uint64_t prev = atomic_load_explicit(&ha.term, memory_order_acquire); + for (int round = 1; round <= ROUNDS; round++) { + uint64_t t = hcs_inject_election(&ha, prev + 1, (int32_t)CTRL_PRIMARY); + TEST_ASSERT_EQUAL_UINT64_MESSAGE(prev + 1, t, + "Req 6.1/6.2: each rapid kill/restart round must elect at a STRICTLY " + "higher term (no permanent stall under churn)"); + TEST_ASSERT_EQUAL_INT32_MESSAGE((int32_t)CTRL_PRIMARY, + (int32_t)atomic_load_explicit(&ha.role, memory_order_acquire), + "Req 6.2: a primary must emerge each round (liveness)"); + prev = t; + } + + ha_fini(&ha); + printf("test_5_1_genuine_loss_elects_and_advances_term: done (final_term=%" + PRIu64 ")\n", prev); +} + +/* SAFETY — Election Safety: at most one committing primary per term. */ +static void test_5_1_election_safety_one_primary_per_term(void) +{ + enum { N_VOTERS = 3, TERM = 7, MAJORITY = 2 }; + + /* Three independent voters, each a fresh standby with NO live primary (so the + * heartbeat-fresh deny gate does not pre-empt the vote-once logic under test) + * and an empty log (so the freshness floor never denies on staleness). */ + ha_state_t v[N_VOTERS]; + for (int i = 0; i < N_VOTERS; i++) { + hcs5_make_standby(&v[i]); + atomic_store_explicit(&v[i].term, (uint64_t)(TERM - 1), + memory_order_release); + v[i].primary_ctrl_fd = -1; /* no live primary connection */ + v[i].last_activity = 0; /* not fresh → no heartbeat-deny pre-empt */ + } + + /* An overlapping-kill window: two candidates A and B contest the SAME term T. + * Ask EACH voter for A first, then B, in the same term. */ + int grants_a = 0, grants_b = 0; + for (int i = 0; i < N_VOTERS; i++) { + grants_a += handle_request_vote(&v[i], (uint64_t)TERM, "ctrl-A", + /*config_tag*/1, /*last_lsn*/0, + /*my_config_tag*/1, /*my_last_lsn*/0, + /*in_catch_up*/0); + grants_b += handle_request_vote(&v[i], (uint64_t)TERM, "ctrl-B", + /*config_tag*/1, /*last_lsn*/0, + /*my_config_tag*/1, /*my_last_lsn*/0, + /*in_catch_up*/0); + } + + int a_majority = grants_a >= MAJORITY; + int b_majority = grants_b >= MAJORITY; + int primaries_this_term = (a_majority ? 1 : 0) + (b_majority ? 1 : 0); + printf("test_5_1 election-safety term=%d: grants A=%d B=%d (majority=%d) → " + "primaries_this_term=%d\n", (int)TERM, grants_a, grants_b, + (int)MAJORITY, primaries_this_term); + + /* ELECTION SAFETY (raft/raft.tla): a voter that granted A in term T must DENY + * B in term T (vote-once), so the two candidates can never BOTH reach a + * majority — at most one committing primary per term. */ + TEST_ASSERT_FALSE_MESSAGE(a_majority && b_majority, + "Election Safety (raft.tla): two candidates must NOT both reach a majority " + "in the same term — each voter grants at most one vote per term (votedFor)"); + TEST_ASSERT_LESS_OR_EQUAL_INT_MESSAGE(1, primaries_this_term, + "the per-term committing-primary count must be <= 1 (Req 4.1)"); + + for (int i = 0; i < N_VOTERS; i++) + ha_fini(&v[i]); + printf("test_5_1_election_safety_one_primary_per_term: done\n"); +} + /* ═══════════════════════════════════════════════════════════════════════════ * SCAFFOLDING (PASS) — prove each shared seam is usable by later phases. * ═══════════════════════════════════════════════════════════════════════════ */ @@ -1475,6 +1742,17 @@ void test_ha_core_safety(void) * catches the committed boundary. */ RUN_TEST(test_4_1_follower_reconverges_multientry_divergent_tail); RUN_TEST(test_4_1_apply_gap_blocks_converged_until_applied); + /* Task 5.1 election liveness & safety under churn: + * CORE RED — a control-channel flap while a current-term primary is alive + * must NOT launch a PreVote failover (FAILs on the baseline: + * the RDHUP path has no ha_heartbeat_is_fresh guard). + * LIVENESS — genuine loss still elects; a failed round re-arms; rapid + * rounds advance the term (guards 5.2 vs over-suppression). + * SAFETY — at most one committing primary per term (Election Safety, + * falsifiable, mapped to raft/raft.tla; reports per-term count). */ + RUN_TEST(test_5_1_spurious_failover_suppressed_while_primary_alive); + RUN_TEST(test_5_1_genuine_loss_elects_and_advances_term); + RUN_TEST(test_5_1_election_safety_one_primary_per_term); /* Scaffolding: each shared seam is usable by 2.3 / 4.1 / 5.1. */ RUN_TEST(test_seam_wal_construct_and_probe); RUN_TEST(test_seam_divergence_is_observable);