forked from lei.chen/myxce
test(raft-ha-core-safety): RED tests for election liveness & safety under churn (task 5.1)
Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover root cause (research.md §"Phase 4/5"): - CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a control-channel flap (transient RDHUP) while a current-term primary is alive (fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline (Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup -> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path. GREEN when task 5.2 adds that guard. - LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale heartbeat) still launches a real failover; a failed round re-arms; rapid rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3). - SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest the same term; the real handle_request_vote grants at most one vote per term (A=3/B=0), so at most one committing primary per term. Falsifiable Election Safety check mapped to raft/raft.tla (Req 4.1/4.3). Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
parent
7507a113f8
commit
806257afcf
2 changed files with 281 additions and 1 deletions
|
|
@ -97,7 +97,7 @@
|
||||||
|
|
||||||
- [ ] 5. Election liveness and safety under churn
|
- [ ] 5. Election liveness and safety under churn
|
||||||
- _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._
|
- _Root cause (Phase-4/5 investigation, `research.md`): the live stall is NOT a missing election-restart (retry works) — it is a recovered standby treating a TRANSIENT control-channel RDHUP as "primary died" and launching an UNWINNABLE PreVote-failover storm while a current-term primary is alive (peers deny/ignore, round times out, reconnect, repeat ~15s). The node retries an election it can never win instead of rejoining the live primary._
|
||||||
- [ ] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests
|
- [x] 5.1 RED: spurious-failover suppression + election liveness + Election Safety tests
|
||||||
- Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`)
|
- Primary case: a recovered standby whose primary control channel flaps (transient RDHUP) while a current-term primary is alive and quorum-reachable must NOT enter an unbounded `ha_begin_failover`/`PreVote timed out` loop — it reconnects and rejoins as follower. Liveness case: on GENUINE primary loss, repeated rapid kill/restart rounds yield a higher-term primary each round; a scripted failed round is followed by another election. Safety case: across overlapping kills, at most one member ever reaches committing-primary per term (Election Safety, mapped to `raft/raft.tla`)
|
||||||
- Observable completion: the spurious-failover case FAILS on the baseline (unbounded prevote-timeout loop while a live primary exists); the liveness cases pass/fail per the stall; the Election Safety assertion runs and reports per-term primary counts
|
- Observable completion: the spurious-failover case FAILS on the baseline (unbounded prevote-timeout loop while a live primary exists); the liveness cases pass/fail per the stall; the Election Safety assertion runs and reports per-term primary counts
|
||||||
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_
|
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3_
|
||||||
|
|
@ -147,4 +147,6 @@
|
||||||
|
|
||||||
- **Task 2.2 — payload repair must be TIP-ONLY (consensus-safety, caught in review round 1).** The `(term,lsn,payload-crc)`-aware guard repairs an empty-over-real entry by truncate-to-`lsn-1`+re-append, but this is safe ONLY when `lsn == current_lsn-1` (the conflict is at the tip with nothing committed above). For a BELOW-tip conflict (`lsn < current_lsn-1` — the actual live LSN-9 case with 10/11/12 above), truncating the tail would drop committed entries and roll `committed_lsn` backward (State Machine Safety violation). Below-tip is handled as a loud non-promotable fence (reuses `recovery_unrecoverable` sticky flag), never mutating any boundary; the real reconvergence comes from the 2.4 streamer fix + re-stream. The tip path snapshots/restores `committed_lsn` under `p->lock` so the boundary is never observably lowered. Any future in-WAL repair must preserve: never drop a committed entry above the conflict, never lower `committed_lsn`.
|
- **Task 2.2 — payload repair must be TIP-ONLY (consensus-safety, caught in review round 1).** The `(term,lsn,payload-crc)`-aware guard repairs an empty-over-real entry by truncate-to-`lsn-1`+re-append, but this is safe ONLY when `lsn == current_lsn-1` (the conflict is at the tip with nothing committed above). For a BELOW-tip conflict (`lsn < current_lsn-1` — the actual live LSN-9 case with 10/11/12 above), truncating the tail would drop committed entries and roll `committed_lsn` backward (State Machine Safety violation). Below-tip is handled as a loud non-promotable fence (reuses `recovery_unrecoverable` sticky flag), never mutating any boundary; the real reconvergence comes from the 2.4 streamer fix + re-stream. The tip path snapshots/restores `committed_lsn` under `p->lock` so the boundary is never observably lowered. Any future in-WAL repair must preserve: never drop a committed entry above the conflict, never lower `committed_lsn`.
|
||||||
|
|
||||||
|
- **Task 5.1 — RED lands (3 cases in `test_ha_core_safety.c`).** CORE RED `test_5_1_spurious_failover_suppressed_while_primary_alive` FAILs on baseline (`Expected 0 Was 5`): it drives the REAL `ha_on_primary_ctrl_fd_rdhup` (unit-test `c==NULL` path) 5× with a FRESH heartbeat (`ha_heartbeat_is_fresh()==1`) and counts PreVote launches — baseline launches one unwinnable round per flap. **Root-cause asymmetry the fix targets:** the DEFERRED PreVote path (`ha_failover_stagger_fn`, ha.c ~6664) ALREADY guards with `ha_heartbeat_is_fresh` and skips, but the RDHUP-INITIATION path (`ha_on_primary_ctrl_fd_rdhup`→`ha_begin_failover`→`ha_start_prevote_async` synchronous/`c==NULL` branch) has NO such guard. **Task 5.2 GREEN = add the `ha_heartbeat_is_fresh` guard at the RDHUP initiation path** (reconnect+rejoin instead of campaigning) WITHOUT regressing the two PASSing pins: `test_5_1_genuine_loss_elects_and_advances_term` (genuine loss — stale heartbeat — must STILL launch; failed round re-arms; rapid rounds advance term) guards against over-suppression, and `test_5_1_election_safety_one_primary_per_term` (vote-once via the real `handle_request_vote`, A=3/B=0 → ≤1 primary/term) is the falsifiable Election Safety check mapped to raft.tla. Note: `handle_request_vote` already denies same-term challengers when `primary_ctrl_fd>=0 && ha_heartbeat_is_fresh` — the VOTE path is safe; only failover-INITIATION is not.
|
||||||
|
|
||||||
- **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2.
|
- **Task 2.4 — root-cause source fix + a deferral.** `ha_stream_wal_to_standby` now reads the entry body via `stream_read_entry_body` (bounded retry, max 4 no-progress attempts, partial-read-accumulating, no sleep); on persistent short `pread`/`malloc`-fail it emits a loud ERROR and ABORTS the stream (flushes the valid contiguous prefix, drops the unreadable entry, returns `-EIO`) instead of shipping `PLEN=0`. The incomplete stream is re-driven by the ACK-driven catch-up continuation. Combined with the 2.2 self-healing guard, this closes the live cross-term payload-loss divergence. **Deferred:** the structural retirement of the term-blind `ha_apply_single_entry` bypass — its empty-over-full safety is enforced by the 2.2 guard + pinned by two passing tests, so it cannot corrupt committed state; the structural removal is tracked to task 6.2. Live ≥6-round soak proof is task 7.2.
|
||||||
|
|
|
||||||
|
|
@ -48,6 +48,7 @@
|
||||||
#include <stdio.h>
|
#include <stdio.h>
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <string.h>
|
#include <string.h>
|
||||||
|
#include <time.h> /* clock_gettime/CLOCK_MONOTONIC (task 5.1) */
|
||||||
|
|
||||||
#include <unity/unity.h>
|
#include <unity/unity.h>
|
||||||
|
|
||||||
|
|
@ -1330,6 +1331,272 @@ static void test_4_1_apply_gap_blocks_converged_until_applied(void)
|
||||||
printf("test_4_1_apply_gap_blocks_converged_until_applied: PASS (pin)\n");
|
printf("test_4_1_apply_gap_blocks_converged_until_applied: PASS (pin)\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ═══════════════════════════════════════════════════════════════════════════
|
||||||
|
* TASK 5.1 — election liveness & safety under churn (RED + liveness + safety)
|
||||||
|
*
|
||||||
|
* Live stall root cause (research.md §"Phase 4/5"): the residual chaotic-soak
|
||||||
|
* wedge is NOT a missing election-restart (the retry machinery works). It is a
|
||||||
|
* recovered standby treating a TRANSIENT control-channel RDHUP as "primary
|
||||||
|
* died" and launching an UNWINNABLE PreVote-failover storm while a current-term
|
||||||
|
* primary is alive and quorum-reachable (peers deny/ignore, the round times out,
|
||||||
|
* reconnect, repeat ~15s). The node retries an election it can never win instead
|
||||||
|
* of rejoining the live primary.
|
||||||
|
*
|
||||||
|
* The asymmetry the RED exploits: the DEFERRED PreVote path
|
||||||
|
* (ha_failover_stagger_fn, ha.c) ALREADY guards with ha_heartbeat_is_fresh and
|
||||||
|
* skips a PreVote when a primary is alive — but the RDHUP failover-INITIATION
|
||||||
|
* path (ha_on_primary_ctrl_fd_rdhup → ha_begin_failover → ha_start_prevote_async,
|
||||||
|
* the synchronous / c==NULL branch) has NO such guard, so every flap launches a
|
||||||
|
* fresh unwinnable round. handle_request_vote already denies same-term
|
||||||
|
* challengers while a live primary connection is fresh — the VOTE path is safe;
|
||||||
|
* only the failover-initiation path is not.
|
||||||
|
*
|
||||||
|
* (CORE RED) test_5_1_spurious_failover_suppressed_while_primary_alive: a
|
||||||
|
* recovered standby whose control channel flaps while a current-term
|
||||||
|
* primary is alive (fresh heartbeat) must NOT launch a PreVote failover —
|
||||||
|
* it reconnects + rejoins as follower. FAILS on the baseline (one
|
||||||
|
* unwinnable PreVote per flap); GREEN when task 5.2 adds the
|
||||||
|
* ha_heartbeat_is_fresh guard the staggered path already has.
|
||||||
|
*
|
||||||
|
* (LIVENESS) test_5_1_genuine_loss_elects_and_advances_term: GENUINE primary
|
||||||
|
* loss (stale heartbeat) still launches a real failover (the suppression
|
||||||
|
* must fire ONLY for a live primary, never a dead one); a scripted failed
|
||||||
|
* round re-arms and another election can start; rapid kill/restart rounds
|
||||||
|
* each elect at a strictly higher term. Guards 5.2 against OVER-suppression
|
||||||
|
* reintroducing the round-N stall (Req 6.1/6.2/6.3).
|
||||||
|
*
|
||||||
|
* (SAFETY) test_5_1_election_safety_one_primary_per_term: across an
|
||||||
|
* overlapping-kill window two candidates contest the SAME term; each voter
|
||||||
|
* grants at most one vote per term (votedFor), so the two can never BOTH
|
||||||
|
* reach a majority — at most one committing primary per term. Falsifiable
|
||||||
|
* verification mapped to raft/raft.tla Election Safety (Req 4.1/4.3): it
|
||||||
|
* runs and reports the per-term primary count; a failure would disprove the
|
||||||
|
* single-leader-per-term assumption and promote a concrete adapter guard to
|
||||||
|
* required work.
|
||||||
|
*
|
||||||
|
* Requirements: 4.1, 4.3, 6.1, 6.2, 6.3, 7.3.
|
||||||
|
* ═══════════════════════════════════════════════════════════════════════════ */
|
||||||
|
|
||||||
|
/* XCE_TESTING PreVote orchestration seams declared in ha.c (same externs the
|
||||||
|
* test_ha_prevote.c suite uses). ha_on_primary_ctrl_fd_rdhup, ha_do_demote,
|
||||||
|
* handle_request_vote and ha_heartbeat_is_fresh are public (ha.h). */
|
||||||
|
extern void ha_start_prevote_async(ha_state_t* ha, cluster_t* c);
|
||||||
|
extern void pre_vote_reply_dispatch_for_test(ha_state_t* ha, cluster_t* c,
|
||||||
|
int peer_idx, int granted);
|
||||||
|
extern int (*ha_prevote_rearm_liveness_hook)(ha_state_t* ha, cluster_t* c);
|
||||||
|
|
||||||
|
/* Make ha->raft a 3-node standby (self=0) so a PreVote tally needs a peer grant
|
||||||
|
* to win (quorum=2; self-grant=1) — mirrors make_three_node in test_ha_prevote.c.
|
||||||
|
* ha_init seeds only a 1-node placeholder, so widen the topology directly. */
|
||||||
|
static void hcs5_make_standby(ha_state_t* ha)
|
||||||
|
{
|
||||||
|
TEST_ASSERT_EQUAL_INT(0, ha_init(ha, &g_hcs_cfg, 0));
|
||||||
|
ha_do_demote(ha); /* role = CTRL_STANDBY */
|
||||||
|
ha->raft.n_peers = 3;
|
||||||
|
ha->raft.self_idx = 0;
|
||||||
|
ha->raft.role = RAFT_FOLLOWER;
|
||||||
|
ha->n_peers = 3;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Mark the primary as alive NOW (a heartbeat just arrived) so
|
||||||
|
* ha_heartbeat_is_fresh()==1 — the "current-term primary is alive and
|
||||||
|
* quorum-reachable" condition the spurious-failover suppression must honor. */
|
||||||
|
static void hcs5_mark_primary_fresh(ha_state_t* ha)
|
||||||
|
{
|
||||||
|
struct timespec now;
|
||||||
|
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||||
|
ha->last_activity = now.tv_sec; /* elapsed ~0ms < heartbeat_timeout_ms */
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Mark the primary as genuinely gone: the last heartbeat is older than the
|
||||||
|
* timeout (NOT a transient flap) so ha_heartbeat_is_fresh()==0. */
|
||||||
|
static void hcs5_mark_primary_dead(ha_state_t* ha)
|
||||||
|
{
|
||||||
|
struct timespec now;
|
||||||
|
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||||
|
ha->last_activity =
|
||||||
|
now.tv_sec - (time_t)(g_hcs_cfg.heartbeat_timeout_ms / 1000) - 5;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Re-arm counter for the LOST-round liveness assertion (Req 6.3). */
|
||||||
|
static int g_hcs5_rearm_calls;
|
||||||
|
static int hcs5_stub_rearm(ha_state_t* ha, cluster_t* c)
|
||||||
|
{
|
||||||
|
(void)ha;
|
||||||
|
(void)c;
|
||||||
|
g_hcs5_rearm_calls++;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* CORE RED — a flap while a live current-term primary exists must NOT fail over. */
|
||||||
|
static void test_5_1_spurious_failover_suppressed_while_primary_alive(void)
|
||||||
|
{
|
||||||
|
ha_state_t ha;
|
||||||
|
hcs5_make_standby(&ha);
|
||||||
|
|
||||||
|
enum { CURRENT_TERM = 6, FLAPS = 5 };
|
||||||
|
atomic_store_explicit(&ha.term, (uint64_t)CURRENT_TERM, memory_order_release);
|
||||||
|
|
||||||
|
/* The primary is alive at the current term: a heartbeat just arrived. */
|
||||||
|
hcs5_mark_primary_fresh(&ha);
|
||||||
|
TEST_ASSERT_TRUE_MESSAGE(ha_heartbeat_is_fresh(&ha),
|
||||||
|
"precondition: the primary must read as alive (fresh heartbeat) — this is "
|
||||||
|
"the same predicate the vote-deny gate and the staggered-failover guard "
|
||||||
|
"already consult");
|
||||||
|
|
||||||
|
uint64_t term_before = atomic_load_explicit(&ha.term, memory_order_acquire);
|
||||||
|
|
||||||
|
/* The control channel flaps repeatedly (transient RDHUP). Each flap must NOT
|
||||||
|
* launch an unwinnable PreVote failover while the primary is alive — the node
|
||||||
|
* should reconnect + rejoin as follower. Count the launches per flap. */
|
||||||
|
int launches = 0;
|
||||||
|
for (int flap = 0; flap < FLAPS; flap++) {
|
||||||
|
ha.raft.pre_vote_in_progress = 0; /* observe THIS flap's decision */
|
||||||
|
hcs5_mark_primary_fresh(&ha); /* a heartbeat precedes each live flap */
|
||||||
|
ha_on_primary_ctrl_fd_rdhup(&ha, NULL); /* unit-test-mode failover entry */
|
||||||
|
if (ha.raft.pre_vote_in_progress)
|
||||||
|
launches++;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* CORE RED: on the baseline every RDHUP unconditionally reaches
|
||||||
|
* ha_begin_failover → ha_start_prevote_async (the synchronous / c==NULL path
|
||||||
|
* has NO ha_heartbeat_is_fresh guard, unlike ha_failover_stagger_fn), so
|
||||||
|
* launches == FLAPS — an unbounded PreVote storm while a live primary exists.
|
||||||
|
* Task 5.2 must add the freshness guard at the RDHUP initiation path so a
|
||||||
|
* flap reconnects instead of campaigning. */
|
||||||
|
TEST_ASSERT_EQUAL_INT_MESSAGE(0, launches,
|
||||||
|
"RED: a control-channel flap while a current-term primary is alive must "
|
||||||
|
"NOT launch a PreVote failover — the node must reconnect + rejoin. The "
|
||||||
|
"baseline launches one unwinnable PreVote per flap (the spurious-failover "
|
||||||
|
"storm); task 5.2 adds the ha_heartbeat_is_fresh guard the staggered path "
|
||||||
|
"already has.");
|
||||||
|
|
||||||
|
/* The node never campaigned: still CTRL_STANDBY at the unchanged term, no
|
||||||
|
* round left in progress. */
|
||||||
|
TEST_ASSERT_EQUAL_INT_MESSAGE(CTRL_STANDBY,
|
||||||
|
(int)atomic_load_explicit(&ha.role, memory_order_acquire),
|
||||||
|
"a suppressed flap leaves the node CTRL_STANDBY (rejoins the live primary)");
|
||||||
|
TEST_ASSERT_EQUAL_UINT64_MESSAGE(term_before,
|
||||||
|
atomic_load_explicit(&ha.term, memory_order_acquire),
|
||||||
|
"a suppressed flap must not bump the term");
|
||||||
|
TEST_ASSERT_EQUAL_INT_MESSAGE(0, ha.raft.pre_vote_in_progress,
|
||||||
|
"no PreVote round may be left in progress after the flaps");
|
||||||
|
|
||||||
|
ha_fini(&ha);
|
||||||
|
printf("test_5_1_spurious_failover_suppressed_while_primary_alive: done "
|
||||||
|
"(launches=%d/%d)\n", launches, (int)FLAPS);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* LIVENESS — genuine loss still elects; failed round re-arms; term advances. */
|
||||||
|
static void test_5_1_genuine_loss_elects_and_advances_term(void)
|
||||||
|
{
|
||||||
|
ha_state_t ha;
|
||||||
|
hcs5_make_standby(&ha);
|
||||||
|
|
||||||
|
enum { START_TERM = 4, ROUNDS = 4 };
|
||||||
|
atomic_store_explicit(&ha.term, (uint64_t)START_TERM, memory_order_release);
|
||||||
|
|
||||||
|
/* (a) GENUINE primary loss (stale heartbeat, NOT a transient flap) MUST still
|
||||||
|
* launch a real failover — the suppression of task 5.2 fires ONLY for a live
|
||||||
|
* primary; over-suppressing a dead one would re-introduce the round-N stall. */
|
||||||
|
hcs5_mark_primary_dead(&ha);
|
||||||
|
TEST_ASSERT_FALSE_MESSAGE(ha_heartbeat_is_fresh(&ha),
|
||||||
|
"precondition: genuine loss — the heartbeat is stale (primary gone)");
|
||||||
|
ha.raft.pre_vote_in_progress = 0;
|
||||||
|
ha_on_primary_ctrl_fd_rdhup(&ha, NULL);
|
||||||
|
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress,
|
||||||
|
"Req 6.1: genuine primary loss must launch a real PreVote failover — the "
|
||||||
|
"suppression must NEVER fire for a dead primary (no over-suppression)");
|
||||||
|
|
||||||
|
/* (b) a scripted FAILED round (both peers deny) re-arms liveness, and a later
|
||||||
|
* genuine failure can start a fresh round (Req 6.3 — never park without a
|
||||||
|
* primary while quorum is reachable). */
|
||||||
|
g_hcs5_rearm_calls = 0;
|
||||||
|
ha_prevote_rearm_liveness_hook = hcs5_stub_rearm;
|
||||||
|
pre_vote_reply_dispatch_for_test(&ha, NULL, 1, 0 /*deny*/);
|
||||||
|
pre_vote_reply_dispatch_for_test(&ha, NULL, 2, 0 /*deny*/);
|
||||||
|
TEST_ASSERT_GREATER_OR_EQUAL_INT_MESSAGE(1, g_hcs5_rearm_calls,
|
||||||
|
"Req 6.3: a failed round must re-arm liveness so another election can start");
|
||||||
|
TEST_ASSERT_EQUAL_INT_MESSAGE(0, ha.raft.pre_vote_in_progress,
|
||||||
|
"the lost round is over (in-progress cleared)");
|
||||||
|
ha_start_prevote_async(&ha, NULL); /* a later genuine failure → fresh round */
|
||||||
|
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress,
|
||||||
|
"Req 6.3: a fresh round arms after an earlier failed one (failover is not "
|
||||||
|
"wedged)");
|
||||||
|
ha_prevote_rearm_liveness_hook = NULL;
|
||||||
|
|
||||||
|
/* (c) liveness under churn: each completed kill/restart round elects at a
|
||||||
|
* STRICTLY HIGHER term and a primary emerges — modeled deterministically via
|
||||||
|
* the election-injection seam (a chaotic round, no sleeps). */
|
||||||
|
uint64_t prev = atomic_load_explicit(&ha.term, memory_order_acquire);
|
||||||
|
for (int round = 1; round <= ROUNDS; round++) {
|
||||||
|
uint64_t t = hcs_inject_election(&ha, prev + 1, (int32_t)CTRL_PRIMARY);
|
||||||
|
TEST_ASSERT_EQUAL_UINT64_MESSAGE(prev + 1, t,
|
||||||
|
"Req 6.1/6.2: each rapid kill/restart round must elect at a STRICTLY "
|
||||||
|
"higher term (no permanent stall under churn)");
|
||||||
|
TEST_ASSERT_EQUAL_INT32_MESSAGE((int32_t)CTRL_PRIMARY,
|
||||||
|
(int32_t)atomic_load_explicit(&ha.role, memory_order_acquire),
|
||||||
|
"Req 6.2: a primary must emerge each round (liveness)");
|
||||||
|
prev = t;
|
||||||
|
}
|
||||||
|
|
||||||
|
ha_fini(&ha);
|
||||||
|
printf("test_5_1_genuine_loss_elects_and_advances_term: done (final_term=%"
|
||||||
|
PRIu64 ")\n", prev);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* SAFETY — Election Safety: at most one committing primary per term. */
|
||||||
|
static void test_5_1_election_safety_one_primary_per_term(void)
|
||||||
|
{
|
||||||
|
enum { N_VOTERS = 3, TERM = 7, MAJORITY = 2 };
|
||||||
|
|
||||||
|
/* Three independent voters, each a fresh standby with NO live primary (so the
|
||||||
|
* heartbeat-fresh deny gate does not pre-empt the vote-once logic under test)
|
||||||
|
* and an empty log (so the freshness floor never denies on staleness). */
|
||||||
|
ha_state_t v[N_VOTERS];
|
||||||
|
for (int i = 0; i < N_VOTERS; i++) {
|
||||||
|
hcs5_make_standby(&v[i]);
|
||||||
|
atomic_store_explicit(&v[i].term, (uint64_t)(TERM - 1),
|
||||||
|
memory_order_release);
|
||||||
|
v[i].primary_ctrl_fd = -1; /* no live primary connection */
|
||||||
|
v[i].last_activity = 0; /* not fresh → no heartbeat-deny pre-empt */
|
||||||
|
}
|
||||||
|
|
||||||
|
/* An overlapping-kill window: two candidates A and B contest the SAME term T.
|
||||||
|
* Ask EACH voter for A first, then B, in the same term. */
|
||||||
|
int grants_a = 0, grants_b = 0;
|
||||||
|
for (int i = 0; i < N_VOTERS; i++) {
|
||||||
|
grants_a += handle_request_vote(&v[i], (uint64_t)TERM, "ctrl-A",
|
||||||
|
/*config_tag*/1, /*last_lsn*/0,
|
||||||
|
/*my_config_tag*/1, /*my_last_lsn*/0,
|
||||||
|
/*in_catch_up*/0);
|
||||||
|
grants_b += handle_request_vote(&v[i], (uint64_t)TERM, "ctrl-B",
|
||||||
|
/*config_tag*/1, /*last_lsn*/0,
|
||||||
|
/*my_config_tag*/1, /*my_last_lsn*/0,
|
||||||
|
/*in_catch_up*/0);
|
||||||
|
}
|
||||||
|
|
||||||
|
int a_majority = grants_a >= MAJORITY;
|
||||||
|
int b_majority = grants_b >= MAJORITY;
|
||||||
|
int primaries_this_term = (a_majority ? 1 : 0) + (b_majority ? 1 : 0);
|
||||||
|
printf("test_5_1 election-safety term=%d: grants A=%d B=%d (majority=%d) → "
|
||||||
|
"primaries_this_term=%d\n", (int)TERM, grants_a, grants_b,
|
||||||
|
(int)MAJORITY, primaries_this_term);
|
||||||
|
|
||||||
|
/* ELECTION SAFETY (raft/raft.tla): a voter that granted A in term T must DENY
|
||||||
|
* B in term T (vote-once), so the two candidates can never BOTH reach a
|
||||||
|
* majority — at most one committing primary per term. */
|
||||||
|
TEST_ASSERT_FALSE_MESSAGE(a_majority && b_majority,
|
||||||
|
"Election Safety (raft.tla): two candidates must NOT both reach a majority "
|
||||||
|
"in the same term — each voter grants at most one vote per term (votedFor)");
|
||||||
|
TEST_ASSERT_LESS_OR_EQUAL_INT_MESSAGE(1, primaries_this_term,
|
||||||
|
"the per-term committing-primary count must be <= 1 (Req 4.1)");
|
||||||
|
|
||||||
|
for (int i = 0; i < N_VOTERS; i++)
|
||||||
|
ha_fini(&v[i]);
|
||||||
|
printf("test_5_1_election_safety_one_primary_per_term: done\n");
|
||||||
|
}
|
||||||
|
|
||||||
/* ═══════════════════════════════════════════════════════════════════════════
|
/* ═══════════════════════════════════════════════════════════════════════════
|
||||||
* SCAFFOLDING (PASS) — prove each shared seam is usable by later phases.
|
* SCAFFOLDING (PASS) — prove each shared seam is usable by later phases.
|
||||||
* ═══════════════════════════════════════════════════════════════════════════ */
|
* ═══════════════════════════════════════════════════════════════════════════ */
|
||||||
|
|
@ -1475,6 +1742,17 @@ void test_ha_core_safety(void)
|
||||||
* catches the committed boundary. */
|
* catches the committed boundary. */
|
||||||
RUN_TEST(test_4_1_follower_reconverges_multientry_divergent_tail);
|
RUN_TEST(test_4_1_follower_reconverges_multientry_divergent_tail);
|
||||||
RUN_TEST(test_4_1_apply_gap_blocks_converged_until_applied);
|
RUN_TEST(test_4_1_apply_gap_blocks_converged_until_applied);
|
||||||
|
/* Task 5.1 election liveness & safety under churn:
|
||||||
|
* CORE RED — a control-channel flap while a current-term primary is alive
|
||||||
|
* must NOT launch a PreVote failover (FAILs on the baseline:
|
||||||
|
* the RDHUP path has no ha_heartbeat_is_fresh guard).
|
||||||
|
* LIVENESS — genuine loss still elects; a failed round re-arms; rapid
|
||||||
|
* rounds advance the term (guards 5.2 vs over-suppression).
|
||||||
|
* SAFETY — at most one committing primary per term (Election Safety,
|
||||||
|
* falsifiable, mapped to raft/raft.tla; reports per-term count). */
|
||||||
|
RUN_TEST(test_5_1_spurious_failover_suppressed_while_primary_alive);
|
||||||
|
RUN_TEST(test_5_1_genuine_loss_elects_and_advances_term);
|
||||||
|
RUN_TEST(test_5_1_election_safety_one_primary_per_term);
|
||||||
/* Scaffolding: each shared seam is usable by 2.3 / 4.1 / 5.1. */
|
/* Scaffolding: each shared seam is usable by 2.3 / 4.1 / 5.1. */
|
||||||
RUN_TEST(test_seam_wal_construct_and_probe);
|
RUN_TEST(test_seam_wal_construct_and_probe);
|
||||||
RUN_TEST(test_seam_divergence_is_observable);
|
RUN_TEST(test_seam_divergence_is_observable);
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue