lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): randomize per-campaign election timeout to [T,2T] (task 5.7)

Browse source 
The per-campaign election watchdog was armed with a FIXED election_timeout_ms
(raft_start_election -> on_schedule_timeout -> raft_cb_schedule_timeout). Under
cold-connection churn a just-restarted survivor opens a fresh TCP+SSL endpoint
per vote round, so its single needed grant can land a hair AFTER a constant
deadline on EVERY campaign -- a resonance: the campaign times out, the term
re-escalates, and the now-stale grant is dropped (DropStaleResponse). Forever.
Terms climb unbounded; the freshest survivor never tallies quorum. This is the
final round-4 chaotic-soak election-liveness stall.

Fix (Ongaro thesis 3.4): re-randomize the watchdog per campaign to the canonical
[T, 2T] interval in the HA adapter, so some campaign's deadline falls AFTER its
grant RTT and the grant lands in-window. Done in raft_cb_schedule_timeout so the
pure (sim-tested) raft library stays deterministic. Independent of
election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter)
so it spreads even when jmax is small/0; never below the configured floor so
genuine-loss failover is not sped up past election_timeout_ms.

Preserves Election Safety (no change to term/vote/DropStaleResponse logic).

new ha_jittered_election_delay() helper + 2 unit tests (RED verified: stubbing
jitter to 0 fails test_5_7_campaign_timeout_is_randomized with the resonance
message; PIN guards floor + modulo-by-zero). Suite: 24 Tests / 0 Failures.
This commit is contained in:
Rongsong Shen 2026-06-30 13:29:15 +00:00
parent 2b182ecde1
commit 74784b2b18
3 changed files with 131 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

87
src/controller/testsuite/test_ha_core_safety.c
View file

@ -1811,6 +1811,90 @@ static void test_5_6_genuine_loss_still_fails_over_when_not_electing(void)
printf("test_5_6_genuine_loss_still_fails_over_when_not_electing: PASS (pin)\n");
}
/* ═══════════════════════════════════════════════════════════════════════════
* TASK 5.7 — the per-campaign election watchdog must be RANDOMIZED [T, 2T]
*
* Surfaced after 5.6 (research.md §"Task 5.6 fix VERIFIED"): with RDHUP no
* longer preempting an in-progress election, round 4 STILL stalls. The freshest
* survivor (ctrl-2) wins PreVote, enters the real election, and ctrl-3 GRANTS
* it — but every grant arrives as vote_reply_stale_drop (reply term < ctrl-2's
* current term). Root cause: raft_start_election arms the campaign watchdog with
* a FIXED election_timeout_ms (raft.c on_schedule_timeout). Under cold-connection
* churn (a just-restarted survivor opens a fresh TCP+SSL endpoint per vote round)
* the single needed grant lands a hair AFTER that constant deadline on EVERY
* campaign — a resonance: the campaign times out, the term re-escalates, and the
* now-stale grant is dropped (DropStaleResponse). Forever. Terms climb 25→30+.
*
* Fix (Ongaro §3.4): re-randomize the watchdog per campaign to [T, 2T] so some
* campaign's deadline falls AFTER its grant RTT and the grant lands in-window.
*
* (CORE RED) test_5_7_campaign_timeout_is_randomized: the watchdog delay must
* VARY across campaigns and stay within [T, 2T]. A fixed timeout yields
* one distinct value (RED) — it cannot break the resonance.
*
* (PIN) test_5_7_floor_preserved_and_zero_safe: the randomized delay never
* drops BELOW the configured floor (genuine-loss failover is never sped up
* past election_timeout_ms), and base==0 is modulo-safe.
*
* Requirements: 6.1, 6.2, 6.3 (Election liveness under sustained failover churn).
* Preserves Election Safety (no change to term/vote/DropStaleResponse logic).
* ═══════════════════════════════════════════════════════════════════════════ */
/* CORE RED — the campaign watchdog must be re-randomized to [T, 2T] each call. */
static void test_5_7_campaign_timeout_is_randomized(void)
{
ha_state_t ha;
hcs5_make_standby(&ha);
ha.my_priority = 1;
ha.jitter_seed = 0; /* force a deterministic re-seed */
const uint32_t base = 5000; /* HA_DEFAULT_ELECTION_TIMEOUT_MS */
uint32_t samples[32];
for (int i = 0; i < 32; i++) {
samples[i] = ha_jittered_election_delay(&ha, base);
TEST_ASSERT_TRUE_MESSAGE(samples[i] >= base && samples[i] <= 2u * base,
"campaign watchdog must lie in the canonical Raft interval [T, 2T]");
}
int distinct = 0;
for (int i = 0; i < 32; i++) {
int seen = 0;
for (int j = 0; j < i; j++)
if (samples[j] == samples[i]) { seen = 1; break; }
if (!seen) distinct++;
}
TEST_ASSERT_TRUE_MESSAGE(distinct > 1,
"RED: the per-campaign election timeout must be RE-RANDOMIZED each "
"campaign. A fixed watchdog resonates with a constant cold-connection "
"grant RTT — the grant lands just past every deadline, the term climbs "
"unbounded, and every grant is dropped as stale (task 5.7).");
ha_fini(&ha);
printf("test_5_7_campaign_timeout_is_randomized: done\n");
}
/* PIN — never below the configured floor; base==0 is modulo-safe. */
static void test_5_7_floor_preserved_and_zero_safe(void)
{
ha_state_t ha;
hcs5_make_standby(&ha);
ha.my_priority = 1;
/* base==0 must not divide/modulo by zero — returns 0. */
TEST_ASSERT_EQUAL_UINT32_MESSAGE(0u, ha_jittered_election_delay(&ha, 0),
"PIN: base==0 must be modulo-safe and return 0");
/* The randomized delay never drops below the configured floor, so genuine
* single-survivor failover is never sped up past election_timeout_ms. */
for (int i = 0; i < 64; i++)
TEST_ASSERT_TRUE_MESSAGE(ha_jittered_election_delay(&ha, 2000) >= 2000u,
"PIN: the randomized campaign timeout must never fall BELOW the "
"configured election_timeout_ms floor");
ha_fini(&ha);
printf("test_5_7_floor_preserved_and_zero_safe: PASS (pin)\n");
}
/* ═══════════════════════════════════════════════════════════════════════════
* SCAFFOLDING (PASS) — prove each shared seam is usable by later phases.
* ═══════════════════════════════════════════════════════════════════════════ */
@ -1982,6 +2066,9 @@ void test_ha_core_safety(void)
* PIN — genuine loss (STANDBY, not electing) still fails over. */
RUN_TEST(test_5_6_rdhup_does_not_restart_election_in_progress);
RUN_TEST(test_5_6_genuine_loss_still_fails_over_when_not_electing);
RUN_TEST(test_5_7_campaign_timeout_is_randomized);
RUN_TEST(test_5_7_floor_preserved_and_zero_safe);
/* Scaffolding: each shared seam is usable by 2.3 / 4.1 / 5.1. */
RUN_TEST(test_seam_wal_construct_and_probe);
RUN_TEST(test_seam_divergence_is_observable);