lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): randomize per-campaign election timeout to [T,2T] (task 5.7)

Browse source 
The per-campaign election watchdog was armed with a FIXED election_timeout_ms
(raft_start_election -> on_schedule_timeout -> raft_cb_schedule_timeout). Under
cold-connection churn a just-restarted survivor opens a fresh TCP+SSL endpoint
per vote round, so its single needed grant can land a hair AFTER a constant
deadline on EVERY campaign -- a resonance: the campaign times out, the term
re-escalates, and the now-stale grant is dropped (DropStaleResponse). Forever.
Terms climb unbounded; the freshest survivor never tallies quorum. This is the
final round-4 chaotic-soak election-liveness stall.

Fix (Ongaro thesis 3.4): re-randomize the watchdog per campaign to the canonical
[T, 2T] interval in the HA adapter, so some campaign's deadline falls AFTER its
grant RTT and the grant lands in-window. Done in raft_cb_schedule_timeout so the
pure (sim-tested) raft library stays deterministic. Independent of
election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter)
so it spreads even when jmax is small/0; never below the configured floor so
genuine-loss failover is not sped up past election_timeout_ms.

Preserves Election Safety (no change to term/vote/DropStaleResponse logic).

new ha_jittered_election_delay() helper + 2 unit tests (RED verified: stubbing
jitter to 0 fails test_5_7_campaign_timeout_is_randomized with the resonance
message; PIN guards floor + modulo-by-zero). Suite: 24 Tests / 0 Failures.
This commit is contained in:
Rongsong Shen 2026-06-30 13:29:15 +00:00
parent 2b182ecde1
commit 74784b2b18
3 changed files with 131 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

20
src/controller/ha.h
View file

@ -1402,6 +1402,26 @@ void ha_begin_failover(ha_state_t *ha, cluster_t *c);
*/
uint32_t ha_election_stagger_ms(ha_state_t *ha);
/*
* ha_jittered_election_delay: the per-CAMPAIGN election-timeout watchdog delay,
* randomized to the canonical Raft interval [base, 2*base] (Ongaro §3.4) and
* re-drawn on every campaign.
*
* task 5.7: the campaign watchdog (raft_start_election → on_schedule_timeout →
* raft_cb_schedule_timeout) was previously armed with a FIXED election_timeout_ms.
* Under cold-connection churn a survivor's single needed grant can land a hair
* AFTER a constant deadline on EVERY campaign — a resonance that times out,
* re-escalates the term, and then drops the now-stale grant (DropStaleResponse)
* forever (unbounded term climb; the final round-4 chaotic-soak stall).
* Randomizing the watchdog per campaign breaks the resonance: some campaign's
* deadline falls AFTER its grant RTT, the grant lands in-window, quorum is
* reached. Independent of election_jitter_max_ms (which tunes only the
* sub-second startup/stagger jitter) so it still spreads when jmax is small/0.
* Never returns below `base`, so genuine-loss failover is never sped up past the
* configured election_timeout_ms floor. base==0 returns 0 (no modulo-by-zero).
*/
uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base);
/*
* ha_dispatch_repl_action_for_standby: evaluate the replication decision for
* one standby and execute the resulting action.