lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-ha-core-safety): randomize per-campaign election timeout to [T,2T] (task 5.7)

Browse source 
The per-campaign election watchdog was armed with a FIXED election_timeout_ms
(raft_start_election -> on_schedule_timeout -> raft_cb_schedule_timeout). Under
cold-connection churn a just-restarted survivor opens a fresh TCP+SSL endpoint
per vote round, so its single needed grant can land a hair AFTER a constant
deadline on EVERY campaign -- a resonance: the campaign times out, the term
re-escalates, and the now-stale grant is dropped (DropStaleResponse). Forever.
Terms climb unbounded; the freshest survivor never tallies quorum. This is the
final round-4 chaotic-soak election-liveness stall.

Fix (Ongaro thesis 3.4): re-randomize the watchdog per campaign to the canonical
[T, 2T] interval in the HA adapter, so some campaign's deadline falls AFTER its
grant RTT and the grant lands in-window. Done in raft_cb_schedule_timeout so the
pure (sim-tested) raft library stays deterministic. Independent of
election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter)
so it spreads even when jmax is small/0; never below the configured floor so
genuine-loss failover is not sped up past election_timeout_ms.

Preserves Election Safety (no change to term/vote/DropStaleResponse logic).

new ha_jittered_election_delay() helper + 2 unit tests (RED verified: stubbing
jitter to 0 fails test_5_7_campaign_timeout_is_randomized with the resonance
message; PIN guards floor + modulo-by-zero). Suite: 24 Tests / 0 Failures.
This commit is contained in:
Rongsong Shen 2026-06-30 13:29:15 +00:00
parent 2b182ecde1
commit 74784b2b18
3 changed files with 131 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

24
src/controller/ha.c
View file

@ -449,6 +449,14 @@ static void raft_cb_schedule_timeout(void *ctx, uint32_t delay_ms)
t->c = c;
t->ctx = NULL; /* raft path does not use ha_elect_ctx_t */
/* task 5.7: randomize the per-campaign watchdog to [delay_ms, 2*delay_ms]
* (Ongaro §3.4). The library passes a FIXED election_timeout_ms; arming the
* watchdog at exactly that value lets a constant cold-connection grant RTT
* land just past every deadline forever (resonance → unbounded term climb).
* Re-randomizing here breaks the resonance without touching the pure
* (sim-tested) library. */
delay_ms = ha_jittered_election_delay(ha, delay_ms);
int rc = delay_task_ms(&c->delaymgr, (int)delay_ms, 1,
raft_election_timeout_fn, t);
if (rc < 0) {
@ -6671,6 +6679,22 @@ uint32_t ha_election_stagger_ms(ha_state_t *ha)
return delay;
}
/* ha_jittered_election_delay: see ha.h. Canonical Raft randomized election
* timeout — [base, 2*base], re-drawn each campaign — applied to the per-campaign
* watchdog so a constant cold-connection grant RTT cannot resonate just past a
* fixed deadline forever (task 5.7). Independent of election_jitter_max_ms and
* never below the configured floor. */
uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base)
{
if (!ha || base == 0)
return base;
if (ha->jitter_seed == 0)
ha_seed_jitter(&ha->jitter_seed, ha->my_priority);
uint32_t extra = (uint32_t)(xorshift64_next(&ha->jitter_seed)
% ((uint64_t)base + 1));
return base + extra;
}
/* failover_stagger_ctx_t: carries ha/c into the delaymgr-scheduled PreVote start.
* Freed by ha_failover_stagger_fn (one-shot). */
typedef struct {