lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(raft-ha-core-safety): root-cause round-4 election-escalation tail (task 5.10b)

Browse source 
freshness-deny escalation loop under a just-restarted-survivor overlap: a
behind-log candidate is freshness-denied every term (18->24 observed), times
out, re-campaigns at term+1 -> ~18 escalations before convergence. NOT RNG
(seed is /dev/urandom); 5.7 jitter bounds retry rate, not the escalation.
Fix direction: a repeatedly-freshness-denied candidate must recognize it is
behind and defer/catch-up instead of laddering the cluster term. Needs a
deterministic 2-survivor repro before a fix (5.5/5.6/5.7 family).
This commit is contained in:
Rongsong Shen 2026-07-01 01:15:16 +00:00
parent 20cfd2751d
commit adda4b8b9c
1 changed files with 31 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

31
.kiro/specs/raft-ha-core-safety/research.md
View file

@ -854,3 +854,34 @@ lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2
Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 /
ha-commit-durability catch-up backfill).
## Task 5.10(b) — round-4 election-escalation tail: root cause (2026-06-30)
The bug-1306-fix soak (`/home/claude/tmp/ha-1306fix`, committed_job_loss=0) hit a round-4
election-escalation storm: terms laddered 6→~25 in one round window, overrunning the 40s
harness window ("NO new primary within window"), before converging (elected round 5).
**Not RNG-correlation.** `ha_seed_jitter` seeds from `/dev/urandom` (good entropy); the two
survivors' xorshift streams are independent. So the jitter `[T,2T]` is working as designed.
**Mechanism = a FRESHNESS-DENY escalation loop under a just-restarted-survivor overlap.**
Round 4 killed the term-6 leader (ctrl-1) while ctrl-2 was JUST RESTARTED (low/behind log).
ctrl-2 campaigned and was denied `reason=freshness` at EVERY term 18→24 (waldump/log: ctrl-2
log shows freshness at 18,19,20,21,22,23,24), timing out each campaign → term++ → re-campaign.
The fresher survivor did not promptly win either (ctrl-3 led at 4, then not again until 27),
so the two survivors churned terms for ~18 increments before one converged. 5.7's per-campaign
jitter bounds the retry RATE but NOT the escalation, because the driver is repeated freshness
rejection of a behind candidate, not a timing collision the jitter can desync.
**Fix direction (NOT a quick patch — same 5.5/5.6/5.7 election-liveness family):** a candidate
that is repeatedly freshness-denied must recognize it is BEHIND and stop laddering its term —
defer to the fresher survivor (catch up first) rather than re-campaign at term+1 forever. I.e.
gate the real-election term increment on "could I actually win the freshness check?" (a behind
node should PreVote-fail and back off to catch-up, not inflate the cluster term). Must preserve
genuine-loss liveness (R6), the 5.2 spurious-failover suppression, and Election Safety.
**Validation requires a DETERMINISTIC 2-survivor repro** (one behind-log survivor + one fresher,
primary killed; assert the cluster elects within a BOUNDED number of terms, no unbounded
freshness-deny escalation) before any fix — speculative tuning here risks regressing 5.5/5.6/5.7.
Stochastic: this tail appeared in 1 of 2 ten-round soaks (the other laddered single-increment
terms 4→13). Tracked as task 5.10(b).