From adda4b8b9c3f5ee2d831aa54429f8ea03ad1a5d0 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Wed, 1 Jul 2026 01:15:16 +0000 Subject: [PATCH] docs(raft-ha-core-safety): root-cause round-4 election-escalation tail (task 5.10b) freshness-deny escalation loop under a just-restarted-survivor overlap: a behind-log candidate is freshness-denied every term (18->24 observed), times out, re-campaigns at term+1 -> ~18 escalations before convergence. NOT RNG (seed is /dev/urandom); 5.7 jitter bounds retry rate, not the escalation. Fix direction: a repeatedly-freshness-denied candidate must recognize it is behind and defer/catch-up instead of laddering the cluster term. Needs a deterministic 2-survivor repro before a fix (5.5/5.6/5.7 family). --- .kiro/specs/raft-ha-core-safety/research.md | 31 +++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 7327e758..f0f2287d 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -854,3 +854,34 @@ lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2 Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 / ha-commit-durability catch-up backfill). + +## Task 5.10(b) — round-4 election-escalation tail: root cause (2026-06-30) + +The bug-1306-fix soak (`/home/claude/tmp/ha-1306fix`, committed_job_loss=0) hit a round-4 +election-escalation storm: terms laddered 6→~25 in one round window, overrunning the 40s +harness window ("NO new primary within window"), before converging (elected round 5). + +**Not RNG-correlation.** `ha_seed_jitter` seeds from `/dev/urandom` (good entropy); the two +survivors' xorshift streams are independent. So the jitter `[T,2T]` is working as designed. + +**Mechanism = a FRESHNESS-DENY escalation loop under a just-restarted-survivor overlap.** +Round 4 killed the term-6 leader (ctrl-1) while ctrl-2 was JUST RESTARTED (low/behind log). +ctrl-2 campaigned and was denied `reason=freshness` at EVERY term 18→24 (waldump/log: ctrl-2 +log shows freshness at 18,19,20,21,22,23,24), timing out each campaign → term++ → re-campaign. +The fresher survivor did not promptly win either (ctrl-3 led at 4, then not again until 27), +so the two survivors churned terms for ~18 increments before one converged. 5.7's per-campaign +jitter bounds the retry RATE but NOT the escalation, because the driver is repeated freshness +rejection of a behind candidate, not a timing collision the jitter can desync. + +**Fix direction (NOT a quick patch — same 5.5/5.6/5.7 election-liveness family):** a candidate +that is repeatedly freshness-denied must recognize it is BEHIND and stop laddering its term — +defer to the fresher survivor (catch up first) rather than re-campaign at term+1 forever. I.e. +gate the real-election term increment on "could I actually win the freshness check?" (a behind +node should PreVote-fail and back off to catch-up, not inflate the cluster term). Must preserve +genuine-loss liveness (R6), the 5.2 spurious-failover suppression, and Election Safety. + +**Validation requires a DETERMINISTIC 2-survivor repro** (one behind-log survivor + one fresher, +primary killed; assert the cluster elects within a BOUNDED number of terms, no unbounded +freshness-deny escalation) before any fix — speculative tuning here risks regressing 5.5/5.6/5.7. +Stochastic: this tail appeared in 1 of 2 ten-round soaks (the other laddered single-increment +terms 4→13). Tracked as task 5.10(b).