diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 7327e758..f0f2287d 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -854,3 +854,34 @@ lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2 Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 / ha-commit-durability catch-up backfill). + +## Task 5.10(b) — round-4 election-escalation tail: root cause (2026-06-30) + +The bug-1306-fix soak (`/home/claude/tmp/ha-1306fix`, committed_job_loss=0) hit a round-4 +election-escalation storm: terms laddered 6→~25 in one round window, overrunning the 40s +harness window ("NO new primary within window"), before converging (elected round 5). + +**Not RNG-correlation.** `ha_seed_jitter` seeds from `/dev/urandom` (good entropy); the two +survivors' xorshift streams are independent. So the jitter `[T,2T]` is working as designed. + +**Mechanism = a FRESHNESS-DENY escalation loop under a just-restarted-survivor overlap.** +Round 4 killed the term-6 leader (ctrl-1) while ctrl-2 was JUST RESTARTED (low/behind log). +ctrl-2 campaigned and was denied `reason=freshness` at EVERY term 18→24 (waldump/log: ctrl-2 +log shows freshness at 18,19,20,21,22,23,24), timing out each campaign → term++ → re-campaign. +The fresher survivor did not promptly win either (ctrl-3 led at 4, then not again until 27), +so the two survivors churned terms for ~18 increments before one converged. 5.7's per-campaign +jitter bounds the retry RATE but NOT the escalation, because the driver is repeated freshness +rejection of a behind candidate, not a timing collision the jitter can desync. + +**Fix direction (NOT a quick patch — same 5.5/5.6/5.7 election-liveness family):** a candidate +that is repeatedly freshness-denied must recognize it is BEHIND and stop laddering its term — +defer to the fresher survivor (catch up first) rather than re-campaign at term+1 forever. I.e. +gate the real-election term increment on "could I actually win the freshness check?" (a behind +node should PreVote-fail and back off to catch-up, not inflate the cluster term). Must preserve +genuine-loss liveness (R6), the 5.2 spurious-failover suppression, and Election Safety. + +**Validation requires a DETERMINISTIC 2-survivor repro** (one behind-log survivor + one fresher, +primary killed; assert the cluster elects within a BOUNDED number of terms, no unbounded +freshness-deny escalation) before any fix — speculative tuning here risks regressing 5.5/5.6/5.7. +Stochastic: this tail appeared in 1 of 2 ten-round soaks (the other laddered single-increment +terms 4→13). Tracked as task 5.10(b).