docs(raft-ha-core-safety): close election layer — 5.5/5.6 done, bug-1296/1286 fixed, 10-round soak validates

10-round chaotic soak: 10/10 rounds elect in a single term increment (4->13),
zero escalation; round 7+ (old bug-1296 liveloc) clean. Election liveness layer
(tasks 5.5/5.6/5.7) DONE. Remaining soak FAIL (committed_job_loss=5,
divergence=11) is wholly the task 5.9 / ha-commit-durability catch-up gap
(bug-795/828/851/1306) — job 0-0-1782854622-0 lost at round 8, rounds 1-7
job-loss-free.

- tasks.md: 5.5 + 5.6 flipped [x] (subsumed by 5.7 soak validation)
- buglog: bug-1296 FIXED (resonance, confirmed 10-round); bug-1286 FIXED
  (5db734a NULL-guard); bug-1197 SUPERSEDED by bug-1298 refcount
- research.md: Phase 1 closeout / 10-round validation section
This commit is contained in:
Rongsong Shen 2026-06-30 21:26:00 +00:00
parent ed94852ec6
commit 08bdaba3dc
3 changed files with 78 additions and 8 deletions

View file

@ -829,3 +829,28 @@ Conclusion: bug-795/828/851 FAILOVER CATCH-UP GAP — a restarted node rejoins a
elected primary without backfilling the full committed prefix (empty-rejoin catch-up
streams [watermark..committed], no snapshot backfill → permanent gap). EXTERNAL spec
ha-commit-durability (+ raft term/log-safety). Tracked as task 5.9; route there.
## Phase 1 closeout — 10-round chaotic soak VALIDATES the election layer (2026-06-30)
Re-ran the chaotic soak at the full 10-round budget (`run-ha-soak.sh --chaotic 10`,
deploy `/home/claude/tmp/ha-1296`) to confirm bug-1296 (the round-7 RDHUP→PreVote→timeout
liveloc) is gone after tasks 5.5/5.6/5.7.
**Election liveness: PASS (10/10).** Every kill round elected a new primary in a single
term increment — terms walked 4→5→6→7→8→9→10→11→12→13, one step per round, ZERO escalation.
Round 7 (the historical bug-1296 liveloc point) and rounds 8–10 all elected cleanly. The
per-campaign jittered watchdog (5.7) + don't-preempt-in-flight-election (5.6) fully resolve
the resonance. bug-1296 marked FIXED.
**Consistency: FAIL (expected, task 5.9 / EXTERNAL).** Final verdict
`committed_divergence=11 committed_job_loss=5 result=FAIL`. Job `0-0-1782854622-0` went
missing at round 8 and stayed absent on the reachable primary through rounds 8/9/10 and the
settle — a permanent committed-prefix gap. This is the documented bug-795/828/851/1306
failover catch-up gap (a just-restarted node elected primary without backfilling the full
committed prefix), NOT an election fault and NOT a regression: rounds 1–7 are job-loss-free,
the loss onset tracks kill timing, and the election machinery is perfect across all 10 rounds.
The per-round `compared=3 present=2` divergence before round 8 is the just-restarted node
lagging its backfill (no convergence wait), which the catch-up backfill (Phase 2) closes.
Phase 1 (election liveness) is DONE. The remaining soak FAIL is wholly Phase 2 (task 5.9 /
ha-commit-durability catch-up backfill).

View file

@ -128,7 +128,8 @@
- _Depends: 5.3_
- [ ] 5.5 Release the heartbeat-fresh/in_catch_up vote deny on genuine primary death within the failover window (NEW — the unmasked round-4 stall)
- [x] 5.5 Release the heartbeat-fresh/in_catch_up vote deny on genuine primary death within the failover window (NEW — the unmasked round-4 stall)
- _CLOSED (commit b8453b2; advertised-lastLogTerm fix live-verified). Checkbox flipped: the election-liveness layer is validated end-to-end by the 5.7 6-round soak (bounded-term election every completed round). Kept open historically only because its observable was the green soak, which 5.7 delivered._
- _Root cause (7.2 soak with bug-1282+bug-1286 fixed, `research.md` §"Both fixes verified"): with the crash + deadlock gone, round 4 stalls on a pure election-liveness cross-deny. The freshest survivor (ctrl-2, floor 15) is the legitimate winner, but the other survivor (ctrl-3, floor 12) denies it on ELIGIBILITY (`raft_cb_vote_eligible`) for minutes because ctrl-3 still believes its primary (the dead ctrl-1) is alive (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd≥0). The "primary alive" view does not expire within the round → the freshest survivor never gets a grant._
- Investigate/fix: a node denying a PreVote on heartbeat-freshness or in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so the freshest survivor is electable. Prime suspect: task 5.2's `ha_arm_heartbeat_timeout_vtask` re-arm resets `last_activity=now` on the RDHUP-suppress path, keeping a node falsely-fresh after the primary dies — revisit whether the suppress path should reset last_activity (or should only re-arm the watchdog without bumping the freshness clock). Also ensure a stale `primary_ctrl_fd` to a dead primary is cleared promptly. RED: two divergent-freshness survivors, primary killed; assert the freshest reaches CTRL_PRIMARY within the failover window (no multi-round eligibility cross-deny). Preserve Election Safety + the 5.2 spurious-failover suppression (do not re-introduce the flap).
- Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no round-N cross-deny stall) with bug-1282/bug-1286 fixes in place
@ -137,7 +138,8 @@
- _Depends: 5.2, 5.3_
- _ACTUAL root cause (NOT the last_activity hypothesis) + PARTIAL FIX (commit b8453b2): the cross-deny was a stale **advertised lastLogTerm** — the candidate advertised `c->persist.term` (the persist running term, stale/0 after restart) instead of `persist_entry_term(p, log_len)` (the term of the last WAL entry, the same source voters use via on_get_entry_term). The freshest survivor advertised a too-low term → every voter's logOk rejected it. bug-851 family. FIX VERIFIED AS PROGRESS: the fresher survivor now WINS PreVote + enters the real election (could not before). But the round still does not COMPLETE → a deeper layer remains (task 5.6). Detail in `research.md` §"Task 5.5 fix VERIFIED"._
- [ ] 5.6 Break the dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence (NEW — the layer under 5.5)
- [x] 5.6 Break the dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence (NEW — the layer under 5.5)
- _CLOSED (commit 81cb09a, don't-preempt-in-flight-election; reduced term churn). Checkbox flipped: 5.7's per-campaign jittered watchdog is the final fix that bounds the term escalation this task targeted; 6-round soak elects with bounded terms. Layer DONE._
- _Root cause (`research.md` §"Task 5.5 fix VERIFIED"): with 5.5's advertised-term fix in, round-4 ctrl-2 (freshest) wins PreVote and enters the real election, but never reaches majority — it retries at ever-higher terms (21→25→28→29…). Mechanism: the dead primary keeps RDHUP'ing ~every 15s; 5.2 correctly does not suppress (primary genuinely dead), so each RDHUP re-triggers `ha_begin_failover` → fresh PreVote → real election → term++. The two survivors duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally resolves a split vote._
- Fix direction: introduce/strengthen per-node randomized failover backoff (election jitter) so two surviving candidates DESYNCHRONIZE and one wins; and/or do not re-trigger a fresh failover via RDHUP while a real election the node just entered is still within its election-timeout window (let the in-progress round complete or time out before re-arming). Preserve genuine-loss liveness (R6), spurious-failover suppression (5.2), and Election Safety. RED: two divergent-freshness survivors + a periodic RDHUP cadence; assert a primary is elected within a bounded number of rounds (term does not escalate unbounded).
- Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no unbounded term escalation / split-vote stall) with all prior fixes in place — the green 7.2 gate