diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index 4192d09..b096734 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -569,3 +569,41 @@ the freshest survivor is electable. bug-1282 (ABBA iomgr deadlock) and bug-1286 (round-4 restart SIGSEGV, NULL ep->owner) are FIXED and VERIFIED (no crash / no deadlock / no fdatasync across round-4 churn). The remaining 7.2 blocker is this election-liveness cross-deny - the hard core of Phase 5, no longer masked. + +--- + +# Task 5.5 fix VERIFIED as progress; round-4 now reaches a DEEPER layer (dueling-candidate term escalation) + +Landed the 5.5 fix (commit b8453b2: advertise persist_entry_term(p, log_len) as lastLogTerm instead +of the stale c->persist.term) and re-ran the 6-round soak. Round-4 frozen state: ctrl-2 tip 15 @ +term 5 (freshest), ctrl-3 tip 11 @ term 4 (staler). + +## 5.5 is correct and made real progress +BEFORE the fix the freshest survivor advertised a too-low lastLogTerm and EVERY voter's logOk +rejected it — it could not even win a PreVote. AFTER the fix ctrl-2 WINS PreVote (pre_vote_won) AND +enters the real election ("PreVote round WON — entering real election"). The advertised-term +asymmetry is fixed; the fresher candidate is now electable in principle. + +## The deeper layer it exposed (next task) +ctrl-2 wins PreVote -> real election (term N) -> RequestVote -> replies but NO became_leader -> +retries at term N+1, N+2 ... prospective term climbing 21->25->28->29 unbounded. Mechanism: ctrl-2's +primary (dead ctrl-1) keeps RDHUP'ing ~every 15s; 5.2 does NOT suppress (primary genuinely dead), +so each RDHUP re-triggers ha_begin_failover -> fresh PreVote -> real election -> term++. ctrl-2 and +ctrl-3 duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), +and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally +resolves a split vote. Neither reaches majority; the term escalates without converging. + +Classic Raft split-vote / dueling-candidates liveness problem, amplified by the RDHUP failover +re-trigger. SEPARATE layer from 5.5. Next: break the duel (per-node randomized failover backoff / +election jitter that desynchronizes the two survivors; and/or do not re-trigger a fresh failover +via RDHUP while a real election the node just entered is still within its timeout window). + +## Session net (4 layers peeled on the round-4 stall) +1. bug-1282 ABBA iomgr deadlock — FIXED (25e741d). +2. bug-1286 NULL ep->owner restart SIGSEGV — FIXED (5db734a). +3. bug-851/5.5 stale advertised lastLogTerm — FIXED (b8453b2), verified: the fresher survivor now + wins PreVote + enters the real election (could not before). +4. REMAINING: dueling-candidate split-vote + unbounded term escalation under the RDHUP failover + cadence — the round-4 election round still does not complete. New follow-up. +Consensus SAFETY intact throughout (no two leaders; job safety holds every completed round); this is +purely election LIVENESS under sustained churn. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 9791cf0..836630a 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -135,6 +135,15 @@ - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ - _Boundary: Election Liveness_ - _Depends: 5.2, 5.3_ + - _ACTUAL root cause (NOT the last_activity hypothesis) + PARTIAL FIX (commit b8453b2): the cross-deny was a stale **advertised lastLogTerm** — the candidate advertised `c->persist.term` (the persist running term, stale/0 after restart) instead of `persist_entry_term(p, log_len)` (the term of the last WAL entry, the same source voters use via on_get_entry_term). The freshest survivor advertised a too-low term → every voter's logOk rejected it. bug-851 family. FIX VERIFIED AS PROGRESS: the fresher survivor now WINS PreVote + enters the real election (could not before). But the round still does not COMPLETE → a deeper layer remains (task 5.6). Detail in `research.md` §"Task 5.5 fix VERIFIED"._ + +- [ ] 5.6 Break the dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence (NEW — the layer under 5.5) + - _Root cause (`research.md` §"Task 5.5 fix VERIFIED"): with 5.5's advertised-term fix in, round-4 ctrl-2 (freshest) wins PreVote and enters the real election, but never reaches majority — it retries at ever-higher terms (21→25→28→29…). Mechanism: the dead primary keeps RDHUP'ing ~every 15s; 5.2 correctly does not suppress (primary genuinely dead), so each RDHUP re-triggers `ha_begin_failover` → fresh PreVote → real election → term++. The two survivors duel in lockstep (each denies the other's real vote on already-voted-this-term / stale-term), and the deterministic ~15s RDHUP cadence defeats the randomized-backoff tiebreak that normally resolves a split vote._ + - Fix direction: introduce/strengthen per-node randomized failover backoff (election jitter) so two surviving candidates DESYNCHRONIZE and one wins; and/or do not re-trigger a fresh failover via RDHUP while a real election the node just entered is still within its election-timeout window (let the in-progress round complete or time out before re-arming). Preserve genuine-loss liveness (R6), spurious-failover suppression (5.2), and Election Safety. RED: two divergent-freshness survivors + a periodic RDHUP cadence; assert a primary is elected within a bounded number of rounds (term does not escalate unbounded). + - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no unbounded term escalation / split-vote stall) with all prior fixes in place — the green 7.2 gate + - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_ + - _Boundary: Election Liveness_ + - _Depends: 5.5_ - [ ] 6. Integration: single-seam invariant end-to-end - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end @@ -162,7 +171,7 @@ - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). That is the remaining 7.2 blocker. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._ + - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). 5.5 (advertised-term) is FIXED (b8453b2, verified: fresher survivor now wins PreVote + enters real election); the remaining 7.2 blocker is now **task 5.6** — dueling-candidate split-vote / unbounded term escalation under the RDHUP failover cadence. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure