docs(raft-ha-core-safety): task 7.2 chaotic soak NO-GO — classify + add follow-up 5.3
First acceptance chaotic soak (6 rounds, real-fs ext4, release binaries rebuilt
at HEAD 01419e1 with Phase-2 payload fix + Phase-5 5.2 suppression). Outcome:
PASS (job safety + landed fixes):
- Zero committed-job loss (all tracked jobs present on the new primary each round).
- Committed prefix [1..11] byte-identical across all 3 members — Phase-2 payload-loss
signature absent. All counted walcompare divergences are present=2 catch-up-LAG
MISSING tails of restarted-uncaught followers (tips 19/15/11), not corruption.
- Rounds 1-3 elect a strictly higher-term primary; 5.2 spurious-failover suppression
confirmed working (round-4 genuine-loss takes the failover path, not suppression).
NO-GO (R8.3 liveness):
- Round-4 election STALL after 3 overlapping kills: the up-to-date leader (tip 19)
is killed, both survivors are lagging (15, 11) but each holds the full committed
prefix; their PreVote rounds repeatedly time out with no grant quorum, so no
primary is elected. Distinct from the 5.2 flap. Contributors: freshness
mutual-denial, in_catch_up-denies-all-votes (Req 2.5, obs 1240) with no live
primary to catch up from, and/or unestablished peer vote connections.
Per the chaotic-harness contingency: recorded full classification in research.md
§"Task 7.2"; added task 5.3 (lagging-but-safe survivor must be electable under
overlapping churn); 7.2 marked _Blocked_ on 5.3; logged bug-1272. Frozen evidence
at /home/claude/tmp/ha-7p2-frozen.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
parent
01419e1310
commit
da25ad53e1
3 changed files with 1617 additions and 823 deletions
|
|
@ -110,6 +110,15 @@
|
|||
- _Boundary: Election Liveness_
|
||||
- _Depends: 4.2_
|
||||
|
||||
- [ ] 5.3 Lagging-but-safe survivor must be electable under overlapping churn (NEW — surfaced by the 7.2 soak)
|
||||
- _Root cause (task 7.2 first soak, `research.md` §"Task 7.2"): distinct from the 5.2 flap. After ≥3 overlapping kills the only fully-up-to-date node is killed and the two survivors are both restarted-and-lagging (e.g. tips 15 and 11 after the leader at 19 dies). Their PreVote rounds repeatedly time out with no grant quorum — prospective terms frozen, never bumped — so no primary is elected, even though the fresher survivor holds the FULL committed prefix (Leader Completeness satisfied) and is a SAFE candidate. Observed contributors: freshness mutual-denial between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240) with no live primary left to catch up FROM, and/or peer-to-peer vote connections never established between two nodes that each only held a control channel to the now-dead primary._
|
||||
- RED: deterministically construct the round-4 shape (kill the up-to-date leader; two survivors each holding the full committed prefix but at different lagging tips, both flagged in_catch_up, no live primary) and assert the freshest survivor reaches CTRL_PRIMARY at a higher term within the failover window (no permanent PreVote-timeout loop). Reuse the 1.4 kill/election + apply-gap seams + the 5.1 PreVote dispatch seam.
|
||||
- Fix: the catching-up/freshness vote gates must not deadlock when the primary is gone — a survivor that holds the majority-agreed committed prefix must be able to grant/win a PreVote once no live primary remains (drop in_catch_up for vote-eligibility when the primary is unreachable AND the candidate is ≥ the voter's committed floor; ensure the peer vote connection is established during failover). Preserve Election Safety (≤1 primary/term), Leader Completeness (never elect a candidate below the committed floor), and the 5.2 spurious-failover suppression.
|
||||
- Observable completion: the RED passes; a 6-round chaotic soak elects a new primary after EVERY kill round (no round-N stall) with zero committed-job loss; Election Safety still holds
|
||||
- _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
|
||||
- _Boundary: Election Liveness_
|
||||
- _Depends: 5.2_
|
||||
|
||||
- [ ] 6. Integration: single-seam invariant end-to-end
|
||||
- [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
|
||||
- Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
|
||||
|
|
@ -135,7 +144,8 @@
|
|||
- Run the chaotic soak (≥6 rounds, overlapping kills) against real-fs release binaries and assert zero `walcompare` divergence, zero committed-job loss, and 3-of-3 reconvergence plus a new primary after every kill round, with bug-854/bug-924 isolated from the verdict
|
||||
- Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
|
||||
- _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Depends: 1.5, 6.2, 5.2_
|
||||
- _Depends: 1.5, 6.2, 5.2, 5.3_
|
||||
- _Blocked: first soak run (HEAD 01419e1, Phase-2+5 fixes) is NO-GO at round 4 — job safety + Phase-2 prefix integrity hold (zero job loss; committed prefix [1..11] identical; all counted divergences are catch-up-lag MISSING tails, not corruption) and 5.2 spurious-failover suppression is confirmed working, but the cluster STALLS electing a new primary after 3 overlapping kills (two lagging-but-safe survivors mutually fail PreVote). Root cause + full classification in `research.md` §"Task 7.2"; fix owned by NEW task 5.3. Re-run 7.2 after 5.3 lands._
|
||||
|
||||
- [ ] 7.3 Hypothesis-validation closeout
|
||||
- Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue