docs(raft-ha-core-safety): task 7.2 chaotic soak NO-GO — classify + add follow-up 5.3

First acceptance chaotic soak (6 rounds, real-fs ext4, release binaries rebuilt
at HEAD 01419e1 with Phase-2 payload fix + Phase-5 5.2 suppression). Outcome:

PASS (job safety + landed fixes):
- Zero committed-job loss (all tracked jobs present on the new primary each round).
- Committed prefix [1..11] byte-identical across all 3 members — Phase-2 payload-loss
  signature absent. All counted walcompare divergences are present=2 catch-up-LAG
  MISSING tails of restarted-uncaught followers (tips 19/15/11), not corruption.
- Rounds 1-3 elect a strictly higher-term primary; 5.2 spurious-failover suppression
  confirmed working (round-4 genuine-loss takes the failover path, not suppression).

NO-GO (R8.3 liveness):
- Round-4 election STALL after 3 overlapping kills: the up-to-date leader (tip 19)
  is killed, both survivors are lagging (15, 11) but each holds the full committed
  prefix; their PreVote rounds repeatedly time out with no grant quorum, so no
  primary is elected. Distinct from the 5.2 flap. Contributors: freshness
  mutual-denial, in_catch_up-denies-all-votes (Req 2.5, obs 1240) with no live
  primary to catch up from, and/or unestablished peer vote connections.

Per the chaotic-harness contingency: recorded full classification in research.md
§"Task 7.2"; added task 5.3 (lagging-but-safe survivor must be electable under
overlapping churn); 7.2 marked _Blocked_ on 5.3; logged bug-1272. Frozen evidence
at /home/claude/tmp/ha-7p2-frozen.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -250,3 +250,57 @@ The primary↔ctrl-2 control channel keeps **RDHUP-flapping** (~15s period). Bec
 
 ## Process note
 The dispatched investigation subagent backgrounded the soak and hung (~no findings returned); this section was reconstructed by the parent directly from the frozen `~/tmp/ha-estall` controller logs.
+
+---
+
+# Task 7.2 — Acceptance chaotic soak (first run with Phase-2+5 fixes): NO-GO + new finding
+
+Ran `run-ha-soak.sh --chaotic 6` on real-fs (ext4 `/home/claude/tmp/ha-7p2`) with the
+**release binaries rebuilt at HEAD `01419e1`** (Phase-2 payload fix + Phase-5 5.2
+spurious-failover suppression). Frozen at `/home/claude/tmp/ha-7p2-frozen`.
+
+## What PASSED (job-safety + the landed fixes)
+- **Zero committed-JOB loss.** Every round the job-verify confirmed all tracked jobs
+  present on the new primary (round1 6/6, round2 9/9, round3 12/12).
+- **Zero TRUE committed divergence.** `walcompare --committed 11` over the frozen
+  WALs: **"OK — committed prefixes identical"** across all 3 members. The Phase-2
+  payload-loss signature (present=3, one member `pcrc=0` vs majority real) is ABSENT.
+  `lsn=8` is `pcrc=0` *identically on all three* — a legit empty NOOP marker, not loss.
+- **The counted `walcompare` divergences are catch-up LAG, not corruption.** The only
+  boundary-19 divergence is `lsn=12 present=2`: ctrl-3 (tip=11, restarted round 3) is
+  MISSING it; ctrl-1/ctrl-2 hold it with matching content. All "divergences" are
+  `present=2` MISSING tails of restarted-but-uncaught followers (tips ctrl-1=19,
+  ctrl-2=15, ctrl-3=11), which an inflated `--committed` boundary over-claims as
+  committed — they are uncommitted/in-flight catch-up tail, exactly the research.md
+  Phase-2-interim note.
+- **5.2 spurious-failover suppression confirmed working.** Rounds 1–3 each elected a
+  strictly higher-term primary (ctrl-3 t3 → ctrl-2 t4 → ctrl-1 t5): liveness intact,
+  the fix did not over-suppress. The round-4 RDHUP logs show the GENUINE-loss path
+  ("primary_ctrl_fd RDHUP → begin PreVote failover"), NOT the suppression message —
+  correct, because the primary was truly dead (`ha_heartbeat_is_fresh==0`).
+
+## What FAILED — round-4 election stall (NEW residual liveness defect)
+After 3 overlapping kills, round 4 kills ctrl-1 (tip 19, the only fully-up-to-date
+node). The two survivors are both restarted-and-lagging (ctrl-2 tip=15, ctrl-3 tip=11),
+and their PreVote rounds **repeatedly time out before a decision** (no grant quorum):
+each loops `ha_start_prevote_async → ha_prevote_round_timeout_fn (LOSS) → RDHUP →
+ha_begin_failover → …` indefinitely, prospective terms frozen (ctrl-2=5, ctrl-3=6,
+never bumped because no PreVote ever WINS). The soak wedged here until the 10-min cap.
+
+**This is NOT the spurious-failover flap (5.2 closed that) and NOT a job-safety bug.**
+A SAFE candidate exists — ctrl-2 holds the full committed prefix [1..11] (Leader
+Completeness satisfied) and is the freshest survivor — yet the election never converges
+on it under sustained overlapping churn. Observed contributors (need a dedicated
+RED→bisect): freshness mutual-denial (`PreVote DENIED … reason=freshness, floor lsn=11`)
+between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240
+"a catching-up node sends zero PreVote replies") with no live primary left to catch up
+FROM, and/or peer-to-peer vote connections not established between two nodes that each
+only held a control channel to the now-dead primary.
+
+## Verdict
+**7.2 = NO-GO.** Job safety (R8.1/R8.2) holds; the acceptance liveness criterion
+(R8.3 — reconverge 3-of-3 + elect after EVERY kill round) fails at round 4. Per the
+design's chaotic-harness Contingency, this is a recorded new finding, not closure:
+added follow-up **task 5.3** (overlapping-churn election stall — a lagging-but-safe
+survivor must be electable; the catching-up/freshness vote gates must not deadlock when
+the primary is gone). 7.2 stays blocked on 5.3. Re-run 7.2 after 5.3 lands.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -110,6 +110,15 @@
   - _Boundary: Election Liveness_
   - _Depends: 4.2_
 
+- [ ] 5.3 Lagging-but-safe survivor must be electable under overlapping churn (NEW — surfaced by the 7.2 soak)
+  - _Root cause (task 7.2 first soak, `research.md` §"Task 7.2"): distinct from the 5.2 flap. After ≥3 overlapping kills the only fully-up-to-date node is killed and the two survivors are both restarted-and-lagging (e.g. tips 15 and 11 after the leader at 19 dies). Their PreVote rounds repeatedly time out with no grant quorum — prospective terms frozen, never bumped — so no primary is elected, even though the fresher survivor holds the FULL committed prefix (Leader Completeness satisfied) and is a SAFE candidate. Observed contributors: freshness mutual-denial between two lagging survivors, the in_catch_up-denies-all-votes gate (Req 2.5; obs 1240) with no live primary left to catch up FROM, and/or peer-to-peer vote connections never established between two nodes that each only held a control channel to the now-dead primary._
+  - RED: deterministically construct the round-4 shape (kill the up-to-date leader; two survivors each holding the full committed prefix but at different lagging tips, both flagged in_catch_up, no live primary) and assert the freshest survivor reaches CTRL_PRIMARY at a higher term within the failover window (no permanent PreVote-timeout loop). Reuse the 1.4 kill/election + apply-gap seams + the 5.1 PreVote dispatch seam.
+  - Fix: the catching-up/freshness vote gates must not deadlock when the primary is gone — a survivor that holds the majority-agreed committed prefix must be able to grant/win a PreVote once no live primary remains (drop in_catch_up for vote-eligibility when the primary is unreachable AND the candidate is ≥ the voter's committed floor; ensure the peer vote connection is established during failover). Preserve Election Safety (≤1 primary/term), Leader Completeness (never elect a candidate below the committed floor), and the 5.2 spurious-failover suppression.
+  - Observable completion: the RED passes; a 6-round chaotic soak elects a new primary after EVERY kill round (no round-N stall) with zero committed-job loss; Election Safety still holds
+  - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
+  - _Boundary: Election Liveness_
+  - _Depends: 5.2_
+
 - [ ] 6. Integration: single-seam invariant end-to-end
 - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
   - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
@@ -135,7 +144,8 @@
   - Run the chaotic soak (≥6 rounds, overlapping kills) against real-fs release binaries and assert zero `walcompare` divergence, zero committed-job loss, and 3-of-3 reconvergence plus a new primary after every kill round, with bug-854/bug-924 isolated from the verdict
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
-  - _Depends: 1.5, 6.2, 5.2_
+  - _Depends: 1.5, 6.2, 5.2, 5.3_
+  - _Blocked: first soak run (HEAD 01419e1, Phase-2+5 fixes) is NO-GO at round 4 — job safety + Phase-2 prefix integrity hold (zero job loss; committed prefix [1..11] identical; all counted divergences are catch-up-lag MISSING tails, not corruption) and 5.2 spurious-failover suppression is confirmed working, but the cluster STALLS electing a new primary after 3 overlapping kills (two lagging-but-safe survivors mutually fail PreVote). Root cause + full classification in `research.md` §"Task 7.2"; fix owned by NEW task 5.3. Re-run 7.2 after 5.3 lands._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure