docs(raft-ha-core-safety): bug-1282+bug-1286 verified fixed; unmask round-4 election cross-deny (task 5.5)

6-round chaotic soak with both fixes: rounds 1-3 elect; round 4 has NO segfault,
NO fdatasync, NO iomgr deadlock — both fixes held. The round-4 stall is now the
pure election-liveness cross-deny that the crash/deadlock masked: the freshest
survivor (floor 15) is the legitimate winner, but the other survivor (floor 12)
denies it on eligibility because it still believes its dead primary is alive
(in_catch_up + heartbeat-not-yet-stale). Suspect 5.2's last_activity re-arm
keeping a node falsely-fresh after primary death.

Adds task 5.5 (release the heartbeat-fresh/in_catch_up deny on genuine primary
death within the failover window) as the remaining 7.2 blocker. bug-1282 and
bug-1286 are fixed and verified; consensus safety unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -537,3 +537,35 @@ flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bu
 
 7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock)
 is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected.
+
+---
+
+# Both fixes verified (bug-1282 + bug-1286) — round-4 now a CLEAN election-liveness cross-deny
+
+6-round chaotic soak on real-fs release WITH bug-1282 (deadlock) + bug-1286 (NULL ep->owner crash)
+both fixed: rounds 1-3 elect; round 4 has NO segfault, NO fdatasync, NO iomgr deadlock (both fixes
+held, confirmed). The round-4 stall is now the GENUINE consensus-liveness edge the crash/deadlock
+were masking:
+
+- ctrl-1 (primary term 6) killed round 4.
+- ctrl-2 (recovered durable_floor=15, the FRESHER survivor) denies ctrl-3 on FRESHNESS (ctrl-3
+  advertises 12 < 15) - correct.
+- ctrl-3 (floor 12) denies the fresher ctrl-2 on ELIGIBILITY (raft_cb_vote_eligible), repeatedly
+  for 2+ minutes - because ctrl-3 still believes its primary (the dead ctrl-1) is alive
+  (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd still >=0). So the legitimate winner
+  (ctrl-2) never gets ctrl-3's grant -> stall.
+
+This is requirements.md symptom #3 (round-N election stall) in pure form. Contributors to chase:
+(a) ctrl-3's "primary alive" view not expiring within the round - the heartbeat-fresh/in_catch_up
+eligibility deny (gated on primary_ctrl_fd>=0 && ha_heartbeat_is_fresh by task 5.3) does not lift
+because last_activity is not going stale; SUSPECT task 5.2's ha_arm_heartbeat_timeout_vtask re-arm
+resetting last_activity=now on the RDHUP-suppress path keeps a node falsely-fresh after the primary
+dies - revisit whether the suppress path should reset last_activity. (b) a stale primary_ctrl_fd to
+the dead primary not cleared promptly. Next Phase-5 task: a node denying on heartbeat-freshness/
+in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so
+the freshest survivor is electable.
+
+## Session net
+bug-1282 (ABBA iomgr deadlock) and bug-1286 (round-4 restart SIGSEGV, NULL ep->owner) are FIXED and
+VERIFIED (no crash / no deadlock / no fdatasync across round-4 churn). The remaining 7.2 blocker is
+this election-liveness cross-deny - the hard core of Phase 5, no longer masked.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -128,6 +128,14 @@
   - _Depends: 5.3_
 
 
+- [ ] 5.5 Release the heartbeat-fresh/in_catch_up vote deny on genuine primary death within the failover window (NEW — the unmasked round-4 stall)
+  - _Root cause (7.2 soak with bug-1282+bug-1286 fixed, `research.md` §"Both fixes verified"): with the crash + deadlock gone, round 4 stalls on a pure election-liveness cross-deny. The freshest survivor (ctrl-2, floor 15) is the legitimate winner, but the other survivor (ctrl-3, floor 12) denies it on ELIGIBILITY (`raft_cb_vote_eligible`) for minutes because ctrl-3 still believes its primary (the dead ctrl-1) is alive (in_catch_up + heartbeat-not-yet-stale, primary_ctrl_fd≥0). The "primary alive" view does not expire within the round → the freshest survivor never gets a grant._
+  - Investigate/fix: a node denying a PreVote on heartbeat-freshness or in_catch_up must detect a genuinely-dead primary within the failover window and release the deny so the freshest survivor is electable. Prime suspect: task 5.2's `ha_arm_heartbeat_timeout_vtask` re-arm resets `last_activity=now` on the RDHUP-suppress path, keeping a node falsely-fresh after the primary dies — revisit whether the suppress path should reset last_activity (or should only re-arm the watchdog without bumping the freshness clock). Also ensure a stale `primary_ctrl_fd` to a dead primary is cleared promptly. RED: two divergent-freshness survivors, primary killed; assert the freshest reaches CTRL_PRIMARY within the failover window (no multi-round eligibility cross-deny). Preserve Election Safety + the 5.2 spurious-failover suppression (do not re-introduce the flap).
+  - Observable completion: the RED passes; a 6-round chaotic soak elects after every kill round (no round-N cross-deny stall) with bug-1282/bug-1286 fixes in place
+  - _Requirements: 4.1, 4.3, 6.1, 6.2, 6.3_
+  - _Boundary: Election Liveness_
+  - _Depends: 5.2, 5.3_
+
 - [ ] 6. Integration: single-seam invariant end-to-end
 - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
   - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
@@ -154,7 +162,7 @@
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
   - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
-  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). STILL blocks the gate until that lands. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1286 … ROOT-CAUSED". Re-run 7.2 after bug-1286 is fixed._
+  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). is **FIXED** (commit 5db734a: NULL-guard `c = ep->owner` in myreq_handler). With BOTH bug-1282 and bug-1286 fixed, the round-4 soak shows NO crash / NO deadlock / NO fdatasync — but still stalls on a THIRD, now-unmasked cause: **task 5.5** — a pure election-liveness cross-deny (the freshest survivor is denied on eligibility because the other survivor still believes its dead primary is alive). That is the remaining 7.2 blocker. Consensus safety (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"Both fixes verified". Re-run 7.2 after 5.5 lands._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure