docs(raft-ha-core-safety): bug-1282 fixed + isolate bug-1283 (the round-4 stall had two causes)

bug-1282 (ABBA iomgr deadlock) is FIXED and verified (commit 25e741d): zero
__lll_lock_wait threads post-fix, unit suite green. But a plain release soak WITH
the fix still segfaults a restarted standby at round 4 — and deadlocks hang, they
do not SIGSEGV, so that crash is a SEPARATE bug (bug-1283), most likely an
endpoint-lifetime UAF under chaotic restart (bug-1197 family).

bug-1283 is a heisenbug (invisible under ASAN and gdb; no usable core in the
sandbox) and likely UPSTREAM raft-drive-async / core libxce endpoint quiescence.
7.2 stays blocked on it. Consensus correctness (Phase-2/5.2/5.3) is unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -478,3 +478,33 @@ caught a real dispatch-layer deadlock.
 Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are
 correct and unit-validated; elections + job safety hold for every round that completes. The
 round-4 blocker is the upstream dispatch deadlock, not consensus logic.
+
+---
+
+# bug-1282 FIXED + bug-1283 isolated (the round-4 stall had TWO causes)
+
+Implemented the bug-1282 fix (commit 25e741d): promoted `endpoint_t.busy` to `_Atomic int` and
+made `endpoint_block_tasks`/`endpoint_unblock_tasks`/`endpoint_is_task_blocked` lockless, removing
+the `ep->lock` acquire from the vtask-eligibility path that ran under the dataq `q->lock`.
+
+## Verified
+- **Deadlock gone:** post-fix gdb soak shows ZERO `__lll_lock_wait` threads across all 3 nodes and
+  `endpoint_is_task_blocked` no longer appears in any lock path. The ABBA inversion is eliminated.
+- **No regression:** `controller_unit_test` all green (incl. `test_ha_core_safety` 20/0); libxce +
+  controller + unit test build clean.
+
+## But the round-4 stall had TWO independent causes
+A plain release soak WITH the fix STILL segfaults a restarted standby at round 4 (ctrl-3,
+"Segmentation fault (core dumped)"). **Deadlocks hang; they do not SIGSEGV** — so this crash is a
+SEPARATE memory bug (now bug-1283), not bug-1282. It is most likely an endpoint-lifetime UAF under
+chaotic restart (bug-1197 family): the dispatch eligibility path reads `vdata->u.iodata.ep` /
+endpoint state while io_worker tears down or mutates the same endpoint. It is a heisenbug —
+invisible under ASAN and under gdb (gdb perturbs it into a cold-start timeout / no crash) — and the
+sandbox has no usable core (read-only `core_pattern` → non-running systemd-coredump), so a
+backtrace needs a non-ASAN DEBUG (-O0 -g) build under gdb or a host with a live coredump pipeline.
+
+## Net for 7.2
+The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wedge). The remaining
+blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint
+quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and
+elections + job safety hold for every round that completes.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -154,7 +154,7 @@
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
   - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
-  - _Blocked on UPSTREAM (bug-1282), NOT consensus. ROOT-CAUSED (task 5.4, via gdb): the round-N stall is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (ep->lock → wants q->lock via dataq_enq) vs `vdispatcher` `dataq_deq` (q->lock → wants ep->lock via the `vtask_eligible`→`endpoint_is_task_blocked` match callback). io_worker wedges → the node stops accepting/answering PreVotes → two restarted survivors can't elect. The release SIGSEGV is the same ep/q race corrupting under churn. This is the raft-drive-async / core libxce async-io DISPATCH layer (a Non-Goal here) → fix routed to raft-drive-async (bug-1282). Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes (2-round soak PASSES; rounds 1–3 elect; committed prefix byte-identical). Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"ROOT CAUSE NAILED". Re-run 7.2 after bug-1282 lands upstream._
+  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1283 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — STILL blocks the gate; likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family), heisenbug invisible under ASAN+gdb, needs a non-ASAN DEBUG+core+gdb env. Likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1282 FIXED + bug-1283 isolated". Re-run 7.2 after bug-1283 is fixed._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure