docs(raft-ha-core-safety): bug-1282 fixed + isolate bug-1283 (the round-4 stall had two causes)

bug-1282 (ABBA iomgr deadlock) is FIXED and verified (commit 25e741d): zero
__lll_lock_wait threads post-fix, unit suite green. But a plain release soak WITH
the fix still segfaults a restarted standby at round 4 — and deadlocks hang, they
do not SIGSEGV, so that crash is a SEPARATE bug (bug-1283), most likely an
endpoint-lifetime UAF under chaotic restart (bug-1197 family).

bug-1283 is a heisenbug (invisible under ASAN and gdb; no usable core in the
sandbox) and likely UPSTREAM raft-drive-async / core libxce endpoint quiescence.
7.2 stays blocked on it. Consensus correctness (Phase-2/5.2/5.3) is unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC

.kiro/specs/raft-ha-core-safety/research.md

@@ -478,3 +478,33 @@ caught a real dispatch-layer deadlock.
 Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are
 correct and unit-validated; elections + job safety hold for every round that completes. The
 round-4 blocker is the upstream dispatch deadlock, not consensus logic.
+
+---
+
+# bug-1282 FIXED + bug-1283 isolated (the round-4 stall had TWO causes)
+
+Implemented the bug-1282 fix (commit 25e741d): promoted `endpoint_t.busy` to `_Atomic int` and
+made `endpoint_block_tasks`/`endpoint_unblock_tasks`/`endpoint_is_task_blocked` lockless, removing
+the `ep->lock` acquire from the vtask-eligibility path that ran under the dataq `q->lock`.
+
+## Verified
+- **Deadlock gone:** post-fix gdb soak shows ZERO `__lll_lock_wait` threads across all 3 nodes and
+  `endpoint_is_task_blocked` no longer appears in any lock path. The ABBA inversion is eliminated.
+- **No regression:** `controller_unit_test` all green (incl. `test_ha_core_safety` 20/0); libxce +
+  controller + unit test build clean.
+
+## But the round-4 stall had TWO independent causes
+A plain release soak WITH the fix STILL segfaults a restarted standby at round 4 (ctrl-3,
+"Segmentation fault (core dumped)"). **Deadlocks hang; they do not SIGSEGV** — so this crash is a
+SEPARATE memory bug (now bug-1283), not bug-1282. It is most likely an endpoint-lifetime UAF under
+chaotic restart (bug-1197 family): the dispatch eligibility path reads `vdata->u.iodata.ep` /
+endpoint state while io_worker tears down or mutates the same endpoint. It is a heisenbug —
+invisible under ASAN and under gdb (gdb perturbs it into a cold-start timeout / no crash) — and the
+sandbox has no usable core (read-only `core_pattern` → non-running systemd-coredump), so a
+backtrace needs a non-ASAN DEBUG (-O0 -g) build under gdb or a host with a live coredump pipeline.
+
+## Net for 7.2
+The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wedge). The remaining
+blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint
+quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and
+elections + job safety hold for every round that completes.

.kiro/specs/raft-ha-core-safety/tasks.md

@@ -154,7 +154,7 @@
   - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
   - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
   - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
-  - _Blocked on UPSTREAM (bug-1282), NOT consensus. ROOT-CAUSED (task 5.4, via gdb): the round-N stall is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (ep->lock → wants q->lock via dataq_enq) vs `vdispatcher` `dataq_deq` (q->lock → wants ep->lock via the `vtask_eligible`→`endpoint_is_task_blocked` match callback). io_worker wedges → the node stops accepting/answering PreVotes → two restarted survivors can't elect. The release SIGSEGV is the same ep/q race corrupting under churn. This is the raft-drive-async / core libxce async-io DISPATCH layer (a Non-Goal here) → fix routed to raft-drive-async (bug-1282). Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes (2-round soak PASSES; rounds 1–3 elect; committed prefix byte-identical). Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"ROOT CAUSE NAILED". Re-run 7.2 after bug-1282 lands upstream._
+  - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1283 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — STILL blocks the gate; likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family), heisenbug invisible under ASAN+gdb, needs a non-ASAN DEBUG+core+gdb env. Likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1282 FIXED + bug-1283 isolated". Re-run 7.2 after bug-1283 is fixed._
 
 - [ ] 7.3 Hypothesis-validation closeout
   - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure

.wolf/buglog.json

@@ -21221,7 +21221,7 @@
       "error_message": "ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.",
       "file": "src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)",
       "root_cause": "dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.",
-      "fix": "NOT FIXED. UPSTREAM raft-drive-async / core libxce async-io dispatch (out of raft-ha-core-safety scope). Minimal: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order. Blocks raft-ha-core-safety 7.2 acceptance gate (and thus the raft-drive-async dispatch flag-flip).",
+      "fix": "FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.",
       "tags": [
         "deadlock",
         "lock-order",
@@ -21247,7 +21247,7 @@
       "error_message": "Function not marked async",
       "file": ".kiro/specs/raft-ha-core-safety/tasks.md",
       "root_cause": "Function uses await but wasn't declared async",
-      "fix": "Added async modifier",
+      "fix": "FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.",
       "tags": [
         "auto-detected",
         "async-fix",
@@ -21272,6 +21272,38 @@
       "related_bugs": [],
       "occurrences": 1,
       "last_seen": "2026-06-30T04:17:58.579Z"
+    },
+    {
+      "id": "bug-1284",
+      "timestamp": "2026-06-30T07:56:51.924Z",
+      "error_message": "Significant refactor of ",
+      "file": "src/lib/netio.c",
+      "root_cause": "10 lines replaced/restructured",
+      "fix": "Rewrote 30→31 lines (10 removed)",
+      "tags": [
+        "auto-detected",
+        "refactor",
+        "c"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-06-30T07:56:51.924Z"
+    },
+    {
+      "id": "bug-1285",
+      "timestamp": "2026-06-30T08:24:48.860Z",
+      "error_message": "Incorrect value in code",
+      "file": ".kiro/specs/raft-ha-core-safety/tasks.md",
+      "root_cause": "Had `vtask_eligible`",
+      "fix": "Changed to `endpoint_is_task_blocked`",
+      "tags": [
+        "auto-detected",
+        "wrong-value",
+        "md"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-06-30T08:24:48.860Z"
     }
   ]
 }