Using gdb (SIGQUIT all-thread backtrace on the wedged nodes), nailed the round-4
blocker to a deterministic ABBA lock-order deadlock in src/lib async dispatch —
identical on both wedged survivors:
- io_worker (netio.c:672): holds ep->lock, blocks on q->lock (vtask_enq->dataq_enq)
- vdispatcher dataq_deq (dataq.c:65): holds q->lock, blocks on ep->lock via the
vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) cb
io_worker wedges -> node stops accepting/answering PreVotes -> two restarted
survivors can't elect -> round-4 stall. The release SIGSEGV is the same ep/q race
corrupting under churn.
Defect: dataq_deq runs its match callback while holding q->lock, and the callback
takes ep->lock — inverting io_worker's ep->lock->q->lock order. Minimal fix: make
ep->busy atomic + lockless in endpoint_is_task_blocked (or io_worker drops ep->lock
before dataq_enq, or one global lock order).
Ownership: src/lib async-I/O dispatch (vtaskmgr/dataq/netio) = raft-drive-async /
core libxce, a Non-Goal of this spec. Per upstream-ownership protocol, routed to
raft-drive-async (bug-1282); 7.2 stays blocked until the dispatch lock-order is
repaired, then re-run the gate. The gate worked as designed — 7.2 must be GREEN
before raft-drive-async flips dispatch flags ON, and it caught a real dispatch
deadlock. Task 5.4 (root-cause + assign ownership) is DONE. Consensus correctness
(Phase-2/5.2/5.3) is unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Rongsong Shen
bfe69b1e85