docs(raft-ha-core-safety): root-cause 7.2 round-N blocker to an ABBA dispatch deadlock; route to raft-drive-async (bug-1282)
Using gdb (SIGQUIT all-thread backtrace on the wedged nodes), nailed the round-4 blocker to a deterministic ABBA lock-order deadlock in src/lib async dispatch — identical on both wedged survivors: - io_worker (netio.c:672): holds ep->lock, blocks on q->lock (vtask_enq->dataq_enq) - vdispatcher dataq_deq (dataq.c:65): holds q->lock, blocks on ep->lock via the vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) cb io_worker wedges -> node stops accepting/answering PreVotes -> two restarted survivors can't elect -> round-4 stall. The release SIGSEGV is the same ep/q race corrupting under churn. Defect: dataq_deq runs its match callback while holding q->lock, and the callback takes ep->lock — inverting io_worker's ep->lock->q->lock order. Minimal fix: make ep->busy atomic + lockless in endpoint_is_task_blocked (or io_worker drops ep->lock before dataq_enq, or one global lock order). Ownership: src/lib async-I/O dispatch (vtaskmgr/dataq/netio) = raft-drive-async / core libxce, a Non-Goal of this spec. Per upstream-ownership protocol, routed to raft-drive-async (bug-1282); 7.2 stays blocked until the dispatch lock-order is repaired, then re-run the gate. The gate worked as designed — 7.2 must be GREEN before raft-drive-async flips dispatch flags ON, and it caught a real dispatch deadlock. Task 5.4 (root-cause + assign ownership) is DONE. Consensus correctness (Phase-2/5.2/5.3) is unaffected. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
parent
e80abd5cc4
commit
bfe69b1e85
3 changed files with 105 additions and 8 deletions
|
|
@ -21142,8 +21142,8 @@
|
|||
"timestamp": "2026-06-30T03:25:00Z",
|
||||
"error_message": "Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.",
|
||||
"file": "src/controller (restart/recovery/election path — exact site unknown, no backtrace)",
|
||||
"root_cause": "UNIFIED via gdb: one restart-time I/O-subsystem corruption manifesting as a SIGSEGV (release) OR an accept-loop hang (under gdb, which perturbs the memory bug away from a crash). NOT a consensus-logic gap — PreVote/eligibility/freshness (incl. 5.3) are correct; the node is I/O-dead after restart. Predates this session (baseline 21e26a1 failed round 3-4). Heisenbug: invisible under ASAN AND gdb; no usable core (read-only core_pattern -> non-running systemd-coredump).",
|
||||
"fix": "NOT FIXED (task 5.4). Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipeline. 5.4 must also rule out 5.2/5.3 (they sit in the restart/failover path). Diagnostic enablers landed: per-node stderr capture (75243fb) + the gdb-wrapper technique.",
|
||||
"root_cause": "ROOT-CAUSED via gdb SIGQUIT all-thread bt: an ABBA lock-order deadlock in src/lib async dispatch. io_worker (netio.c:672) holds ep->lock then blocks on q->lock (dataq_enq); vdispatcher dataq_deq (dataq.c:65) holds q->lock then blocks on ep->lock via the vtask_eligible->endpoint_is_task_blocked (netio.c:1001) match callback. io_worker dies -> node stops accepting/answering PreVotes -> round-4 stall. In release the same ep/q race corrupts -> SIGSEGV. See bug-1282 (the actual dispatch defect).",
|
||||
"fix": "The crash/hang IS bug-1282 (UPSTREAM raft-drive-async dispatch deadlock). raft-ha-core-safety 7.2 stays blocked until bug-1282 is fixed upstream, then re-run the gate. No raft-ha-core-safety code change.",
|
||||
"tags": [
|
||||
"7.2",
|
||||
"iomgr",
|
||||
|
|
@ -21161,6 +21161,7 @@
|
|||
],
|
||||
"related_bugs": [
|
||||
"bug-1272",
|
||||
"bug-1282",
|
||||
"bug-924"
|
||||
],
|
||||
"occurrences": 3,
|
||||
|
|
@ -21213,6 +21214,64 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T03:57:32.785Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1282",
|
||||
"timestamp": "2026-06-30T04:20:00Z",
|
||||
"error_message": "ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.",
|
||||
"file": "src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)",
|
||||
"root_cause": "dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.",
|
||||
"fix": "NOT FIXED. UPSTREAM raft-drive-async / core libxce async-io dispatch (out of raft-ha-core-safety scope). Minimal: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order. Blocks raft-ha-core-safety 7.2 acceptance gate (and thus the raft-drive-async dispatch flag-flip).",
|
||||
"tags": [
|
||||
"deadlock",
|
||||
"lock-order",
|
||||
"abba",
|
||||
"vtaskmgr",
|
||||
"dataq",
|
||||
"netio",
|
||||
"io_worker",
|
||||
"vdispatcher",
|
||||
"raft-drive-async",
|
||||
"async-io",
|
||||
"upstream"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1281"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-06-30T04:20:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1282",
|
||||
"timestamp": "2026-06-30T04:17:08.329Z",
|
||||
"error_message": "Function not marked async",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Function uses await but wasn't declared async",
|
||||
"fix": "Added async modifier",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"async-fix",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T04:17:08.329Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1283",
|
||||
"timestamp": "2026-06-30T04:17:58.579Z",
|
||||
"error_message": "Incorrect value in code",
|
||||
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
|
||||
"root_cause": "Had `xce_controller`",
|
||||
"fix": "Changed to `src/lib`",
|
||||
"tags": [
|
||||
"auto-detected",
|
||||
"wrong-value",
|
||||
"md"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T04:17:58.579Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue