lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(raft-ha-core-safety): bug-1282 fixed + isolate bug-1283 (the round-4 stall had two causes)

Browse source 
bug-1282 (ABBA iomgr deadlock) is FIXED and verified (commit 25e741d): zero
__lll_lock_wait threads post-fix, unit suite green. But a plain release soak WITH
the fix still segfaults a restarted standby at round 4 — and deadlocks hang, they
do not SIGSEGV, so that crash is a SEPARATE bug (bug-1283), most likely an
endpoint-lifetime UAF under chaotic restart (bug-1197 family).

bug-1283 is a heisenbug (invisible under ASAN and gdb; no usable core in the
sandbox) and likely UPSTREAM raft-drive-async / core libxce endpoint quiescence.
7.2 stays blocked on it. Consensus correctness (Phase-2/5.2/5.3) is unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 08:25:04 +00:00
parent 25e741d8c7
commit e90bf835f9
3 changed files with 65 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

36
.wolf/buglog.json
View file

@ -21221,7 +21221,7 @@
"error_message": "ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.",
"file": "src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)",
"root_cause": "dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.",
"fix": "NOT FIXED. UPSTREAM raft-drive-async / core libxce async-io dispatch (out of raft-ha-core-safety scope). Minimal: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order. Blocks raft-ha-core-safety 7.2 acceptance gate (and thus the raft-drive-async dispatch flag-flip).",
"fix": "FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.",
"tags": [
"deadlock",
"lock-order",
@ -21247,7 +21247,7 @@
"error_message": "Function not marked async",
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
"root_cause": "Function uses await but wasn't declared async",
"fix": "Added async modifier",
"fix": "FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.",
"tags": [
"auto-detected",
"async-fix",
@ -21272,6 +21272,38 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T04:17:58.579Z"
},
{
"id": "bug-1284",
"timestamp": "2026-06-30T07:56:51.924Z",
"error_message": "Significant refactor of ",
"file": "src/lib/netio.c",
"root_cause": "10 lines replaced/restructured",
"fix": "Rewrote 30→31 lines (10 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T07:56:51.924Z"
},
{
"id": "bug-1285",
"timestamp": "2026-06-30T08:24:48.860Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
"root_cause": "Had `vtask_eligible`",
"fix": "Changed to `endpoint_is_task_blocked`",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T08:24:48.860Z"
}
]
}