From e90bf835f9581db31c2f7bfea97e48d7fb7e4c19 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 30 Jun 2026 08:25:04 +0000 Subject: [PATCH] docs(raft-ha-core-safety): bug-1282 fixed + isolate bug-1283 (the round-4 stall had two causes) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bug-1282 (ABBA iomgr deadlock) is FIXED and verified (commit 25e741d): zero __lll_lock_wait threads post-fix, unit suite green. But a plain release soak WITH the fix still segfaults a restarted standby at round 4 — and deadlocks hang, they do not SIGSEGV, so that crash is a SEPARATE bug (bug-1283), most likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family). bug-1283 is a heisenbug (invisible under ASAN and gdb; no usable core in the sandbox) and likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. 7.2 stays blocked on it. Consensus correctness (Phase-2/5.2/5.3) is unaffected. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC --- .kiro/specs/raft-ha-core-safety/research.md | 30 +++++++++++++++++ .kiro/specs/raft-ha-core-safety/tasks.md | 2 +- .wolf/buglog.json | 36 +++++++++++++++++++-- 3 files changed, 65 insertions(+), 3 deletions(-) diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index c8c654ba..ce0a2a1b 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -478,3 +478,33 @@ caught a real dispatch-layer deadlock. Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are correct and unit-validated; elections + job safety hold for every round that completes. The round-4 blocker is the upstream dispatch deadlock, not consensus logic. + +--- + +# bug-1282 FIXED + bug-1283 isolated (the round-4 stall had TWO causes) + +Implemented the bug-1282 fix (commit 25e741d): promoted `endpoint_t.busy` to `_Atomic int` and +made `endpoint_block_tasks`/`endpoint_unblock_tasks`/`endpoint_is_task_blocked` lockless, removing +the `ep->lock` acquire from the vtask-eligibility path that ran under the dataq `q->lock`. + +## Verified +- **Deadlock gone:** post-fix gdb soak shows ZERO `__lll_lock_wait` threads across all 3 nodes and + `endpoint_is_task_blocked` no longer appears in any lock path. The ABBA inversion is eliminated. +- **No regression:** `controller_unit_test` all green (incl. `test_ha_core_safety` 20/0); libxce + + controller + unit test build clean. + +## But the round-4 stall had TWO independent causes +A plain release soak WITH the fix STILL segfaults a restarted standby at round 4 (ctrl-3, +"Segmentation fault (core dumped)"). **Deadlocks hang; they do not SIGSEGV** — so this crash is a +SEPARATE memory bug (now bug-1283), not bug-1282. It is most likely an endpoint-lifetime UAF under +chaotic restart (bug-1197 family): the dispatch eligibility path reads `vdata->u.iodata.ep` / +endpoint state while io_worker tears down or mutates the same endpoint. It is a heisenbug — +invisible under ASAN and under gdb (gdb perturbs it into a cold-start timeout / no crash) — and the +sandbox has no usable core (read-only `core_pattern` → non-running systemd-coredump), so a +backtrace needs a non-ASAN DEBUG (-O0 -g) build under gdb or a host with a live coredump pipeline. + +## Net for 7.2 +The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wedge). The remaining +blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint +quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and +elections + job safety hold for every round that completes. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index dcbc0fe8..d4377800 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -154,7 +154,7 @@ - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - - _Blocked on UPSTREAM (bug-1282), NOT consensus. ROOT-CAUSED (task 5.4, via gdb): the round-N stall is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (ep->lock → wants q->lock via dataq_enq) vs `vdispatcher` `dataq_deq` (q->lock → wants ep->lock via the `vtask_eligible`→`endpoint_is_task_blocked` match callback). io_worker wedges → the node stops accepting/answering PreVotes → two restarted survivors can't elect. The release SIGSEGV is the same ep/q race corrupting under churn. This is the raft-drive-async / core libxce async-io DISPATCH layer (a Non-Goal here) → fix routed to raft-drive-async (bug-1282). Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes (2-round soak PASSES; rounds 1–3 elect; committed prefix byte-identical). Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"ROOT CAUSE NAILED". Re-run 7.2 after bug-1282 lands upstream._ + - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1283 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — STILL blocks the gate; likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family), heisenbug invisible under ASAN+gdb, needs a non-ASAN DEBUG+core+gdb env. Likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1282 FIXED + bug-1283 isolated". Re-run 7.2 after bug-1283 is fixed._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure diff --git a/.wolf/buglog.json b/.wolf/buglog.json index b1af3874..f70c50e1 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21221,7 +21221,7 @@ "error_message": "ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.", "file": "src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)", "root_cause": "dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.", - "fix": "NOT FIXED. UPSTREAM raft-drive-async / core libxce async-io dispatch (out of raft-ha-core-safety scope). Minimal: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order. Blocks raft-ha-core-safety 7.2 acceptance gate (and thus the raft-drive-async dispatch flag-flip).", + "fix": "FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.", "tags": [ "deadlock", "lock-order", @@ -21247,7 +21247,7 @@ "error_message": "Function not marked async", "file": ".kiro/specs/raft-ha-core-safety/tasks.md", "root_cause": "Function uses await but wasn't declared async", - "fix": "Added async modifier", + "fix": "FIXED (commit 25e741d): promoted endpoint_t.busy from a 1-bit bitfield to _Atomic int; endpoint_block_tasks/endpoint_unblock_tasks/endpoint_is_task_blocked now lockless (release/acquire), removing the ep->lock acquire from the vtask-eligibility path that ran under q->lock. Verified: post-fix gdb soak has ZERO __lll_lock_wait threads + no endpoint_is_task_blocked in any lock path; controller_unit_test all green. Mirrors dispatch_inflight (bug-1197). NOTE: a SEPARATE release-only restart segfault (bug-1283) remains and still blocks 7.2.", "tags": [ "auto-detected", "async-fix", @@ -21272,6 +21272,38 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-30T04:17:58.579Z" + }, + { + "id": "bug-1284", + "timestamp": "2026-06-30T07:56:51.924Z", + "error_message": "Significant refactor of ", + "file": "src/lib/netio.c", + "root_cause": "10 lines replaced/restructured", + "fix": "Rewrote 30→31 lines (10 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T07:56:51.924Z" + }, + { + "id": "bug-1285", + "timestamp": "2026-06-30T08:24:48.860Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Had `vtask_eligible`", + "fix": "Changed to `endpoint_is_task_blocked`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T08:24:48.860Z" } ] } \ No newline at end of file