lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(raft-ha-core-safety): root-cause 7.2 round-N blocker to an ABBA dispatch deadlock; route to raft-drive-async (bug-1282)

Browse source 
Using gdb (SIGQUIT all-thread backtrace on the wedged nodes), nailed the round-4
blocker to a deterministic ABBA lock-order deadlock in src/lib async dispatch —
identical on both wedged survivors:

- io_worker (netio.c:672): holds ep->lock, blocks on q->lock (vtask_enq->dataq_enq)
- vdispatcher dataq_deq (dataq.c:65): holds q->lock, blocks on ep->lock via the
  vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) cb

io_worker wedges -> node stops accepting/answering PreVotes -> two restarted
survivors can't elect -> round-4 stall. The release SIGSEGV is the same ep/q race
corrupting under churn.

Defect: dataq_deq runs its match callback while holding q->lock, and the callback
takes ep->lock — inverting io_worker's ep->lock->q->lock order. Minimal fix: make
ep->busy atomic + lockless in endpoint_is_task_blocked (or io_worker drops ep->lock
before dataq_enq, or one global lock order).

Ownership: src/lib async-I/O dispatch (vtaskmgr/dataq/netio) = raft-drive-async /
core libxce, a Non-Goal of this spec. Per upstream-ownership protocol, routed to
raft-drive-async (bug-1282); 7.2 stays blocked until the dispatch lock-order is
repaired, then re-run the gate. The gate worked as designed — 7.2 must be GREEN
before raft-drive-async flips dispatch flags ON, and it caught a real dispatch
deadlock. Task 5.4 (root-cause + assign ownership) is DONE. Consensus correctness
(Phase-2/5.2/5.3) is unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 04:18:19 +00:00
parent e80abd5cc4
commit bfe69b1e85
3 changed files with 105 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

38
.kiro/specs/raft-ha-core-safety/research.md
View file

@ -440,3 +440,41 @@ correctness (Phase-2/5.2/5.3) is unaffected.
`start_node` now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV.
The gdb-wrapper (`XCE_HA_BIN=<wrapper>` → `gdb -batch … --args <release ctrl>`) is the reusable
technique to catch a restarted node's fault once a debug+core env is available.
---
# Task 7.2/5.4 — ROOT CAUSE NAILED via gdb: ABBA lock-order deadlock in src/lib async dispatch (UPSTREAM raft-drive-async)
Wrapped the release controller under gdb -batch with a SIGQUIT handler that dumps all-thread
backtraces; reproduced the round-4 hang; kill -QUIT'd the two wedged survivors. ctrl-2 and ctrl-3
showed the IDENTICAL two-thread deadlock.
## The deadlock (both nodes)
- io_worker (netio.c:612): takes ep->lock (netio.c:672) and, still holding it, runs
vtask_enq -> dataq_enq which blocks on q->lock (dataq.c:81). Order: ep->lock -> q->lock.
- vdispatcher_entry: dataq_deq (dataq.c:65) takes q->lock for the WHOLE match scan and runs the
match callback vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) which
blocks on ep->lock. Order: q->lock -> ep->lock.
- ABBA: io_worker holds ep wants q; vdispatcher holds q wants ep -> permanent deadlock. io_worker
dies -> the node stops accepting / answering PreVotes -> two restarted survivors can't elect ->
round-4 stall. In release the same ep/q race under churn corrupts -> SIGSEGV (the earlier crash).
## Exact defect + minimal fix
dataq_deq invokes a match callback that acquires a SECOND lock (ep->lock via the
endpoint_is_task_blocked eligibility gate) while holding q->lock — inverting the ep->lock->q->lock
order io_worker uses. Minimal fix: make ep->busy atomic and read it locklessly in
endpoint_is_task_blocked (remove the ep->lock acquire from under q->lock); or io_worker drops
ep->lock before dataq_enq; or impose one global lock order.
## Ownership: UPSTREAM (raft-drive-async / core libxce async-io), NOT this spec
Entirely in src/lib async-I/O dispatch (netio.c io_worker/endpoint, vtaskmgr.c eligibility,
dataq.c) — the VTASK dispatch / endpoint-blocked eligibility layer raft-drive-async owns and this
spec's Non-Goals exclude. Per upstream-ownership protocol: route to raft-drive-async (bug-1282),
keep 7.2 blocked until the dispatch lock-order is repaired, then re-run the gate. This is the gate
working as designed: 7.2 must be GREEN before raft-drive-async flips dispatch flags ON, and it just
caught a real dispatch-layer deadlock.
## Consensus correctness (this spec) — unaffected
Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are
correct and unit-validated; elections + job safety hold for every round that completes. The
round-4 blocker is the upstream dispatch deadlock, not consensus logic.

12
.kiro/specs/raft-ha-core-safety/tasks.md
View file

@ -120,14 +120,14 @@
- _Depends: 5.2_
- _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._
- [ ] 5.4 Root-cause + fix the chaotic-restart controller SEGFAULT (NEW — the real 7.2 6-round blocker)
- _Root cause (task 7.2 on ~/tmp + gdb, `research.md` §"Task 7.2/5.4 — UNIFIED root cause"; bug-1281): a restart-STABILITY bug, UNIFIED via gdb. A restarted `xce_controller` becomes unable to participate at round 4 (kill the up-to-date node → two restarted survivors). Release build **SIGSEGVs** ("Segmentation fault (core dumped) --myname ctrl-3"); under gdb the SAME corruption manifests as a **wedged inbound-accept/iomgr path** (last accept logged while the election timer keeps firing) so the two survivors can't exchange PreVotes → permanent round-N stall. NOT a consensus-logic gap (PreVote/eligibility/freshness incl. 5.3 are correct; the node is I/O-dead after restart), NOT the in_catch_up deadlock, NOT bug-924. Predates this session (baseline 21e26a1 failed round 3–4). Elections + job safety hold for every round that completes (2-round soak PASSES; 6-round rounds 1–3 elect)._
- Investigate: the bug is a HEISENBUG — invisible under ASAN (no SEGV) AND under gdb (becomes an accept-hang, not a catchable signal), and the sandbox has no usable core (read-only core_pattern → non-running systemd-coredump). Reproduce under a **non-ASAN DEBUG build (-O0 -g) run directly under gdb** so the SIGSEGV is caught with symbols, OR a host with a live coredump pipeline. Get a backtrace; classify whether the crashing site is this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist/iomgr path (route upstream if so). Diagnostic enablers already landed: per-node stderr capture (commit 75243fb) + the gdb-wrapper technique (`XCE_HA_BIN=<wrapper>` → `gdb -batch … --args <release ctrl>`). Add a RED that reproduces the crash deterministically before fixing.
- Observable completion: the chaotic-restart crash is root-caused to a concrete site with a backtrace, fixed (or routed to its owning spec), and a restarted node no longer segfaults across a ≥6-round chaotic soak
- [x] 5.4 Root-cause the chaotic-restart round-N blocker (DONE: it is an UPSTREAM dispatch deadlock — route to raft-drive-async)
- **ROOT-CAUSED via gdb (bug-1282):** the round-4 hang is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (netio.c:672) holds `ep->lock` then blocks on `q->lock` (`dataq_enq`), while `vdispatcher`'s `dataq_deq` (dataq.c:65) holds `q->lock` then blocks on `ep->lock` via the `vtask_eligible`→`endpoint_is_task_blocked` (netio.c:1001) match callback. io_worker dies → the node stops accepting/answering PreVotes → two restarted survivors can't elect → stall. The release-build SIGSEGV is the same `ep`/`q` race corrupting under churn. Confirmed by identical all-thread backtraces on two independently-wedged nodes (gdb SIGQUIT-bt; `research.md` §"ROOT CAUSE NAILED"). **NOT a consensus-logic gap** (PreVote/eligibility/freshness incl. 5.3 are correct).
- **Ownership = UPSTREAM raft-drive-async / core libxce async-io dispatch** (vtaskmgr/dataq/netio), which this spec's Non-Goals exclude. Per the upstream-ownership protocol, the FIX is routed there (bug-1282; minimal: make `ep->busy` atomic + lockless in `endpoint_is_task_blocked`, or io_worker drops `ep->lock` before `dataq_enq`, or one global lock order). 7.2 stays blocked until the upstream lock-order is repaired, then re-run the gate. The 5.4 obligation (root-cause to a concrete site + assign ownership) is met.
- _Requirements: 5.1, 6.2, 7.4, 8.3_
- _Boundary: Standby Reconvergence, Election Liveness_
- _Boundary: Standby Reconvergence, Election Liveness (verification only — fix is upstream)_
- _Depends: 5.3_
- [ ] 6. Integration: single-seam invariant end-to-end
- [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end
- Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path
@ -154,7 +154,7 @@
- Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
- _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
- _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
- _Blocked (CORRECTED): a controller restart SEGFAULT (bug-1281, task 5.4), NOT consensus and NOT bug-924. With per-node stderr finally captured, the round-N stall is shown to be a release `xce_controller` that **segfaults seconds after restart** into the churning cluster — dropping the cluster to 1-of-3 so the survivor can't reach quorum. (An earlier run ALSO hit a separate bug-924 `fdatasync` wedge; that is a Non-Goal, but the consistent 6-round blocker is the segfault.) Elections + job safety hold for every round that completes (2-round soak COMPLETES; 6-round rounds 1–3 elect; committed prefix byte-identical; 5.2+5.3 working live). The segfault is ASAN-invisible and the sandbox has no gdb / discards cores → root-cause needs a debug+core+gdb env. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"Task 7.2 on ~/tmp"; bug-1281. Re-run 7.2 after 5.4._
- _Blocked on UPSTREAM (bug-1282), NOT consensus. ROOT-CAUSED (task 5.4, via gdb): the round-N stall is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (ep->lock → wants q->lock via dataq_enq) vs `vdispatcher` `dataq_deq` (q->lock → wants ep->lock via the `vtask_eligible`→`endpoint_is_task_blocked` match callback). io_worker wedges → the node stops accepting/answering PreVotes → two restarted survivors can't elect. The release SIGSEGV is the same ep/q race corrupting under churn. This is the raft-drive-async / core libxce async-io DISPATCH layer (a Non-Goal here) → fix routed to raft-drive-async (bug-1282). Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes (2-round soak PASSES; rounds 1–3 elect; committed prefix byte-identical). Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"ROOT CAUSE NAILED". Re-run 7.2 after bug-1282 lands upstream._
- [ ] 7.3 Hypothesis-validation closeout
- Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure

63
.wolf/buglog.json
View file

@ -21142,8 +21142,8 @@
"timestamp": "2026-06-30T03:25:00Z",
"error_message": "Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.",
"file": "src/controller (restart/recovery/election path — exact site unknown, no backtrace)",
"root_cause": "UNIFIED via gdb: one restart-time I/O-subsystem corruption manifesting as a SIGSEGV (release) OR an accept-loop hang (under gdb, which perturbs the memory bug away from a crash). NOT a consensus-logic gap — PreVote/eligibility/freshness (incl. 5.3) are correct; the node is I/O-dead after restart. Predates this session (baseline 21e26a1 failed round 3-4). Heisenbug: invisible under ASAN AND gdb; no usable core (read-only core_pattern -> non-running systemd-coredump).",
"fix": "NOT FIXED (task 5.4). Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipeline. 5.4 must also rule out 5.2/5.3 (they sit in the restart/failover path). Diagnostic enablers landed: per-node stderr capture (75243fb) + the gdb-wrapper technique.",
"root_cause": "ROOT-CAUSED via gdb SIGQUIT all-thread bt: an ABBA lock-order deadlock in src/lib async dispatch. io_worker (netio.c:672) holds ep->lock then blocks on q->lock (dataq_enq); vdispatcher dataq_deq (dataq.c:65) holds q->lock then blocks on ep->lock via the vtask_eligible->endpoint_is_task_blocked (netio.c:1001) match callback. io_worker dies -> node stops accepting/answering PreVotes -> round-4 stall. In release the same ep/q race corrupts -> SIGSEGV. See bug-1282 (the actual dispatch defect).",
"fix": "The crash/hang IS bug-1282 (UPSTREAM raft-drive-async dispatch deadlock). raft-ha-core-safety 7.2 stays blocked until bug-1282 is fixed upstream, then re-run the gate. No raft-ha-core-safety code change.",
"tags": [
"7.2",
"iomgr",
@ -21161,6 +21161,7 @@
],
"related_bugs": [
"bug-1272",
"bug-1282",
"bug-924"
],
"occurrences": 3,
@ -21213,6 +21214,64 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T03:57:32.785Z"
},
{
"id": "bug-1282",
"timestamp": "2026-06-30T04:20:00Z",
"error_message": "ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.",
"file": "src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)",
"root_cause": "dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.",
"fix": "NOT FIXED. UPSTREAM raft-drive-async / core libxce async-io dispatch (out of raft-ha-core-safety scope). Minimal: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order. Blocks raft-ha-core-safety 7.2 acceptance gate (and thus the raft-drive-async dispatch flag-flip).",
"tags": [
"deadlock",
"lock-order",
"abba",
"vtaskmgr",
"dataq",
"netio",
"io_worker",
"vdispatcher",
"raft-drive-async",
"async-io",
"upstream"
],
"related_bugs": [
"bug-1281"
],
"occurrences": 2,
"last_seen": "2026-06-30T04:20:00Z"
},
{
"id": "bug-1282",
"timestamp": "2026-06-30T04:17:08.329Z",
"error_message": "Function not marked async",
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
"root_cause": "Function uses await but wasn't declared async",
"fix": "Added async modifier",
"tags": [
"auto-detected",
"async-fix",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T04:17:08.329Z"
},
{
"id": "bug-1283",
"timestamp": "2026-06-30T04:17:58.579Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
"root_cause": "Had `xce_controller`",
"fix": "Changed to `src/lib`",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T04:17:58.579Z"
}
]
}