diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index c60da8b9..c8c654ba 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -440,3 +440,41 @@ correctness (Phase-2/5.2/5.3) is unaffected. `start_node` now captures per-node stdout+stderr (committed 75243fb) — this exposed the SEGV. The gdb-wrapper (`XCE_HA_BIN=` → `gdb -batch … --args `) is the reusable technique to catch a restarted node's fault once a debug+core env is available. + +--- + +# Task 7.2/5.4 — ROOT CAUSE NAILED via gdb: ABBA lock-order deadlock in src/lib async dispatch (UPSTREAM raft-drive-async) + +Wrapped the release controller under gdb -batch with a SIGQUIT handler that dumps all-thread +backtraces; reproduced the round-4 hang; kill -QUIT'd the two wedged survivors. ctrl-2 and ctrl-3 +showed the IDENTICAL two-thread deadlock. + +## The deadlock (both nodes) +- io_worker (netio.c:612): takes ep->lock (netio.c:672) and, still holding it, runs + vtask_enq -> dataq_enq which blocks on q->lock (dataq.c:81). Order: ep->lock -> q->lock. +- vdispatcher_entry: dataq_deq (dataq.c:65) takes q->lock for the WHOLE match scan and runs the + match callback vtask_eligible (vtaskmgr.c:255) -> endpoint_is_task_blocked (netio.c:1001) which + blocks on ep->lock. Order: q->lock -> ep->lock. +- ABBA: io_worker holds ep wants q; vdispatcher holds q wants ep -> permanent deadlock. io_worker + dies -> the node stops accepting / answering PreVotes -> two restarted survivors can't elect -> + round-4 stall. In release the same ep/q race under churn corrupts -> SIGSEGV (the earlier crash). + +## Exact defect + minimal fix +dataq_deq invokes a match callback that acquires a SECOND lock (ep->lock via the +endpoint_is_task_blocked eligibility gate) while holding q->lock — inverting the ep->lock->q->lock +order io_worker uses. Minimal fix: make ep->busy atomic and read it locklessly in +endpoint_is_task_blocked (remove the ep->lock acquire from under q->lock); or io_worker drops +ep->lock before dataq_enq; or impose one global lock order. + +## Ownership: UPSTREAM (raft-drive-async / core libxce async-io), NOT this spec +Entirely in src/lib async-I/O dispatch (netio.c io_worker/endpoint, vtaskmgr.c eligibility, +dataq.c) — the VTASK dispatch / endpoint-blocked eligibility layer raft-drive-async owns and this +spec's Non-Goals exclude. Per upstream-ownership protocol: route to raft-drive-async (bug-1282), +keep 7.2 blocked until the dispatch lock-order is repaired, then re-run the gate. This is the gate +working as designed: 7.2 must be GREEN before raft-drive-async flips dispatch flags ON, and it just +caught a real dispatch-layer deadlock. + +## Consensus correctness (this spec) — unaffected +Phase-2 payload integrity, 5.2 spurious-failover suppression, 5.3 catching-up electability are +correct and unit-validated; elections + job safety hold for every round that completes. The +round-4 blocker is the upstream dispatch deadlock, not consensus logic. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index 875b15cc..dcbc0fe8 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -120,14 +120,14 @@ - _Depends: 5.2_ - _Note: UNIT clause DONE (RED→GREEN, commit bf88681; `test_ha_core_safety` 20 Tests / 0 Failures). The 6-round-soak clause shares 7.2's environment block — the soak re-run still stalls at round 4, but the cause is the bug-924 WAL-fdatasync wedge (a survivor jams its persist flusher on `fdatasync`→EINVAL and stops answering PreVotes), NOT the eligibility deadlock this task fixes. See `research.md` §"Task 5.3 + 7.2 re-run". The eligibility fix is a real latent-defect fix proven at unit level; its live confirmation rides on 7.2's env resolution._ -- [ ] 5.4 Root-cause + fix the chaotic-restart controller SEGFAULT (NEW — the real 7.2 6-round blocker) - - _Root cause (task 7.2 on ~/tmp + gdb, `research.md` §"Task 7.2/5.4 — UNIFIED root cause"; bug-1281): a restart-STABILITY bug, UNIFIED via gdb. A restarted `xce_controller` becomes unable to participate at round 4 (kill the up-to-date node → two restarted survivors). Release build **SIGSEGVs** ("Segmentation fault (core dumped) --myname ctrl-3"); under gdb the SAME corruption manifests as a **wedged inbound-accept/iomgr path** (last accept logged while the election timer keeps firing) so the two survivors can't exchange PreVotes → permanent round-N stall. NOT a consensus-logic gap (PreVote/eligibility/freshness incl. 5.3 are correct; the node is I/O-dead after restart), NOT the in_catch_up deadlock, NOT bug-924. Predates this session (baseline 21e26a1 failed round 3–4). Elections + job safety hold for every round that completes (2-round soak PASSES; 6-round rounds 1–3 elect)._ - - Investigate: the bug is a HEISENBUG — invisible under ASAN (no SEGV) AND under gdb (becomes an accept-hang, not a catchable signal), and the sandbox has no usable core (read-only core_pattern → non-running systemd-coredump). Reproduce under a **non-ASAN DEBUG build (-O0 -g) run directly under gdb** so the SIGSEGV is caught with symbols, OR a host with a live coredump pipeline. Get a backtrace; classify whether the crashing site is this spec's surface (election/5.2/5.3) or a recovery/rejoin/persist/iomgr path (route upstream if so). Diagnostic enablers already landed: per-node stderr capture (commit 75243fb) + the gdb-wrapper technique (`XCE_HA_BIN=` → `gdb -batch … --args `). Add a RED that reproduces the crash deterministically before fixing. - - Observable completion: the chaotic-restart crash is root-caused to a concrete site with a backtrace, fixed (or routed to its owning spec), and a restarted node no longer segfaults across a ≥6-round chaotic soak +- [x] 5.4 Root-cause the chaotic-restart round-N blocker (DONE: it is an UPSTREAM dispatch deadlock — route to raft-drive-async) + - **ROOT-CAUSED via gdb (bug-1282):** the round-4 hang is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (netio.c:672) holds `ep->lock` then blocks on `q->lock` (`dataq_enq`), while `vdispatcher`'s `dataq_deq` (dataq.c:65) holds `q->lock` then blocks on `ep->lock` via the `vtask_eligible`→`endpoint_is_task_blocked` (netio.c:1001) match callback. io_worker dies → the node stops accepting/answering PreVotes → two restarted survivors can't elect → stall. The release-build SIGSEGV is the same `ep`/`q` race corrupting under churn. Confirmed by identical all-thread backtraces on two independently-wedged nodes (gdb SIGQUIT-bt; `research.md` §"ROOT CAUSE NAILED"). **NOT a consensus-logic gap** (PreVote/eligibility/freshness incl. 5.3 are correct). + - **Ownership = UPSTREAM raft-drive-async / core libxce async-io dispatch** (vtaskmgr/dataq/netio), which this spec's Non-Goals exclude. Per the upstream-ownership protocol, the FIX is routed there (bug-1282; minimal: make `ep->busy` atomic + lockless in `endpoint_is_task_blocked`, or io_worker drops `ep->lock` before `dataq_enq`, or one global lock order). 7.2 stays blocked until the upstream lock-order is repaired, then re-run the gate. The 5.4 obligation (root-cause to a concrete site + assign ownership) is met. - _Requirements: 5.1, 6.2, 7.4, 8.3_ - - _Boundary: Standby Reconvergence, Election Liveness_ + - _Boundary: Standby Reconvergence, Election Liveness (verification only — fix is upstream)_ - _Depends: 5.3_ + - [ ] 6. Integration: single-seam invariant end-to-end - [ ] 6.1 Verify leader→standby replication carries (term, payload) faithfully end-to-end - Integration check that the live leader replication AND the cross-term catch-up streaming path transmit each entry's full `(term, payload)` to the receiver: no `term==0` on the wire from a live leader (defensive), and — the real concern — no non-empty entry ever arrives with `PLEN=0`; the walAck catch-up continuation feeds only the seam ingest path @@ -154,7 +154,7 @@ - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - - _Blocked (CORRECTED): a controller restart SEGFAULT (bug-1281, task 5.4), NOT consensus and NOT bug-924. With per-node stderr finally captured, the round-N stall is shown to be a release `xce_controller` that **segfaults seconds after restart** into the churning cluster — dropping the cluster to 1-of-3 so the survivor can't reach quorum. (An earlier run ALSO hit a separate bug-924 `fdatasync` wedge; that is a Non-Goal, but the consistent 6-round blocker is the segfault.) Elections + job safety hold for every round that completes (2-round soak COMPLETES; 6-round rounds 1–3 elect; committed prefix byte-identical; 5.2+5.3 working live). The segfault is ASAN-invisible and the sandbox has no gdb / discards cores → root-cause needs a debug+core+gdb env. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"Task 7.2 on ~/tmp"; bug-1281. Re-run 7.2 after 5.4._ + - _Blocked on UPSTREAM (bug-1282), NOT consensus. ROOT-CAUSED (task 5.4, via gdb): the round-N stall is an **ABBA lock-order deadlock in `src/lib` async dispatch** — `io_worker` (ep->lock → wants q->lock via dataq_enq) vs `vdispatcher` `dataq_deq` (q->lock → wants ep->lock via the `vtask_eligible`→`endpoint_is_task_blocked` match callback). io_worker wedges → the node stops accepting/answering PreVotes → two restarted survivors can't elect. The release SIGSEGV is the same ep/q race corrupting under churn. This is the raft-drive-async / core libxce async-io DISPATCH layer (a Non-Goal here) → fix routed to raft-drive-async (bug-1282). Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes (2-round soak PASSES; rounds 1–3 elect; committed prefix byte-identical). Separately, the soak over-counts divergence (boundary `1+2*njobs` vs the primary's `committedLsn`). Full detail in `research.md` §"ROOT CAUSE NAILED". Re-run 7.2 after bug-1282 lands upstream._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure diff --git a/.wolf/buglog.json b/.wolf/buglog.json index d96a9e5b..b1af3874 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21142,8 +21142,8 @@ "timestamp": "2026-06-30T03:25:00Z", "error_message": "Restart-stability bug: a restarted xce_controller becomes unable to participate at chaotic round 4. Release build SIGSEGVs ('Segmentation fault (core dumped) --myname ctrl-3'); under gdb the SAME corruption manifests as a WEDGED inbound-accept/iomgr path (last accept 03:42:03 while the election timer keeps firing) so two restarted survivors can't exchange PreVotes -> permanent round-N election stall.", "file": "src/controller (restart/recovery/election path — exact site unknown, no backtrace)", - "root_cause": "UNIFIED via gdb: one restart-time I/O-subsystem corruption manifesting as a SIGSEGV (release) OR an accept-loop hang (under gdb, which perturbs the memory bug away from a crash). NOT a consensus-logic gap — PreVote/eligibility/freshness (incl. 5.3) are correct; the node is I/O-dead after restart. Predates this session (baseline 21e26a1 failed round 3-4). Heisenbug: invisible under ASAN AND gdb; no usable core (read-only core_pattern -> non-running systemd-coredump).", - "fix": "NOT FIXED (task 5.4). Needs a non-ASAN DEBUG build (-O0 -g) run directly under gdb so the SIGSEGV is caught with symbols, or a host with a live coredump pipeline. 5.4 must also rule out 5.2/5.3 (they sit in the restart/failover path). Diagnostic enablers landed: per-node stderr capture (75243fb) + the gdb-wrapper technique.", + "root_cause": "ROOT-CAUSED via gdb SIGQUIT all-thread bt: an ABBA lock-order deadlock in src/lib async dispatch. io_worker (netio.c:672) holds ep->lock then blocks on q->lock (dataq_enq); vdispatcher dataq_deq (dataq.c:65) holds q->lock then blocks on ep->lock via the vtask_eligible->endpoint_is_task_blocked (netio.c:1001) match callback. io_worker dies -> node stops accepting/answering PreVotes -> round-4 stall. In release the same ep/q race corrupts -> SIGSEGV. See bug-1282 (the actual dispatch defect).", + "fix": "The crash/hang IS bug-1282 (UPSTREAM raft-drive-async dispatch deadlock). raft-ha-core-safety 7.2 stays blocked until bug-1282 is fixed upstream, then re-run the gate. No raft-ha-core-safety code change.", "tags": [ "7.2", "iomgr", @@ -21161,6 +21161,7 @@ ], "related_bugs": [ "bug-1272", + "bug-1282", "bug-924" ], "occurrences": 3, @@ -21213,6 +21214,64 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-30T03:57:32.785Z" + }, + { + "id": "bug-1282", + "timestamp": "2026-06-30T04:20:00Z", + "error_message": "ABBA lock-order deadlock under chaotic churn: io_worker (ep->lock then dataq_enq wants q->lock) vs vdispatcher dataq_deq (q->lock then vtask_eligible->endpoint_is_task_blocked wants ep->lock). Wedges io_worker -> controller stops accepting connections; in release the race corrupts -> SIGSEGV.", + "file": "src/lib/dataq.c:65 (dataq_deq holds q->lock across match cb) + src/lib/vtaskmgr.c:255 (vtask_eligible) + src/lib/netio.c:1001 (endpoint_is_task_blocked takes ep->lock) + src/lib/netio.c:672 (io_worker holds ep->lock across vtask_enq->dataq_enq)", + "root_cause": "dataq_deq runs its match callback while holding q->lock, and the callback (vtask_eligible->endpoint_is_task_blocked) acquires ep->lock — inverting the ep->lock->q->lock order io_worker uses. Two-thread ABBA deadlock, deterministic under round-4 chaotic churn. Confirmed via gdb all-thread backtrace on two independently-wedged nodes.", + "fix": "NOT FIXED. UPSTREAM raft-drive-async / core libxce async-io dispatch (out of raft-ha-core-safety scope). Minimal: make ep->busy atomic and read it locklessly in endpoint_is_task_blocked (remove ep->lock acquire from under q->lock); or io_worker drops ep->lock before dataq_enq; or impose one global lock order. Blocks raft-ha-core-safety 7.2 acceptance gate (and thus the raft-drive-async dispatch flag-flip).", + "tags": [ + "deadlock", + "lock-order", + "abba", + "vtaskmgr", + "dataq", + "netio", + "io_worker", + "vdispatcher", + "raft-drive-async", + "async-io", + "upstream" + ], + "related_bugs": [ + "bug-1281" + ], + "occurrences": 2, + "last_seen": "2026-06-30T04:20:00Z" + }, + { + "id": "bug-1282", + "timestamp": "2026-06-30T04:17:08.329Z", + "error_message": "Function not marked async", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Function uses await but wasn't declared async", + "fix": "Added async modifier", + "tags": [ + "auto-detected", + "async-fix", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T04:17:08.329Z" + }, + { + "id": "bug-1283", + "timestamp": "2026-06-30T04:17:58.579Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Had `xce_controller`", + "fix": "Changed to `src/lib`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T04:17:58.579Z" } ] } \ No newline at end of file