feat(xce-build): add non-ASAN debug + release profiles; root-cause bug-1286 with the debug build
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build, backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer -> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses its own default build dir so variants don't clobber each other; arg2 may still be a legacy build_dir. asan runs the unit suites; debug/release are build-only (the suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924) and print how to run a suite directly. Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces them AND yields a symbolized backtrace under gdb. It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4 segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining blocker (bug-1286, root-caused); bug-1282 deadlock is fixed. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
parent
e90bf835f9
commit
02e9275706
5 changed files with 208 additions and 25 deletions
|
|
@ -508,3 +508,32 @@ The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wed
|
|||
blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint
|
||||
quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and
|
||||
elections + job safety hold for every round that completes.
|
||||
|
||||
---
|
||||
|
||||
# bug-1286 (the round-4 segfault) ROOT-CAUSED — NULL ep->owner — via the new /xce-build debug profile
|
||||
|
||||
The `/xce-build ... debug` profile added this session (non-ASAN `-O0 -g3`, no sanitizer → `build.dbg`)
|
||||
reproduced the heisenbug AND gave a symbolized backtrace under gdb (ASAN and gdb-on-release both
|
||||
perturbed it away). The captured SIGSEGV:
|
||||
|
||||
```
|
||||
Thread received SIGSEGV
|
||||
#0 ha_dispatch_gate (ha=0x3ff0, opcode=37) controller/ha.c:3194 (atomic_load &ha->role)
|
||||
#1 myreq_handler controller/cluster.c:251
|
||||
#2 vtask_entry controller/cluster.c:693
|
||||
#3 run_task / vcpu_entry lib/vtaskmgr.c
|
||||
```
|
||||
|
||||
Root cause: `myreq_handler` does `cluster_t* c = (cluster_t*)ep->owner;` then
|
||||
`ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a
|
||||
`walCatchupEntry` (op 37) vtask while `ep->owner == NULL`, so `c == NULL` and
|
||||
`&c->ha == NULL + offsetof(cluster_t,ha) == 0x3ff0` — a garbage pointer `ha_dispatch_gate`
|
||||
dereferences → SIGSEGV. (DISTINCT from the bug-1282 deadlock, which is fixed; deadlocks hang,
|
||||
they do not crash.) Fix: NULL-guard `c = ep->owner` in `myreq_handler` (drop/reject the request if
|
||||
the endpoint's owner is not yet attached) instead of dereferencing; deeper, ensure `ep->owner` is
|
||||
set at accept time before the endpoint is monitored for input and never cleared while a vtask is in
|
||||
flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bug-1283).
|
||||
|
||||
7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock)
|
||||
is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected.
|
||||
|
|
|
|||
|
|
@ -154,7 +154,7 @@
|
|||
- Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
|
||||
- _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
|
||||
- _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1283 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — STILL blocks the gate; likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family), heisenbug invisible under ASAN+gdb, needs a non-ASAN DEBUG+core+gdb env. Likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1282 FIXED + bug-1283 isolated". Re-run 7.2 after bug-1283 is fixed._
|
||||
- _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). STILL blocks the gate until that lands. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1286 … ROOT-CAUSED". Re-run 7.2 after bug-1286 is fixed._
|
||||
|
||||
- [ ] 7.3 Hypothesis-validation closeout
|
||||
- Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue