lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(xce-build): add non-ASAN debug + release profiles; root-cause bug-1286 with the debug build

Browse source 
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.

Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.

It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
This commit is contained in:
Rongsong Shen 2026-06-30 09:15:36 +00:00
parent e90bf835f9
commit 02e9275706
5 changed files with 208 additions and 25 deletions
Download patch file Download diff file Expand all files Collapse all files

15
.claude/skills/xce-build/SKILL.md
View file

@ -1,6 +1,6 @@
---
name: xce-build
description: "Build XCE components with tests. Usage: /xce-build [component] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli"
description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
context: fork
disable-model-invocation: false
allowed-tools: Bash(*)
@ -8,10 +8,19 @@ allowed-tools: Bash(*)
## build xce source code
Args: `[component] [profile|build_dir] [build_dir]`
- **component** — `all` (default), `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`
- **profile** — `asan` (default: debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. Arg 2 may also be a legacy explicit build_dir (anything that isn't a profile keyword); arg 3 overrides the build dir.
- Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked).
Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`.
```bash
COMPONENT="$0"
BUILD_DIR="$1"
PROFILE="$1"
BUILD_DIR="$2"
cd src && \
nix develop --command bash tools/xce-build.sh ${COMPONENT:+$COMPONENT} ${BUILD_DIR:+$BUILD_DIR}
nix develop --command bash tools/xce-build.sh ${COMPONENT:+$COMPONENT} ${PROFILE:+$PROFILE} ${BUILD_DIR:+$BUILD_DIR}
```

29
.kiro/specs/raft-ha-core-safety/research.md
View file

@ -508,3 +508,32 @@ The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wed
blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint
quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and
elections + job safety hold for every round that completes.
---
# bug-1286 (the round-4 segfault) ROOT-CAUSED — NULL ep->owner — via the new /xce-build debug profile
The `/xce-build ... debug` profile added this session (non-ASAN `-O0 -g3`, no sanitizer → `build.dbg`)
reproduced the heisenbug AND gave a symbolized backtrace under gdb (ASAN and gdb-on-release both
perturbed it away). The captured SIGSEGV:
```
Thread received SIGSEGV
#0 ha_dispatch_gate (ha=0x3ff0, opcode=37) controller/ha.c:3194 (atomic_load &ha->role)
#1 myreq_handler controller/cluster.c:251
#2 vtask_entry controller/cluster.c:693
#3 run_task / vcpu_entry lib/vtaskmgr.c
```
Root cause: `myreq_handler` does `cluster_t* c = (cluster_t*)ep->owner;` then
`ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a
`walCatchupEntry` (op 37) vtask while `ep->owner == NULL`, so `c == NULL` and
`&c->ha == NULL + offsetof(cluster_t,ha) == 0x3ff0` — a garbage pointer `ha_dispatch_gate`
dereferences → SIGSEGV. (DISTINCT from the bug-1282 deadlock, which is fixed; deadlocks hang,
they do not crash.) Fix: NULL-guard `c = ep->owner` in `myreq_handler` (drop/reject the request if
the endpoint's owner is not yet attached) instead of dereferencing; deeper, ensure `ep->owner` is
set at accept time before the endpoint is monitored for input and never cleared while a vtask is in
flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bug-1283).
7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock)
is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected.

2
.kiro/specs/raft-ha-core-safety/tasks.md
View file

@ -154,7 +154,7 @@
- Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip
- _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_
- _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_
- _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1283 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — STILL blocks the gate; likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family), heisenbug invisible under ASAN+gdb, needs a non-ASAN DEBUG+core+gdb env. Likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1282 FIXED + bug-1283 isolated". Re-run 7.2 after bug-1283 is fixed._
- _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). STILL blocks the gate until that lands. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1286 … ROOT-CAUSED". Re-run 7.2 after bug-1286 is fixed._
- [ ] 7.3 Hypothesis-validation closeout
- Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure

90
.wolf/buglog.json
View file

@ -21304,6 +21304,96 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T08:24:48.860Z"
},
{
"id": "bug-1286",
"timestamp": "2026-06-30T05:30:00Z",
"error_message": "xce_controller SIGSEGV in ha_dispatch_gate during chaotic round-4 restart — Thread received SIGSEGV at ha.c:3194 (atomic_load &ha->role) with ha=0x3ff0 (garbage). The release-only crash that blocks raft-ha-core-safety 7.2; DISTINCT from the bug-1282 deadlock (which is fixed).",
"file": "src/controller/cluster.c:251 (myreq_handler) + src/controller/ha.c:3194 (ha_dispatch_gate)",
"root_cause": "NULL ep->owner. myreq_handler does `cluster_t* c = (cluster_t*)ep->owner;` then `ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a walCatchupEntry (op 37) vtask while ep->owner==NULL, so c=NULL and &c->ha = NULL+offsetof(cluster_t,ha) = 0x3ff0 — a garbage pointer ha_dispatch_gate dereferences (atomic_load &ha->role, ha.c:3194) -> SIGSEGV. Backtrace (debug -O0 -g3 under gdb): ha_dispatch_gate(ha=0x3ff0,opcode=37) <- myreq_handler cluster.c:251 <- vtask_entry cluster.c:693 <- run_task/vcpu_entry. ep->owner is NULL because the accept/dispatch path runs a request before owner is wired (or owner cleared mid-flight during teardown).",
"fix": "NOT YET FIXED — root cause nailed via the non-ASAN debug build + gdb. Minimal: NULL-guard `c = ep->owner` in myreq_handler (drop/reject the request if owner not attached) instead of dereferencing. Deeper: set ep->owner at accept time before the endpoint is monitored for input, and never clear it while a vtask is in flight (endpoint-owner lifecycle). Blocks raft-ha-core-safety 7.2 (the second of the two round-4 blockers; bug-1282 deadlock already fixed).",
"tags": [
"segfault",
"null-deref",
"ep-owner",
"myreq_handler",
"cluster.c",
"ha_dispatch_gate",
"restart",
"7.2",
"ROOT-CAUSED"
],
"related_bugs": [
"bug-1282",
"bug-1281",
"bug-1197"
],
"occurrences": 3,
"last_seen": "2026-06-30T05:30:00Z"
},
{
"id": "bug-1287",
"timestamp": "2026-06-30T09:00:43.427Z",
"error_message": "Significant refactor of ",
"file": "src/tools/xce-build.sh",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 19→53 lines (4 removed)",
"tags": [
"auto-detected",
"refactor",
"sh"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T09:00:43.427Z"
},
{
"id": "bug-1288",
"timestamp": "2026-06-30T09:02:01.055Z",
"error_message": "Significant refactor of ",
"file": ".claude/skills/xce-build/SKILL.md",
"root_cause": "3 lines replaced/restructured",
"fix": "Rewrote 17→26 lines (3 removed)",
"tags": [
"auto-detected",
"refactor",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T09:02:01.055Z"
},
{
"id": "bug-1289",
"timestamp": "2026-06-30T09:10:50.002Z",
"error_message": "Significant refactor of ",
"file": "src/tools/xce-build.sh",
"root_cause": "3 lines replaced/restructured",
"fix": "Rewrote 5→10 lines (3 removed)",
"tags": [
"auto-detected",
"refactor",
"sh"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T09:10:50.002Z"
},
{
"id": "bug-1290",
"timestamp": "2026-06-30T09:15:12.861Z",
"error_message": "Incorrect value in code",
"file": ".kiro/specs/raft-ha-core-safety/tasks.md",
"root_cause": "Had 't crash) — STILL blocks the gate; likely an endpo",
"fix": "Changed to 't crash) — is now **ROOT-CAUSED** (via the new `",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-30T09:15:12.861Z"
}
]
}

97
src/tools/xce-build.sh
View file

@ -2,7 +2,45 @@
set -euo pipefail
COMPONENT="${1:-all}"
BUILD_DIR="${2:-xce.build}"
# Build PROFILE controls the meson buildtype + sanitizer (and the default build
# dir, so the variants never clobber each other):
# asan (default) debug + AddressSanitizer → xce.build (unit-test CI)
# debug debug, NO sanitizer (-O0 -g3 -fno-inline -fno-omit-frame-
# pointer) → build.dbg (gdb / core dumps)
# release release -O3, no sanitizer → build.release (soak / perf)
#
# Why `debug` exists (bug-1286): some crashes are heisenbugs that ASAN perturbs
# away (different allocator/layout → no SIGSEGV) and that gdb-on-release turns
# into a timeout/hang. The non-ASAN -O0 -g3 build reproduces them AND yields a
# fully symbolized backtrace under gdb — the only way to root-cause that class.
#
# Arg parsing is backward-compatible: arg2 may be a profile keyword OR a legacy
# build_dir; arg3 is an optional explicit build_dir override.
case "${2:-}" in
asan|debug|release) PROFILE="$2"; BUILD_DIR="${3:-}" ;;
"") PROFILE="asan"; BUILD_DIR="" ;;
*) PROFILE="asan"; BUILD_DIR="$2" ;; # legacy: arg2 = build_dir
esac
if [ -z "${BUILD_DIR:-}" ]; then
case "$PROFILE" in
asan) BUILD_DIR="xce.build" ;;
debug) BUILD_DIR="build.dbg" ;;
release) BUILD_DIR="build.release" ;;
esac
fi
# RUN_TESTS: the unit suites are ASAN-oriented (leak detection) and the meson
# test harness is fragile on this sandbox's overlay fs (bug-924 fdatasync). Only
# the `asan` profile runs them; `debug`/`release` are BUILD-ONLY — their purpose
# is a binary to drive under gdb / a soak, not a test run. Run a suite directly
# against those binaries when you want it (it is overlay-fs sensitive).
case "$PROFILE" in
asan) MESON_OPTS=(--buildtype=debug -Db_sanitize=address); RUN_TESTS=1 ;;
debug) MESON_OPTS=(--buildtype=debug -Db_sanitize=none); RUN_TESTS=0 ;;
release) MESON_OPTS=(--buildtype=release -Db_sanitize=none); RUN_TESTS=0 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SRC_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
@ -10,15 +48,16 @@ SRC_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
cd "$SRC_DIR"
usage() {
echo "Usage: $0 [component] [build_dir]"
echo "Usage: $0 [component] [profile|build_dir] [build_dir]"
echo "Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli"
echo "Profiles: asan (default), debug (non-ASAN -O0 -g3, for gdb/cores), release"
exit 1
}
setup_build() {
echo "=== Clean build: $BUILD_DIR ==="
echo "=== Clean build: $BUILD_DIR (profile=$PROFILE) ==="
rm -rf "$BUILD_DIR"
meson setup "$BUILD_DIR" --buildtype=debug -Db_sanitize=address
meson setup "$BUILD_DIR" "${MESON_OPTS[@]}"
}
case "$COMPONENT" in
@ -26,22 +65,28 @@ case "$COMPONENT" in
setup_build
echo "=== Building lib + tests ==="
meson compile -C "$BUILD_DIR" xcelib_test
echo "=== Running lib tests ==="
meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running lib tests ==="
meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs
fi
;;
controller)
setup_build
echo "=== Building lib + controller + tests ==="
meson compile -C "$BUILD_DIR" xce_controller controller_unit_test
echo "=== Running controller tests ==="
meson test -C "$BUILD_DIR" --suite controller --print-errorlogs
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running controller tests ==="
meson test -C "$BUILD_DIR" --suite controller --print-errorlogs
fi
;;
scheduler)
setup_build
echo "=== Building lib + scheduler + tests ==="
meson compile -C "$BUILD_DIR" xce_scheduler sched_test
echo "=== Running scheduler tests ==="
"./$BUILD_DIR/scheduler/testsuite/sched_test"
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running scheduler tests ==="
"./$BUILD_DIR/scheduler/testsuite/sched_test"
fi
;;
worker)
setup_build
@ -53,30 +98,40 @@ case "$COMPONENT" in
setup_build
echo "=== Building lib + apiserver + xcecli ==="
meson compile -C "$BUILD_DIR" apiserver xcecli
echo "=== Running frontend Go unit tests ==="
cd frontend && make test
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running frontend Go unit tests ==="
cd frontend && make test
fi
;;
xcecli)
setup_build
echo "=== Building xcecli ==="
meson compile -C "$BUILD_DIR" xcecli
echo "=== Running xcecli Go unit tests ==="
cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running xcecli Go unit tests ==="
cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1
fi
;;
all)
setup_build
echo "=== Building all components ==="
meson compile -C "$BUILD_DIR"
echo "=== Running all C tests ==="
meson test -C "$BUILD_DIR" --print-errorlogs
echo "=== Running scheduler test (manual) ==="
"./$BUILD_DIR/scheduler/testsuite/sched_test"
echo "=== Running frontend Go unit tests ==="
cd frontend && make test
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running all C tests ==="
meson test -C "$BUILD_DIR" --print-errorlogs
echo "=== Running scheduler test (manual) ==="
"./$BUILD_DIR/scheduler/testsuite/sched_test"
echo "=== Running frontend Go unit tests ==="
cd frontend && make test
fi
;;
*)
usage
;;
esac
echo "=== Done: $COMPONENT ==="
if [ "$RUN_TESTS" != 1 ]; then
echo "=== Skipped tests (profile=$PROFILE — build only). Binary in $BUILD_DIR; run a suite directly, e.g. ASAN_OPTIONS=detect_leaks=0 ./$BUILD_DIR/controller/controller_unit_test ==="
fi
echo "=== Done: $COMPONENT (profile=$PROFILE, dir=$BUILD_DIR) ==="