diff --git a/.claude/skills/xce-build/SKILL.md b/.claude/skills/xce-build/SKILL.md index af30497d..a3d9fb0b 100644 --- a/.claude/skills/xce-build/SKILL.md +++ b/.claude/skills/xce-build/SKILL.md @@ -1,6 +1,6 @@ --- name: xce-build -description: "Build XCE components with tests. Usage: /xce-build [component] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli" +description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release." context: fork disable-model-invocation: false allowed-tools: Bash(*) @@ -8,10 +8,19 @@ allowed-tools: Bash(*) ## build xce source code +Args: `[component] [profile|build_dir] [build_dir]` + +- **component** — `all` (default), `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli` +- **profile** — `asan` (default: debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. Arg 2 may also be a legacy explicit build_dir (anything that isn't a profile keyword); arg 3 overrides the build dir. +- Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked). + +Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`. + ```bash COMPONENT="$0" -BUILD_DIR="$1" +PROFILE="$1" +BUILD_DIR="$2" cd src && \ - nix develop --command bash tools/xce-build.sh ${COMPONENT:+$COMPONENT} ${BUILD_DIR:+$BUILD_DIR} + nix develop --command bash tools/xce-build.sh ${COMPONENT:+$COMPONENT} ${PROFILE:+$PROFILE} ${BUILD_DIR:+$BUILD_DIR} ``` diff --git a/.kiro/specs/raft-ha-core-safety/research.md b/.kiro/specs/raft-ha-core-safety/research.md index ce0a2a1b..ce994926 100644 --- a/.kiro/specs/raft-ha-core-safety/research.md +++ b/.kiro/specs/raft-ha-core-safety/research.md @@ -508,3 +508,32 @@ The bug-1282 deadlock fix removes ONE of the two round-4 blockers (the iomgr wed blocker is bug-1283 (the restart segfault), likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. 7.2 stays blocked on bug-1283; consensus correctness (Phase-2/5.2/5.3) is unaffected and elections + job safety hold for every round that completes. + +--- + +# bug-1286 (the round-4 segfault) ROOT-CAUSED — NULL ep->owner — via the new /xce-build debug profile + +The `/xce-build ... debug` profile added this session (non-ASAN `-O0 -g3`, no sanitizer → `build.dbg`) +reproduced the heisenbug AND gave a symbolized backtrace under gdb (ASAN and gdb-on-release both +perturbed it away). The captured SIGSEGV: + +``` +Thread received SIGSEGV +#0 ha_dispatch_gate (ha=0x3ff0, opcode=37) controller/ha.c:3194 (atomic_load &ha->role) +#1 myreq_handler controller/cluster.c:251 +#2 vtask_entry controller/cluster.c:693 +#3 run_task / vcpu_entry lib/vtaskmgr.c +``` + +Root cause: `myreq_handler` does `cluster_t* c = (cluster_t*)ep->owner;` then +`ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a +`walCatchupEntry` (op 37) vtask while `ep->owner == NULL`, so `c == NULL` and +`&c->ha == NULL + offsetof(cluster_t,ha) == 0x3ff0` — a garbage pointer `ha_dispatch_gate` +dereferences → SIGSEGV. (DISTINCT from the bug-1282 deadlock, which is fixed; deadlocks hang, +they do not crash.) Fix: NULL-guard `c = ep->owner` in `myreq_handler` (drop/reject the request if +the endpoint's owner is not yet attached) instead of dereferencing; deeper, ensure `ep->owner` is +set at accept time before the endpoint is monitored for input and never cleared while a vtask is in +flight. Tracked as bug-1286 (an earlier auto-logged entry already held the id bug-1283). + +7.2 now has ONE remaining blocker: bug-1286 (root-caused; small NULL-guard fix). bug-1282 (deadlock) +is fixed. Consensus correctness (Phase-2/5.2/5.3) is unaffected. diff --git a/.kiro/specs/raft-ha-core-safety/tasks.md b/.kiro/specs/raft-ha-core-safety/tasks.md index d4377800..4a2257b1 100644 --- a/.kiro/specs/raft-ha-core-safety/tasks.md +++ b/.kiro/specs/raft-ha-core-safety/tasks.md @@ -154,7 +154,7 @@ - Observable completion: the soak exits PASS over ≥6 chaotic rounds with zero divergence and zero job loss — the green result that gates the `raft-drive-async` production flag-flip - _Requirements: 1.4, 7.4, 8.1, 8.2, 8.3, 8.4_ - _Depends: 1.5, 6.2, 5.2, 5.3, 5.4_ - - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1283 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — STILL blocks the gate; likely an endpoint-lifetime UAF under chaotic restart (bug-1197 family), heisenbug invisible under ASAN+gdb, needs a non-ASAN DEBUG+core+gdb env. Likely UPSTREAM raft-drive-async / core libxce endpoint quiescence. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1282 FIXED + bug-1283 isolated". Re-run 7.2 after bug-1283 is fixed._ + - _Blocked on bug-1283 (the round-4 stall had TWO independent causes). (1) bug-1282 — an ABBA lock-order deadlock in `src/lib` async dispatch (`io_worker` ep->lock→q->lock vs `vdispatcher` `dataq_deq` q->lock→ep->lock via `endpoint_is_task_blocked`) — is **FIXED** (commit 25e741d: `endpoint_t.busy` → `_Atomic int`, lockless; verified zero `__lll_lock_wait` post-fix; unit suite green). (2) bug-1286 — a SEPARATE release-only SIGSEGV on a restarted standby at round 4 (deadlocks hang, they don't crash) — is now **ROOT-CAUSED** (via the new `/xce-build debug` non-ASAN `-O0 -g3` profile + gdb): `myreq_handler` (cluster.c:251) does `c = ep->owner` then `ha_dispatch_gate(&c->ha, …)`; on a chaotic restart a `walCatchupEntry` (op 37) vtask runs with `ep->owner == NULL`, so `&c->ha == 0x3ff0` → deref crash at ha.c:3194. Fix = NULL-guard `c = ep->owner` in `myreq_handler` (+ ensure owner is set at accept time / not cleared mid-flight). STILL blocks the gate until that lands. Consensus (Phase-2/5.2/5.3) is sound; elections + job safety hold for every round that completes. Separately, the soak over-counts divergence (boundary `1+2*njobs` vs primary `committedLsn`). Detail in `research.md` §"bug-1286 … ROOT-CAUSED". Re-run 7.2 after bug-1286 is fixed._ - [ ] 7.3 Hypothesis-validation closeout - Confirm the live soak divergence is eliminated by the implemented fixes (not by synthetic unit tests alone); if any residual divergence remains, classify it via `walcompare (term, lsn, pcrc)`, record the new finding in `research.md`, and add a follow-up fix task rather than declaring closure diff --git a/.wolf/buglog.json b/.wolf/buglog.json index f70c50e1..ac1af7f7 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -21304,6 +21304,96 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-30T08:24:48.860Z" + }, + { + "id": "bug-1286", + "timestamp": "2026-06-30T05:30:00Z", + "error_message": "xce_controller SIGSEGV in ha_dispatch_gate during chaotic round-4 restart — Thread received SIGSEGV at ha.c:3194 (atomic_load &ha->role) with ha=0x3ff0 (garbage). The release-only crash that blocks raft-ha-core-safety 7.2; DISTINCT from the bug-1282 deadlock (which is fixed).", + "file": "src/controller/cluster.c:251 (myreq_handler) + src/controller/ha.c:3194 (ha_dispatch_gate)", + "root_cause": "NULL ep->owner. myreq_handler does `cluster_t* c = (cluster_t*)ep->owner;` then `ha_dispatch_gate(&c->ha, opcode)`. On a chaotic restart an inbound endpoint dispatches a walCatchupEntry (op 37) vtask while ep->owner==NULL, so c=NULL and &c->ha = NULL+offsetof(cluster_t,ha) = 0x3ff0 — a garbage pointer ha_dispatch_gate dereferences (atomic_load &ha->role, ha.c:3194) -> SIGSEGV. Backtrace (debug -O0 -g3 under gdb): ha_dispatch_gate(ha=0x3ff0,opcode=37) <- myreq_handler cluster.c:251 <- vtask_entry cluster.c:693 <- run_task/vcpu_entry. ep->owner is NULL because the accept/dispatch path runs a request before owner is wired (or owner cleared mid-flight during teardown).", + "fix": "NOT YET FIXED — root cause nailed via the non-ASAN debug build + gdb. Minimal: NULL-guard `c = ep->owner` in myreq_handler (drop/reject the request if owner not attached) instead of dereferencing. Deeper: set ep->owner at accept time before the endpoint is monitored for input, and never clear it while a vtask is in flight (endpoint-owner lifecycle). Blocks raft-ha-core-safety 7.2 (the second of the two round-4 blockers; bug-1282 deadlock already fixed).", + "tags": [ + "segfault", + "null-deref", + "ep-owner", + "myreq_handler", + "cluster.c", + "ha_dispatch_gate", + "restart", + "7.2", + "ROOT-CAUSED" + ], + "related_bugs": [ + "bug-1282", + "bug-1281", + "bug-1197" + ], + "occurrences": 3, + "last_seen": "2026-06-30T05:30:00Z" + }, + { + "id": "bug-1287", + "timestamp": "2026-06-30T09:00:43.427Z", + "error_message": "Significant refactor of ", + "file": "src/tools/xce-build.sh", + "root_cause": "4 lines replaced/restructured", + "fix": "Rewrote 19→53 lines (4 removed)", + "tags": [ + "auto-detected", + "refactor", + "sh" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T09:00:43.427Z" + }, + { + "id": "bug-1288", + "timestamp": "2026-06-30T09:02:01.055Z", + "error_message": "Significant refactor of ", + "file": ".claude/skills/xce-build/SKILL.md", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 17→26 lines (3 removed)", + "tags": [ + "auto-detected", + "refactor", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T09:02:01.055Z" + }, + { + "id": "bug-1289", + "timestamp": "2026-06-30T09:10:50.002Z", + "error_message": "Significant refactor of ", + "file": "src/tools/xce-build.sh", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 5→10 lines (3 removed)", + "tags": [ + "auto-detected", + "refactor", + "sh" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T09:10:50.002Z" + }, + { + "id": "bug-1290", + "timestamp": "2026-06-30T09:15:12.861Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/raft-ha-core-safety/tasks.md", + "root_cause": "Had 't crash) — STILL blocks the gate; likely an endpo", + "fix": "Changed to 't crash) — is now **ROOT-CAUSED** (via the new `", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-30T09:15:12.861Z" } ] } \ No newline at end of file diff --git a/src/tools/xce-build.sh b/src/tools/xce-build.sh index c8f47084..52056d3f 100755 --- a/src/tools/xce-build.sh +++ b/src/tools/xce-build.sh @@ -2,7 +2,45 @@ set -euo pipefail COMPONENT="${1:-all}" -BUILD_DIR="${2:-xce.build}" + +# Build PROFILE controls the meson buildtype + sanitizer (and the default build +# dir, so the variants never clobber each other): +# asan (default) debug + AddressSanitizer → xce.build (unit-test CI) +# debug debug, NO sanitizer (-O0 -g3 -fno-inline -fno-omit-frame- +# pointer) → build.dbg (gdb / core dumps) +# release release -O3, no sanitizer → build.release (soak / perf) +# +# Why `debug` exists (bug-1286): some crashes are heisenbugs that ASAN perturbs +# away (different allocator/layout → no SIGSEGV) and that gdb-on-release turns +# into a timeout/hang. The non-ASAN -O0 -g3 build reproduces them AND yields a +# fully symbolized backtrace under gdb — the only way to root-cause that class. +# +# Arg parsing is backward-compatible: arg2 may be a profile keyword OR a legacy +# build_dir; arg3 is an optional explicit build_dir override. +case "${2:-}" in + asan|debug|release) PROFILE="$2"; BUILD_DIR="${3:-}" ;; + "") PROFILE="asan"; BUILD_DIR="" ;; + *) PROFILE="asan"; BUILD_DIR="$2" ;; # legacy: arg2 = build_dir +esac + +if [ -z "${BUILD_DIR:-}" ]; then + case "$PROFILE" in + asan) BUILD_DIR="xce.build" ;; + debug) BUILD_DIR="build.dbg" ;; + release) BUILD_DIR="build.release" ;; + esac +fi + +# RUN_TESTS: the unit suites are ASAN-oriented (leak detection) and the meson +# test harness is fragile on this sandbox's overlay fs (bug-924 fdatasync). Only +# the `asan` profile runs them; `debug`/`release` are BUILD-ONLY — their purpose +# is a binary to drive under gdb / a soak, not a test run. Run a suite directly +# against those binaries when you want it (it is overlay-fs sensitive). +case "$PROFILE" in + asan) MESON_OPTS=(--buildtype=debug -Db_sanitize=address); RUN_TESTS=1 ;; + debug) MESON_OPTS=(--buildtype=debug -Db_sanitize=none); RUN_TESTS=0 ;; + release) MESON_OPTS=(--buildtype=release -Db_sanitize=none); RUN_TESTS=0 ;; +esac SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SRC_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" @@ -10,15 +48,16 @@ SRC_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" cd "$SRC_DIR" usage() { - echo "Usage: $0 [component] [build_dir]" + echo "Usage: $0 [component] [profile|build_dir] [build_dir]" echo "Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli" + echo "Profiles: asan (default), debug (non-ASAN -O0 -g3, for gdb/cores), release" exit 1 } setup_build() { - echo "=== Clean build: $BUILD_DIR ===" + echo "=== Clean build: $BUILD_DIR (profile=$PROFILE) ===" rm -rf "$BUILD_DIR" - meson setup "$BUILD_DIR" --buildtype=debug -Db_sanitize=address + meson setup "$BUILD_DIR" "${MESON_OPTS[@]}" } case "$COMPONENT" in @@ -26,22 +65,28 @@ case "$COMPONENT" in setup_build echo "=== Building lib + tests ===" meson compile -C "$BUILD_DIR" xcelib_test - echo "=== Running lib tests ===" - meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs + if [ "$RUN_TESTS" = 1 ]; then + echo "=== Running lib tests ===" + meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs + fi ;; controller) setup_build echo "=== Building lib + controller + tests ===" meson compile -C "$BUILD_DIR" xce_controller controller_unit_test - echo "=== Running controller tests ===" - meson test -C "$BUILD_DIR" --suite controller --print-errorlogs + if [ "$RUN_TESTS" = 1 ]; then + echo "=== Running controller tests ===" + meson test -C "$BUILD_DIR" --suite controller --print-errorlogs + fi ;; scheduler) setup_build echo "=== Building lib + scheduler + tests ===" meson compile -C "$BUILD_DIR" xce_scheduler sched_test - echo "=== Running scheduler tests ===" - "./$BUILD_DIR/scheduler/testsuite/sched_test" + if [ "$RUN_TESTS" = 1 ]; then + echo "=== Running scheduler tests ===" + "./$BUILD_DIR/scheduler/testsuite/sched_test" + fi ;; worker) setup_build @@ -53,30 +98,40 @@ case "$COMPONENT" in setup_build echo "=== Building lib + apiserver + xcecli ===" meson compile -C "$BUILD_DIR" apiserver xcecli - echo "=== Running frontend Go unit tests ===" - cd frontend && make test + if [ "$RUN_TESTS" = 1 ]; then + echo "=== Running frontend Go unit tests ===" + cd frontend && make test + fi ;; xcecli) setup_build echo "=== Building xcecli ===" meson compile -C "$BUILD_DIR" xcecli - echo "=== Running xcecli Go unit tests ===" - cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1 + if [ "$RUN_TESTS" = 1 ]; then + echo "=== Running xcecli Go unit tests ===" + cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1 + fi ;; all) setup_build echo "=== Building all components ===" meson compile -C "$BUILD_DIR" - echo "=== Running all C tests ===" - meson test -C "$BUILD_DIR" --print-errorlogs - echo "=== Running scheduler test (manual) ===" - "./$BUILD_DIR/scheduler/testsuite/sched_test" - echo "=== Running frontend Go unit tests ===" - cd frontend && make test + if [ "$RUN_TESTS" = 1 ]; then + echo "=== Running all C tests ===" + meson test -C "$BUILD_DIR" --print-errorlogs + echo "=== Running scheduler test (manual) ===" + "./$BUILD_DIR/scheduler/testsuite/sched_test" + echo "=== Running frontend Go unit tests ===" + cd frontend && make test + fi ;; *) usage ;; esac -echo "=== Done: $COMPONENT ===" +if [ "$RUN_TESTS" != 1 ]; then + echo "=== Skipped tests (profile=$PROFILE — build only). Binary in $BUILD_DIR; run a suite directly, e.g. ASAN_OPTIONS=detect_leaks=0 ./$BUILD_DIR/controller/controller_unit_test ===" +fi + +echo "=== Done: $COMPONENT (profile=$PROFILE, dir=$BUILD_DIR) ==="