From 5e0579dedea8eee7cfc15d0c66b2f2ba487663de Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 9 Jul 2026 11:57:11 +0000 Subject: [PATCH] feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - xce-build.sh: `controller` component now always builds `controller/xce_test` alongside `xce_controller`, in every profile — xce-test-ha needs both binaries and previously had to build xce_test as a separate step. - xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked subagent never receives a usable task and silently does nothing. Documented that this skill should instead always be dispatched via the plain Agent tool with a fully-resolved, self-contained prompt (not `context: fork`, which failed) — proven to work end-to-end. - xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never invoke via subagent" warning with the precise finding — `context: fork` (Skill-tool isolation) is the confirmed-broken mechanism (its own teardown SIGHUPs the whole controller process group mid-soak/mid-deploy), but Agent-tool subagent dispatch is a different mechanism and is safe (empirically verified: a `setsid nohup ... & disown` process launched from inside an Agent subagent survives that subagent returning, reparented to init). Documented both the foreground-blocking and fire-and-forget subagent patterns for each skill; xce-deploy-ha additionally flagged that its `start.sh &` should gain `setsid` to match the soak driver's stronger detach pattern before relying on unattended subagent dispatch. --- .claude/skills/xce-build/SKILL.md | 26 ++++++++++------- .claude/skills/xce-deploy-ha/SKILL.md | 21 ++++++++++++++ .claude/skills/xce-test-ha/SKILL.md | 41 ++++++++++++++++++--------- src/tools/xce-build.sh | 2 +- 4 files changed, 65 insertions(+), 25 deletions(-) diff --git a/.claude/skills/xce-build/SKILL.md b/.claude/skills/xce-build/SKILL.md index a3d9fb0..4252075 100644 --- a/.claude/skills/xce-build/SKILL.md +++ b/.claude/skills/xce-build/SKILL.md @@ -1,26 +1,32 @@ --- name: xce-build description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release." -context: fork disable-model-invocation: false allowed-tools: Bash(*) --- ## build xce source code -Args: `[component] [profile|build_dir] [build_dir]` +Args: `[component] [profile|build_dir] [build_dir]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. Parse the ARGUMENTS text yourself and run the command below with the parsed values inline. -- **component** — `all` (default), `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli` -- **profile** — `asan` (default: debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. Arg 2 may also be a legacy explicit build_dir (anything that isn't a profile keyword); arg 3 overrides the build dir. +- **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt. +- **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli` +- **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir). - Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked). +- **`controller` always builds `xce_test` too** (`meson compile … xce_controller controller_unit_test controller/xce_test`), in every profile — not just `asan`. `xce_test` is the driver `/xce-test-ha` needs; you never have to build it as a separate step after `/xce-build controller`, and the two binaries in a build dir are always the same commit/profile in lockstep. Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`. -```bash -COMPONENT="$0" -PROFILE="$1" -BUILD_DIR="$2" +Run (substituting the parsed component/profile/build_dir; omit trailing empty ones): -cd src && \ - nix develop --command bash tools/xce-build.sh ${COMPONENT:+$COMPONENT} ${PROFILE:+$PROFILE} ${BUILD_DIR:+$BUILD_DIR} +```bash +nix develop ./src --command bash src/tools/xce-build.sh ``` + +e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. + +## Always dispatch the build via the Agent tool + +Do not run the `nix develop … xce-build.sh` command inline in the current session's Bash tool. Instead, on every invocation of this skill, launch a subagent with the Agent tool, giving it a fully self-contained prompt: repo working directory, the exact resolved command (component/profile/build_dir already substituted — do not make the subagent re-parse ARGUMENTS itself), what the command does, and what to report back (build success/failure, compiler errors, the meson test Ok/Fail summary line, and confirmation the expected binaries were freshly rebuilt with fresh mtimes). Wait for it (do not background it — the build is synchronous and finishes in well under this session's turn budget) and relay its report. + +This is deliberately a plain Agent-tool dispatch, not `context: fork` — that frontmatter field reproducibly failed to thread a non-empty ARGUMENTS payload into the forked subagent (see above), whereas an explicit, self-contained Agent prompt has no such dependency and is proven to work. diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 2a3c6f4..0dfbffe 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -1,6 +1,7 @@ --- name: xce-deploy-ha description: "Deploy and manage a local XCE HA controller cluster for testing. Usage: /xce-deploy-ha [peers] — e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993" +model: haiku --- # XCE HA Cluster Deployment @@ -8,6 +9,26 @@ description: "Deploy and manage a local XCE HA controller cluster for testing. U Deploy a multi-node XCE controller cluster for local testing using `src/tools/deploy-ha.sh`. Covers: build, deploy, start, verify, stop, and cleanup. +**No `context: fork`, intentionally.** That Skill-tool isolation mechanism is +confirmed broken for launching long-lived controller processes (same root +cause as `xce-test-ha`'s soak driver, fixed 2026-07-08): the forked session's +own teardown SIGHUPs the whole process group, killing the cluster this skill +just started. Since this skill's whole point is a cluster that **stays running +after the skill returns** for later interactive use, that failure mode is +fatal here — do not add `context: fork` back. + +**Agent-tool subagent dispatch is fine, though** (different mechanism, +empirically verified safe 2026-07-09 — see `xce-test-ha/SKILL.md`): dispatch +an `Agent` subagent whose job is only to run Steps 1-5 below, ending with +`start.sh ctrl-N &` for each node backgrounded with the usual shell `&` +inside that subagent's own Bash call, then have the subagent return once +health-check (Step 5) passes. The started controllers reparent to init and +keep running independently of the subagent's own lifetime, same as the +`setsid nohup … & disown` pattern verified for the soak driver. Prefer adding +`setsid` to the `start.sh` invocations here too (belt-and-suspenders — this +skill currently uses plain `&`, weaker than the soak driver's `setsid nohup … & +disown`) before relying on subagent dispatch for unattended use. + ## Prerequisites - Build must be current: run `/xce-build controller` first if binary is stale diff --git a/.claude/skills/xce-test-ha/SKILL.md b/.claude/skills/xce-test-ha/SKILL.md index 09e6675..59b9724 100644 --- a/.claude/skills/xce-test-ha/SKILL.md +++ b/.claude/skills/xce-test-ha/SKILL.md @@ -195,20 +195,33 @@ dev shell so the controller/xce_test runtime libs (openssl, etc.) are on the library path. Run it in the **background** (it takes several minutes) and tail the log: -**Launch this directly via the Bash tool from your own (non-forked) session — -never through a sub-agent or a forked/isolated skill invocation.** A 5-10 -round soak runs for several minutes; if the process that launched it exits or -tears down its session before the soak finishes, the child `xce_controller` -processes can be killed outright (observed: a forked/background sub-session -ending sent SIGHUP through to the whole process group, silently killing all 3 -controllers mid-round with no error logged — the run just stops). `setsid -nohup` in `run-ha-soak.sh`'s `start_node()` protects against a SIGHUP from -your own shell's controlling terminal, but it does NOT protect against a -parent session's own process-group/cgroup teardown when that parent's -lifetime is itself bounded (e.g. a forked sub-agent invocation that ends when -it returns its response). Launch with `setsid nohup ... log 2>&1 & -disown` from the main session, then poll/Monitor that log from the same -session — do not delegate the launch itself to a forked context. +**Never launch this via a `context: fork` Skill-tool invocation.** That +specific mechanism is confirmed broken for this driver (bug, fixed +2026-07-08): the forked session's own teardown sends SIGHUP through the whole +process group, silently killing all 3 `xce_controller`s mid-round with no +error logged — the run just stops. `setsid nohup` in `run-ha-soak.sh`'s +`start_node()` protects against a SIGHUP from a controlling terminal, but not +against `context: fork`'s own teardown signal. This skill's frontmatter has no +`context: fork` for exactly this reason — do not add it back. + +**Agent-tool subagent dispatch (the `Agent` tool, not `context: fork`) is a +different mechanism and is safe** — empirically verified 2026-07-09: a process +launched with `setsid nohup … & disown` from inside an `Agent` subagent +survives that subagent returning/completing (reparents to init, PPID=1, runs +to completion unaffected). So either of these work: +- **Foreground**: dispatch an `Agent` subagent with a self-contained prompt + telling it to run the `nix develop … run-ha-soak.sh` command directly + (blocking, no `&`) and wait the full 5-10 minutes for the PASS/FAIL verdict + line, then report it back. Simplest — no detach/reparent involved at all. +- **Fire-and-forget**: dispatch an `Agent` subagent whose only job is to launch + with `setsid nohup … log 2>&1 & disown` and return immediately + (do not wait/sleep inside that subagent); then poll/Monitor the log + independently (from the main session, or a later subagent) once the + detached process is already running. + +Either way, launching directly via the Bash tool from the main session (as +below) also still works and needs no subagent at all — pick whichever fits +whether you want the main session free during the soak. ```bash SKILL_DIR=.claude/skills/xce-test-ha diff --git a/src/tools/xce-build.sh b/src/tools/xce-build.sh index 52056d3..f0688cb 100755 --- a/src/tools/xce-build.sh +++ b/src/tools/xce-build.sh @@ -73,7 +73,7 @@ case "$COMPONENT" in controller) setup_build echo "=== Building lib + controller + tests ===" - meson compile -C "$BUILD_DIR" xce_controller controller_unit_test + meson compile -C "$BUILD_DIR" xce_controller controller_unit_test controller/xce_test if [ "$RUN_TESTS" = 1 ]; then echo "=== Running controller tests ===" meson test -C "$BUILD_DIR" --suite controller --print-errorlogs