feat(skills): fix xce-build/xce-test-ha/xce-deploy-ha subagent dispatch

- xce-build.sh: `controller` component now always builds `controller/xce_test`
  alongside `xce_controller`, in every profile — xce-test-ha needs both
  binaries and previously had to build xce_test as a separate step.
- xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this
  skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked
  subagent never receives a usable task and silently does nothing. Documented
  that this skill should instead always be dispatched via the plain Agent
  tool with a fully-resolved, self-contained prompt (not `context: fork`,
  which failed) — proven to work end-to-end.
- xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never
  invoke via subagent" warning with the precise finding — `context: fork`
  (Skill-tool isolation) is the confirmed-broken mechanism (its own teardown
  SIGHUPs the whole controller process group mid-soak/mid-deploy), but
  Agent-tool subagent dispatch is a different mechanism and is safe
  (empirically verified: a `setsid nohup ... & disown` process launched from
  inside an Agent subagent survives that subagent returning, reparented to
  init). Documented both the foreground-blocking and fire-and-forget
  subagent patterns for each skill; xce-deploy-ha additionally flagged that
  its `start.sh &` should gain `setsid` to match the soak driver's stronger
  detach pattern before relying on unattended subagent dispatch.
This commit is contained in:
Rongsong Shen 2026-07-09 11:57:11 +00:00
parent 016c598f07
commit 5e0579dede
4 changed files with 65 additions and 25 deletions

View file

@ -1,26 +1,32 @@
--- ---
name: xce-build name: xce-build
description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release." description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
context: fork
disable-model-invocation: false disable-model-invocation: false
allowed-tools: Bash(*) allowed-tools: Bash(*)
--- ---
## build xce source code ## build xce source code
Args: `[component] [profile|build_dir] [build_dir]` Args: `[component] [profile|build_dir] [build_dir]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. Parse the ARGUMENTS text yourself and run the command below with the parsed values inline.
- **component** — `all` (default), `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli` - **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt.
- **profile** — `asan` (default: debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. Arg 2 may also be a legacy explicit build_dir (anything that isn't a profile keyword); arg 3 overrides the build dir. - **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`
- **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir).
- Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked). - Use the **`debug`** profile to root-cause heisenbug crashes that ASAN perturbs away and that gdb-on-release turns into a hang/timeout — the non-ASAN `-O0 -g3` build reproduces them and gives a fully symbolized backtrace under gdb (how bug-1282/bug-1286 were cracked).
- **`controller` always builds `xce_test` too** (`meson compile … xce_controller controller_unit_test controller/xce_test`), in every profile — not just `asan`. `xce_test` is the driver `/xce-test-ha` needs; you never have to build it as a separate step after `/xce-build controller`, and the two binaries in a build dir are always the same commit/profile in lockstep.
Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`. Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`.
```bash Run (substituting the parsed component/profile/build_dir; omit trailing empty ones):
COMPONENT="$0"
PROFILE="$1"
BUILD_DIR="$2"
cd src && \ ```bash
nix develop --command bash tools/xce-build.sh ${COMPONENT:+$COMPONENT} ${PROFILE:+$PROFILE} ${BUILD_DIR:+$BUILD_DIR} nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir>
``` ```
e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`.
## Always dispatch the build via the Agent tool
Do not run the `nix develop … xce-build.sh` command inline in the current session's Bash tool. Instead, on every invocation of this skill, launch a subagent with the Agent tool, giving it a fully self-contained prompt: repo working directory, the exact resolved command (component/profile/build_dir already substituted — do not make the subagent re-parse ARGUMENTS itself), what the command does, and what to report back (build success/failure, compiler errors, the meson test Ok/Fail summary line, and confirmation the expected binaries were freshly rebuilt with fresh mtimes). Wait for it (do not background it — the build is synchronous and finishes in well under this session's turn budget) and relay its report.
This is deliberately a plain Agent-tool dispatch, not `context: fork` — that frontmatter field reproducibly failed to thread a non-empty ARGUMENTS payload into the forked subagent (see above), whereas an explicit, self-contained Agent prompt has no such dependency and is proven to work.

View file

@ -1,6 +1,7 @@
--- ---
name: xce-deploy-ha name: xce-deploy-ha
description: "Deploy and manage a local XCE HA controller cluster for testing. Usage: /xce-deploy-ha [peers] — e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993" description: "Deploy and manage a local XCE HA controller cluster for testing. Usage: /xce-deploy-ha [peers] — e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993"
model: haiku
--- ---
# XCE HA Cluster Deployment # XCE HA Cluster Deployment
@ -8,6 +9,26 @@ description: "Deploy and manage a local XCE HA controller cluster for testing. U
Deploy a multi-node XCE controller cluster for local testing using Deploy a multi-node XCE controller cluster for local testing using
`src/tools/deploy-ha.sh`. Covers: build, deploy, start, verify, stop, and cleanup. `src/tools/deploy-ha.sh`. Covers: build, deploy, start, verify, stop, and cleanup.
**No `context: fork`, intentionally.** That Skill-tool isolation mechanism is
confirmed broken for launching long-lived controller processes (same root
cause as `xce-test-ha`'s soak driver, fixed 2026-07-08): the forked session's
own teardown SIGHUPs the whole process group, killing the cluster this skill
just started. Since this skill's whole point is a cluster that **stays running
after the skill returns** for later interactive use, that failure mode is
fatal here — do not add `context: fork` back.
**Agent-tool subagent dispatch is fine, though** (different mechanism,
empirically verified safe 2026-07-09 — see `xce-test-ha/SKILL.md`): dispatch
an `Agent` subagent whose job is only to run Steps 1-5 below, ending with
`start.sh ctrl-N &` for each node backgrounded with the usual shell `&`
inside that subagent's own Bash call, then have the subagent return once
health-check (Step 5) passes. The started controllers reparent to init and
keep running independently of the subagent's own lifetime, same as the
`setsid nohup … & disown` pattern verified for the soak driver. Prefer adding
`setsid` to the `start.sh` invocations here too (belt-and-suspenders — this
skill currently uses plain `&`, weaker than the soak driver's `setsid nohup … &
disown`) before relying on subagent dispatch for unattended use.
## Prerequisites ## Prerequisites
- Build must be current: run `/xce-build controller` first if binary is stale - Build must be current: run `/xce-build controller` first if binary is stale

View file

@ -195,20 +195,33 @@ dev shell so the controller/xce_test runtime libs (openssl, etc.) are on the
library path. Run it in the **background** (it takes several minutes) and tail library path. Run it in the **background** (it takes several minutes) and tail
the log: the log:
**Launch this directly via the Bash tool from your own (non-forked) session — **Never launch this via a `context: fork` Skill-tool invocation.** That
never through a sub-agent or a forked/isolated skill invocation.** A 5-10 specific mechanism is confirmed broken for this driver (bug, fixed
round soak runs for several minutes; if the process that launched it exits or 2026-07-08): the forked session's own teardown sends SIGHUP through the whole
tears down its session before the soak finishes, the child `xce_controller` process group, silently killing all 3 `xce_controller`s mid-round with no
processes can be killed outright (observed: a forked/background sub-session error logged — the run just stops. `setsid nohup` in `run-ha-soak.sh`'s
ending sent SIGHUP through to the whole process group, silently killing all 3 `start_node()` protects against a SIGHUP from a controlling terminal, but not
controllers mid-round with no error logged — the run just stops). `setsid against `context: fork`'s own teardown signal. This skill's frontmatter has no
nohup` in `run-ha-soak.sh`'s `start_node()` protects against a SIGHUP from `context: fork` for exactly this reason — do not add it back.
your own shell's controlling terminal, but it does NOT protect against a
parent session's own process-group/cgroup teardown when that parent's **Agent-tool subagent dispatch (the `Agent` tool, not `context: fork`) is a
lifetime is itself bounded (e.g. a forked sub-agent invocation that ends when different mechanism and is safe** — empirically verified 2026-07-09: a process
it returns its response). Launch with `setsid nohup ... </dev/null >log 2>&1 & launched with `setsid nohup … & disown` from inside an `Agent` subagent
disown` from the main session, then poll/Monitor that log from the same survives that subagent returning/completing (reparents to init, PPID=1, runs
session — do not delegate the launch itself to a forked context. to completion unaffected). So either of these work:
- **Foreground**: dispatch an `Agent` subagent with a self-contained prompt
telling it to run the `nix develop … run-ha-soak.sh` command directly
(blocking, no `&`) and wait the full 5-10 minutes for the PASS/FAIL verdict
line, then report it back. Simplest — no detach/reparent involved at all.
- **Fire-and-forget**: dispatch an `Agent` subagent whose only job is to launch
with `setsid nohup … </dev/null >log 2>&1 & disown` and return immediately
(do not wait/sleep inside that subagent); then poll/Monitor the log
independently (from the main session, or a later subagent) once the
detached process is already running.
Either way, launching directly via the Bash tool from the main session (as
below) also still works and needs no subagent at all — pick whichever fits
whether you want the main session free during the soak.
```bash ```bash
SKILL_DIR=.claude/skills/xce-test-ha SKILL_DIR=.claude/skills/xce-test-ha

View file

@ -73,7 +73,7 @@ case "$COMPONENT" in
controller) controller)
setup_build setup_build
echo "=== Building lib + controller + tests ===" echo "=== Building lib + controller + tests ==="
meson compile -C "$BUILD_DIR" xce_controller controller_unit_test meson compile -C "$BUILD_DIR" xce_controller controller_unit_test controller/xce_test
if [ "$RUN_TESTS" = 1 ]; then if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running controller tests ===" echo "=== Running controller tests ==="
meson test -C "$BUILD_DIR" --suite controller --print-errorlogs meson test -C "$BUILD_DIR" --suite controller --print-errorlogs