feat(build): cap xce-build.sh's run-queue length via a live-computed --jobs flag
Add --jobs N to xce-build.sh: for meson compile it's passed as ninja's -l N load-average ceiling (re-checked before every job launch for the whole build, so it adapts if a concurrent session in this shared sandbox loads the system mid-build), and as a job-count ceiling (-j/--num-processes, -p) for meson test/go test, which have no load-average equivalent. No default -- omitting it preserves prior auto-parallelism behavior exactly. The xce-build skill now computes N live at invocation time from nproc, /proc/meminfo's MemAvailable, and /proc/loadavg, rather than a fixed constant, and passes it through. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DrxU1dSRYNMV9CDTXHFtcR
This commit is contained in:
parent
099dbdabdc
commit
850d186dd8
2 changed files with 94 additions and 26 deletions
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
name: xce-build
|
||||
description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release."
|
||||
description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release. Automatically caps build/test parallelism (--jobs) to the CPU/memory/load available at invocation time."
|
||||
disable-model-invocation: false
|
||||
allowed-tools: Bash(*)
|
||||
---
|
||||
|
|
@ -9,6 +9,27 @@ allowed-tools: Bash(*)
|
|||
|
||||
Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. `--incremental` can appear anywhere in the words (it's stripped out before the rest are parsed positionally — see below). Parse the ARGUMENTS text yourself and run the command below with the parsed values inline.
|
||||
|
||||
## Run-queue-length limit (`--jobs N`) — compute it live before every build, don't skip this
|
||||
|
||||
`xce-build.sh` accepts a `--jobs N` flag that caps the build's run-queue length. For `meson compile` (the ninja backend, and the memory-heaviest step — ASAN+debug C/C++ compiles) it's passed as **`-l N`, ninja's load-average ceiling** — ninja re-checks the system's live load average, literally its run-queue length, before launching *every* new job for the *whole* build, and holds off while it's above N. That's the actual control lever, deliberately not paired with pinning `-j` to the same number: `-j` would just be a one-time snapshot cap decided before the first job starts, whereas `-l` keeps adapting for the entire build — including backing off if some *other* concurrent session's build drives load up partway through (this sandbox routinely runs multiple sessions building at once). `-j` is left at ninja's own default (~nproc+2) as a loose ceiling. `meson test` and `go test` have no load-average equivalent (only flat process-count knobs), so N is used there as a job-count ceiling instead — `-j`/`--num-processes N` for `meson test`, `-p N` for `go test`. The flag has no default — if you omit it, everything falls back to its own auto-parallelism, same as before this flag existed.
|
||||
|
||||
**Always compute N yourself, live, immediately before launching the build** — from the hardware/load state *at the moment the skill is invoked*, not a number remembered from an earlier run or a fixed constant. This sandbox routinely runs multiple concurrent sessions building/testing at once (see `xce-build.sh`'s own `prune_stale_tmp_fixtures` comment), so a stale or hardcoded value can either starve this build or overload a system something else is already using.
|
||||
|
||||
1. Read current hardware/load state:
|
||||
```bash
|
||||
nproc --all
|
||||
awk '/MemAvailable/ {print int($2/1024)}' /proc/meminfo # available memory, MB (reclaimable-aware, unlike `free`'s raw "free" column)
|
||||
cut -d' ' -f1 /proc/loadavg # current 1-min load average = existing run-queue length from other activity
|
||||
```
|
||||
2. Pick a per-job memory budget by profile — ASAN+debug is the heaviest (debug info + sanitizer instrumentation per translation unit): **~1536 MB/job for `asan`** (the default profile), **~1024 MB/job for `debug`/`release`**.
|
||||
3. Compute, reserving ~15% of available memory as headroom for the OS and other sessions:
|
||||
- `mem_jobs = floor(available_mem_mb * 0.85 / per_job_mb)`
|
||||
- `cpu_jobs = max(1, floor(nproc - current_load_avg))` — leaves room for whatever's already running
|
||||
- `JOBS = max(1, min(nproc, mem_jobs, cpu_jobs))`
|
||||
4. State the computed value and its inputs in one line (e.g. `nproc=22, available=20466MB, load=1.2 → --jobs 14`) before launching the build, then pass `--jobs <JOBS>` to `xce-build.sh` alongside the other args.
|
||||
|
||||
This is a judgment call, not a rigid formula — if you notice other heavy build processes already running (`ps aux | grep -E 'cc1plus|ninja|go build'`) or the available-memory read looks unusually low, bias further down.
|
||||
|
||||
- **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt.
|
||||
- **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`, `xceadmin`
|
||||
- **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir).
|
||||
|
|
@ -19,13 +40,13 @@ Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed a
|
|||
|
||||
Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (build + test + wheel, any profile arg is accepted and ignored), `/xce-build xceadmin --incremental` (same, but keeps other components already built in the same dir).
|
||||
|
||||
Run (substituting the parsed component/profile/build_dir; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested):
|
||||
Run (substituting the parsed component/profile/build_dir, and the `--jobs N` value computed per the section above; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested):
|
||||
|
||||
```bash
|
||||
nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir> [--incremental]
|
||||
nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir> --jobs <N> [--incremental]
|
||||
```
|
||||
|
||||
e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. For `/xce-build xceadmin --incremental`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --incremental`.
|
||||
e.g. for `/xce-build controller` with a computed `N=14`: `nix develop ./src --command bash src/tools/xce-build.sh controller --jobs 14`. For `/xce-build xceadmin --incremental` with `N=8`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --jobs 8 --incremental` (`--jobs` and `--incremental` are flags, stripped out from any position before the remaining words are parsed positionally — they don't need to follow profile/build_dir).
|
||||
|
||||
## Run the build via Bash `run_in_background`, not an Agent-tool subagent
|
||||
|
||||
|
|
@ -34,7 +55,7 @@ e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xc
|
|||
Instead, run the resolved command directly with the **Bash tool's own `run_in_background: true`**, from the current session, and let the harness's own task-notification mechanism — not a subagent's self-report — tell you when it's done. This has been empirically proven reliable in this exact repo (2026-09-03: both a full `all debug` build and a `controller` ASAN rebuild were driven this way, and each genuinely blocked until real completion).
|
||||
|
||||
```bash
|
||||
nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir> [--incremental] \
|
||||
nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> <build_dir> --jobs <N> [--incremental] \
|
||||
> <scratchpad>/xce-build-<component>-<profile>.log 2>&1; \
|
||||
echo "EXIT_CODE=$?" >> <scratchpad>/xce-build-<component>-<profile>.log
|
||||
```
|
||||
|
|
|
|||
|
|
@ -22,15 +22,55 @@ set -euo pipefail
|
|||
# want a sibling component alongside an already-built one in the same dir;
|
||||
# rebuild without it (a real clean build) if anything looks off afterward.
|
||||
INCREMENTAL=0
|
||||
JOBS=""
|
||||
ARGS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--incremental) INCREMENTAL=1 ;;
|
||||
*) ARGS+=("$arg") ;;
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--incremental) INCREMENTAL=1; shift ;;
|
||||
--jobs) JOBS="${2:-}"; shift 2 ;;
|
||||
--jobs=*) JOBS="${1#--jobs=}"; shift ;;
|
||||
*) ARGS+=("$1"); shift ;;
|
||||
esac
|
||||
done
|
||||
set -- "${ARGS[@]}"
|
||||
|
||||
# --jobs (run-queue-length limit): opt-in, no default -- when unset,
|
||||
# meson/ninja/go fall back to their own auto-parallelism (typically nproc or
|
||||
# nproc+2 workers), exactly as before this flag existed. Callers are expected
|
||||
# to compute a live, invocation-time value from current CPU count, available
|
||||
# memory, and current load average (this sandbox routinely runs multiple
|
||||
# CONCURRENT sessions building/testing at once -- see prune_stale_tmp_fixtures
|
||||
# below) rather than a fixed constant baked into this script; xce-build's own
|
||||
# skill does that computation before invoking this script.
|
||||
#
|
||||
# For meson compile (the ninja backend, and the memory-heaviest step -- ASAN+
|
||||
# debug C/C++ compiles), the value is applied ONLY as ninja's -l LOAD_AVERAGE:
|
||||
# ninja re-checks the system's live load average -- literally its run-queue
|
||||
# length -- before launching EACH new job, throughout the whole build, and
|
||||
# holds off starting more while it's above this ceiling. That is deliberately
|
||||
# NOT the same as also pinning -j to this number: -j is a one-time snapshot
|
||||
# cap decided before the first job starts, while -l keeps adapting for the
|
||||
# entire build, including backing off if some OTHER concurrent session's
|
||||
# build drives load up partway through. -j is left at ninja's own default
|
||||
# (~nproc+2) as a loose ceiling; -l is the actual throttle.
|
||||
#
|
||||
# meson test and go test have no load-average equivalent (meson test's -j/
|
||||
# --num-processes and go test's -p are both flat process-count caps), so for
|
||||
# those the same value is used as a job-count ceiling instead.
|
||||
if [ -n "$JOBS" ] && ! [[ "$JOBS" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "ERROR: --jobs value must be a positive integer, got: '$JOBS'" >&2
|
||||
exit 1
|
||||
fi
|
||||
MESON_COMPILE_JOB_OPTS=()
|
||||
MESON_TEST_JOB_OPTS=()
|
||||
GO_TEST_JOB_OPTS=()
|
||||
if [ -n "$JOBS" ]; then
|
||||
MESON_COMPILE_JOB_OPTS=(-l "$JOBS")
|
||||
MESON_TEST_JOB_OPTS=(-j "$JOBS")
|
||||
GO_TEST_JOB_OPTS=(-p "$JOBS")
|
||||
echo "=== Run-queue-length limit: $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ==="
|
||||
fi
|
||||
|
||||
COMPONENT="${1:-all}"
|
||||
|
||||
# Build PROFILE controls the meson buildtype + sanitizer (and the default build
|
||||
|
|
@ -168,13 +208,20 @@ fi
|
|||
MESON_OPTS+=(-Dcommit_id="$commit_id" -Dbuild_time="$build_time" -Ddirty_fingerprint="$dirty_fingerprint")
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 [component] [profile|build_dir] [build_dir] [--incremental]"
|
||||
echo "Usage: $0 [component] [profile|build_dir] [build_dir] [--incremental] [--jobs N]"
|
||||
echo "Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin"
|
||||
echo "Profiles: asan (default), debug (non-ASAN -O0 -g3, for gdb/cores), release"
|
||||
echo "--incremental: reconfigure the existing build dir in place instead of a full"
|
||||
echo " rm -rf + fresh setup -- preserves other already-built components in the same"
|
||||
echo " dir (bug-1565). Not the default: see the comment at the top of this script"
|
||||
echo " for the staleness tradeoff. Rebuild without it if anything looks wrong."
|
||||
echo "--jobs N: cap this build's run-queue length at N -- meson compile's -l N"
|
||||
echo " (ninja's load-average ceiling, re-checked live before every job launch for"
|
||||
echo " the whole build), meson test's -j N, go test's -p N. No default -- omit to"
|
||||
echo " let ninja/go auto-detect, as before this flag existed. N should be computed"
|
||||
echo " live from current CPU count, available memory, and current load average"
|
||||
echo " (see the xce-build skill), not a fixed constant, since this sandbox often"
|
||||
echo " runs multiple concurrent build/test sessions at once."
|
||||
exit 1
|
||||
}
|
||||
|
||||
|
|
@ -224,25 +271,25 @@ case "$COMPONENT" in
|
|||
lib)
|
||||
setup_build
|
||||
echo "=== Building lib + tests ==="
|
||||
meson compile -C "$BUILD_DIR" xcelib_test
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcelib_test
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running lib tests ==="
|
||||
meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs
|
||||
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" "libxce test" --print-errorlogs
|
||||
fi
|
||||
;;
|
||||
controller)
|
||||
setup_build
|
||||
echo "=== Building lib + controller + tests ==="
|
||||
meson compile -C "$BUILD_DIR" xce_controller controller_unit_test controller/xce_test
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xce_controller controller_unit_test controller/xce_test
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running controller tests ==="
|
||||
meson test -C "$BUILD_DIR" --suite controller --print-errorlogs
|
||||
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite controller --print-errorlogs
|
||||
fi
|
||||
;;
|
||||
scheduler)
|
||||
setup_build
|
||||
echo "=== Building lib + scheduler + tests ==="
|
||||
meson compile -C "$BUILD_DIR" xce_scheduler sched_test
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xce_scheduler sched_test
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running scheduler tests ==="
|
||||
# bug-1559: run the whole 'scheduler' meson suite (meson test
|
||||
|
|
@ -250,7 +297,7 @@ case "$COMPONENT" in
|
|||
# by path -- that bypassed every other scheduler test target
|
||||
# registered under the same suite tag (component-failover-
|
||||
# adoption's 4 task-3.x binaries among them).
|
||||
meson test -C "$BUILD_DIR" --suite scheduler --print-errorlogs
|
||||
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite scheduler --print-errorlogs
|
||||
fi
|
||||
;;
|
||||
worker)
|
||||
|
|
@ -260,28 +307,28 @@ case "$COMPONENT" in
|
|||
# picks whenever /run/systemd/system is absent, e.g. this sandbox) was
|
||||
# missing from this target list -- /xce-build worker alone left it
|
||||
# unbuilt, so deploy-ha.sh soft-skipped generating start-worker.sh.
|
||||
meson compile -C "$BUILD_DIR" xcecd xcecd_worker xcecd_virtual xrun controller_connector_test
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcecd xcecd_worker xcecd_virtual xrun controller_connector_test
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running worker tests ==="
|
||||
meson test -C "$BUILD_DIR" --suite worker --print-errorlogs
|
||||
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite worker --print-errorlogs
|
||||
fi
|
||||
;;
|
||||
apiserver)
|
||||
setup_build
|
||||
echo "=== Building lib + apiserver + xcecli ==="
|
||||
meson compile -C "$BUILD_DIR" apiserver xcecli
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" apiserver xcecli
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running frontend Go unit tests ==="
|
||||
cd frontend/apiserver && CGO_ENABLED=0 go test ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test ./... -count=1
|
||||
cd frontend/apiserver && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1
|
||||
fi
|
||||
;;
|
||||
xcecli)
|
||||
setup_build
|
||||
echo "=== Building xcecli ==="
|
||||
meson compile -C "$BUILD_DIR" xcecli
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcecli
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running xcecli Go unit tests ==="
|
||||
cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1
|
||||
cd frontend/xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1
|
||||
fi
|
||||
;;
|
||||
xceadmin)
|
||||
|
|
@ -302,7 +349,7 @@ case "$COMPONENT" in
|
|||
# steps this target runs).
|
||||
setup_build
|
||||
echo "=== Building + testing xceadmin (via meson) ==="
|
||||
meson compile -C "$BUILD_DIR" xceadmin
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xceadmin
|
||||
WHEEL_FILE=$(ls "$BUILD_DIR"/xceadmin/dist/xceadmin-*.whl 2>/dev/null || true)
|
||||
if [ -n "$WHEEL_FILE" ]; then
|
||||
echo "=== xceadmin wheel: $WHEEL_FILE ==="
|
||||
|
|
@ -312,14 +359,14 @@ case "$COMPONENT" in
|
|||
all)
|
||||
setup_build
|
||||
echo "=== Building all components ==="
|
||||
meson compile -C "$BUILD_DIR"
|
||||
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}"
|
||||
if [ "$RUN_TESTS" = 1 ]; then
|
||||
echo "=== Running all C tests ==="
|
||||
meson test -C "$BUILD_DIR" --print-errorlogs
|
||||
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --print-errorlogs
|
||||
echo "=== Running scheduler test (manual) ==="
|
||||
"./$BUILD_DIR/scheduler/testsuite/sched_test"
|
||||
echo "=== Running frontend Go unit tests ==="
|
||||
cd frontend/apiserver && CGO_ENABLED=0 go test ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test ./... -count=1
|
||||
cd frontend/apiserver && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue