From 850d186dd8c9cd5d1dfeb7e0b86199a2c2ad4427 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 7 Sep 2026 10:21:23 +0000 Subject: [PATCH] feat(build): cap xce-build.sh's run-queue length via a live-computed --jobs flag Add --jobs N to xce-build.sh: for meson compile it's passed as ninja's -l N load-average ceiling (re-checked before every job launch for the whole build, so it adapts if a concurrent session in this shared sandbox loads the system mid-build), and as a job-count ceiling (-j/--num-processes, -p) for meson test/go test, which have no load-average equivalent. No default -- omitting it preserves prior auto-parallelism behavior exactly. The xce-build skill now computes N live at invocation time from nproc, /proc/meminfo's MemAvailable, and /proc/loadavg, rather than a fixed constant, and passes it through. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01DrxU1dSRYNMV9CDTXHFtcR --- .claude/skills/xce-build/SKILL.md | 31 +++++++++-- src/tools/xce-build.sh | 89 +++++++++++++++++++++++-------- 2 files changed, 94 insertions(+), 26 deletions(-) diff --git a/.claude/skills/xce-build/SKILL.md b/.claude/skills/xce-build/SKILL.md index 9847083f..7ffa7c42 100644 --- a/.claude/skills/xce-build/SKILL.md +++ b/.claude/skills/xce-build/SKILL.md @@ -1,6 +1,6 @@ --- name: xce-build -description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release." +description: "Build XCE components with tests. Usage: /xce-build [component] [profile] [build_dir] [--incremental]. Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin. Profiles: asan (default), debug (non-ASAN -O0 -g3 for gdb/cores), release. Automatically caps build/test parallelism (--jobs) to the CPU/memory/load available at invocation time." disable-model-invocation: false allowed-tools: Bash(*) --- @@ -9,6 +9,27 @@ allowed-tools: Bash(*) Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed as free text after `/xce-build`, e.g. `/xce-build controller debug`. `--incremental` can appear anywhere in the words (it's stripped out before the rest are parsed positionally — see below). Parse the ARGUMENTS text yourself and run the command below with the parsed values inline. +## Run-queue-length limit (`--jobs N`) — compute it live before every build, don't skip this + +`xce-build.sh` accepts a `--jobs N` flag that caps the build's run-queue length. For `meson compile` (the ninja backend, and the memory-heaviest step — ASAN+debug C/C++ compiles) it's passed as **`-l N`, ninja's load-average ceiling** — ninja re-checks the system's live load average, literally its run-queue length, before launching *every* new job for the *whole* build, and holds off while it's above N. That's the actual control lever, deliberately not paired with pinning `-j` to the same number: `-j` would just be a one-time snapshot cap decided before the first job starts, whereas `-l` keeps adapting for the entire build — including backing off if some *other* concurrent session's build drives load up partway through (this sandbox routinely runs multiple sessions building at once). `-j` is left at ninja's own default (~nproc+2) as a loose ceiling. `meson test` and `go test` have no load-average equivalent (only flat process-count knobs), so N is used there as a job-count ceiling instead — `-j`/`--num-processes N` for `meson test`, `-p N` for `go test`. The flag has no default — if you omit it, everything falls back to its own auto-parallelism, same as before this flag existed. + +**Always compute N yourself, live, immediately before launching the build** — from the hardware/load state *at the moment the skill is invoked*, not a number remembered from an earlier run or a fixed constant. This sandbox routinely runs multiple concurrent sessions building/testing at once (see `xce-build.sh`'s own `prune_stale_tmp_fixtures` comment), so a stale or hardcoded value can either starve this build or overload a system something else is already using. + +1. Read current hardware/load state: + ```bash + nproc --all + awk '/MemAvailable/ {print int($2/1024)}' /proc/meminfo # available memory, MB (reclaimable-aware, unlike `free`'s raw "free" column) + cut -d' ' -f1 /proc/loadavg # current 1-min load average = existing run-queue length from other activity + ``` +2. Pick a per-job memory budget by profile — ASAN+debug is the heaviest (debug info + sanitizer instrumentation per translation unit): **~1536 MB/job for `asan`** (the default profile), **~1024 MB/job for `debug`/`release`**. +3. Compute, reserving ~15% of available memory as headroom for the OS and other sessions: + - `mem_jobs = floor(available_mem_mb * 0.85 / per_job_mb)` + - `cpu_jobs = max(1, floor(nproc - current_load_avg))` — leaves room for whatever's already running + - `JOBS = max(1, min(nproc, mem_jobs, cpu_jobs))` +4. State the computed value and its inputs in one line (e.g. `nproc=22, available=20466MB, load=1.2 → --jobs 14`) before launching the build, then pass `--jobs ` to `xce-build.sh` alongside the other args. + +This is a judgment call, not a rigid formula — if you notice other heavy build processes already running (`ps aux | grep -E 'cc1plus|ninja|go build'`) or the available-memory read looks unusually low, bias further down. + - **No `context: fork`, intentionally.** It was tried (twice) and reproducibly breaks this skill: whenever ARGUMENTS is non-empty (e.g. `controller`), the forked subagent receives no usable task — it responds as if handed only ambient project context, never runs the build, and reports nothing back. Bare `/xce-build` (no args) happens to still "succeed" under `context: fork` only because the empty-arg case is indistinguishable from no task at all failing open into some default — not because forking works. Do not re-add `context: fork` here without first confirming the harness actually threads a non-empty ARGUMENTS payload into the forked subagent's prompt. - **component** (1st word, default `all`) — `all`, `lib`, `controller`, `scheduler`, `worker`, `apiserver`, `xcecli`, `xceadmin` - **profile** (2nd word, default `asan`) — `asan` (debug + AddressSanitizer → `xce.build`), `debug` (non-ASAN `-O0 -g3 -fno-inline -fno-omit-frame-pointer` → `build.dbg`), `release` (`-O3`, no sanitizer → `build.release`). Each profile uses its own default build dir so the variants never clobber each other. If the 2nd word isn't one of these three keywords, treat it as a legacy explicit build_dir instead (3rd word, if present, always overrides the build dir). @@ -19,13 +40,13 @@ Args: `[component] [profile|build_dir] [build_dir] [--incremental]` — passed a Examples: `/xce-build controller` (ASAN), `/xce-build controller debug` (gdb/core build), `/xce-build all release`, `/xce-build xceadmin` (build + test + wheel, any profile arg is accepted and ignored), `/xce-build xceadmin --incremental` (same, but keeps other components already built in the same dir). -Run (substituting the parsed component/profile/build_dir; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested): +Run (substituting the parsed component/profile/build_dir, and the `--jobs N` value computed per the section above; omit trailing empty ones; append `--incremental` at the end, anywhere, when requested): ```bash -nix develop ./src --command bash src/tools/xce-build.sh [--incremental] +nix develop ./src --command bash src/tools/xce-build.sh --jobs [--incremental] ``` -e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xce-build.sh controller`. For `/xce-build xceadmin --incremental`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --incremental`. +e.g. for `/xce-build controller` with a computed `N=14`: `nix develop ./src --command bash src/tools/xce-build.sh controller --jobs 14`. For `/xce-build xceadmin --incremental` with `N=8`: `nix develop ./src --command bash src/tools/xce-build.sh xceadmin --jobs 8 --incremental` (`--jobs` and `--incremental` are flags, stripped out from any position before the remaining words are parsed positionally — they don't need to follow profile/build_dir). ## Run the build via Bash `run_in_background`, not an Agent-tool subagent @@ -34,7 +55,7 @@ e.g. for `/xce-build controller`: `nix develop ./src --command bash src/tools/xc Instead, run the resolved command directly with the **Bash tool's own `run_in_background: true`**, from the current session, and let the harness's own task-notification mechanism — not a subagent's self-report — tell you when it's done. This has been empirically proven reliable in this exact repo (2026-09-03: both a full `all debug` build and a `controller` ASAN rebuild were driven this way, and each genuinely blocked until real completion). ```bash -nix develop ./src --command bash src/tools/xce-build.sh [--incremental] \ +nix develop ./src --command bash src/tools/xce-build.sh --jobs [--incremental] \ > /xce-build--.log 2>&1; \ echo "EXIT_CODE=$?" >> /xce-build--.log ``` diff --git a/src/tools/xce-build.sh b/src/tools/xce-build.sh index 5d6e22d5..30ee62eb 100755 --- a/src/tools/xce-build.sh +++ b/src/tools/xce-build.sh @@ -22,15 +22,55 @@ set -euo pipefail # want a sibling component alongside an already-built one in the same dir; # rebuild without it (a real clean build) if anything looks off afterward. INCREMENTAL=0 +JOBS="" ARGS=() -for arg in "$@"; do - case "$arg" in - --incremental) INCREMENTAL=1 ;; - *) ARGS+=("$arg") ;; +while [ $# -gt 0 ]; do + case "$1" in + --incremental) INCREMENTAL=1; shift ;; + --jobs) JOBS="${2:-}"; shift 2 ;; + --jobs=*) JOBS="${1#--jobs=}"; shift ;; + *) ARGS+=("$1"); shift ;; esac done set -- "${ARGS[@]}" +# --jobs (run-queue-length limit): opt-in, no default -- when unset, +# meson/ninja/go fall back to their own auto-parallelism (typically nproc or +# nproc+2 workers), exactly as before this flag existed. Callers are expected +# to compute a live, invocation-time value from current CPU count, available +# memory, and current load average (this sandbox routinely runs multiple +# CONCURRENT sessions building/testing at once -- see prune_stale_tmp_fixtures +# below) rather than a fixed constant baked into this script; xce-build's own +# skill does that computation before invoking this script. +# +# For meson compile (the ninja backend, and the memory-heaviest step -- ASAN+ +# debug C/C++ compiles), the value is applied ONLY as ninja's -l LOAD_AVERAGE: +# ninja re-checks the system's live load average -- literally its run-queue +# length -- before launching EACH new job, throughout the whole build, and +# holds off starting more while it's above this ceiling. That is deliberately +# NOT the same as also pinning -j to this number: -j is a one-time snapshot +# cap decided before the first job starts, while -l keeps adapting for the +# entire build, including backing off if some OTHER concurrent session's +# build drives load up partway through. -j is left at ninja's own default +# (~nproc+2) as a loose ceiling; -l is the actual throttle. +# +# meson test and go test have no load-average equivalent (meson test's -j/ +# --num-processes and go test's -p are both flat process-count caps), so for +# those the same value is used as a job-count ceiling instead. +if [ -n "$JOBS" ] && ! [[ "$JOBS" =~ ^[1-9][0-9]*$ ]]; then + echo "ERROR: --jobs value must be a positive integer, got: '$JOBS'" >&2 + exit 1 +fi +MESON_COMPILE_JOB_OPTS=() +MESON_TEST_JOB_OPTS=() +GO_TEST_JOB_OPTS=() +if [ -n "$JOBS" ]; then + MESON_COMPILE_JOB_OPTS=(-l "$JOBS") + MESON_TEST_JOB_OPTS=(-j "$JOBS") + GO_TEST_JOB_OPTS=(-p "$JOBS") + echo "=== Run-queue-length limit: $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ===" +fi + COMPONENT="${1:-all}" # Build PROFILE controls the meson buildtype + sanitizer (and the default build @@ -168,13 +208,20 @@ fi MESON_OPTS+=(-Dcommit_id="$commit_id" -Dbuild_time="$build_time" -Ddirty_fingerprint="$dirty_fingerprint") usage() { - echo "Usage: $0 [component] [profile|build_dir] [build_dir] [--incremental]" + echo "Usage: $0 [component] [profile|build_dir] [build_dir] [--incremental] [--jobs N]" echo "Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin" echo "Profiles: asan (default), debug (non-ASAN -O0 -g3, for gdb/cores), release" echo "--incremental: reconfigure the existing build dir in place instead of a full" echo " rm -rf + fresh setup -- preserves other already-built components in the same" echo " dir (bug-1565). Not the default: see the comment at the top of this script" echo " for the staleness tradeoff. Rebuild without it if anything looks wrong." + echo "--jobs N: cap this build's run-queue length at N -- meson compile's -l N" + echo " (ninja's load-average ceiling, re-checked live before every job launch for" + echo " the whole build), meson test's -j N, go test's -p N. No default -- omit to" + echo " let ninja/go auto-detect, as before this flag existed. N should be computed" + echo " live from current CPU count, available memory, and current load average" + echo " (see the xce-build skill), not a fixed constant, since this sandbox often" + echo " runs multiple concurrent build/test sessions at once." exit 1 } @@ -224,25 +271,25 @@ case "$COMPONENT" in lib) setup_build echo "=== Building lib + tests ===" - meson compile -C "$BUILD_DIR" xcelib_test + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcelib_test if [ "$RUN_TESTS" = 1 ]; then echo "=== Running lib tests ===" - meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs + meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" "libxce test" --print-errorlogs fi ;; controller) setup_build echo "=== Building lib + controller + tests ===" - meson compile -C "$BUILD_DIR" xce_controller controller_unit_test controller/xce_test + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xce_controller controller_unit_test controller/xce_test if [ "$RUN_TESTS" = 1 ]; then echo "=== Running controller tests ===" - meson test -C "$BUILD_DIR" --suite controller --print-errorlogs + meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite controller --print-errorlogs fi ;; scheduler) setup_build echo "=== Building lib + scheduler + tests ===" - meson compile -C "$BUILD_DIR" xce_scheduler sched_test + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xce_scheduler sched_test if [ "$RUN_TESTS" = 1 ]; then echo "=== Running scheduler tests ===" # bug-1559: run the whole 'scheduler' meson suite (meson test @@ -250,7 +297,7 @@ case "$COMPONENT" in # by path -- that bypassed every other scheduler test target # registered under the same suite tag (component-failover- # adoption's 4 task-3.x binaries among them). - meson test -C "$BUILD_DIR" --suite scheduler --print-errorlogs + meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite scheduler --print-errorlogs fi ;; worker) @@ -260,28 +307,28 @@ case "$COMPONENT" in # picks whenever /run/systemd/system is absent, e.g. this sandbox) was # missing from this target list -- /xce-build worker alone left it # unbuilt, so deploy-ha.sh soft-skipped generating start-worker.sh. - meson compile -C "$BUILD_DIR" xcecd xcecd_worker xcecd_virtual xrun controller_connector_test + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcecd xcecd_worker xcecd_virtual xrun controller_connector_test if [ "$RUN_TESTS" = 1 ]; then echo "=== Running worker tests ===" - meson test -C "$BUILD_DIR" --suite worker --print-errorlogs + meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite worker --print-errorlogs fi ;; apiserver) setup_build echo "=== Building lib + apiserver + xcecli ===" - meson compile -C "$BUILD_DIR" apiserver xcecli + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" apiserver xcecli if [ "$RUN_TESTS" = 1 ]; then echo "=== Running frontend Go unit tests ===" - cd frontend/apiserver && CGO_ENABLED=0 go test ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test ./... -count=1 + cd frontend/apiserver && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1 fi ;; xcecli) setup_build echo "=== Building xcecli ===" - meson compile -C "$BUILD_DIR" xcecli + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcecli if [ "$RUN_TESTS" = 1 ]; then echo "=== Running xcecli Go unit tests ===" - cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1 + cd frontend/xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1 fi ;; xceadmin) @@ -302,7 +349,7 @@ case "$COMPONENT" in # steps this target runs). setup_build echo "=== Building + testing xceadmin (via meson) ===" - meson compile -C "$BUILD_DIR" xceadmin + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xceadmin WHEEL_FILE=$(ls "$BUILD_DIR"/xceadmin/dist/xceadmin-*.whl 2>/dev/null || true) if [ -n "$WHEEL_FILE" ]; then echo "=== xceadmin wheel: $WHEEL_FILE ===" @@ -312,14 +359,14 @@ case "$COMPONENT" in all) setup_build echo "=== Building all components ===" - meson compile -C "$BUILD_DIR" + meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" if [ "$RUN_TESTS" = 1 ]; then echo "=== Running all C tests ===" - meson test -C "$BUILD_DIR" --print-errorlogs + meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --print-errorlogs echo "=== Running scheduler test (manual) ===" "./$BUILD_DIR/scheduler/testsuite/sched_test" echo "=== Running frontend Go unit tests ===" - cd frontend/apiserver && CGO_ENABLED=0 go test ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test ./... -count=1 + cd frontend/apiserver && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1 fi ;; *)