lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(build): cap xce-build.sh's run-queue length via a live-computed --jobs flag

Browse source 
Add --jobs N to xce-build.sh: for meson compile it's passed as ninja's -l N
load-average ceiling (re-checked before every job launch for the whole
build, so it adapts if a concurrent session in this shared sandbox loads
the system mid-build), and as a job-count ceiling (-j/--num-processes,
-p) for meson test/go test, which have no load-average equivalent. No
default -- omitting it preserves prior auto-parallelism behavior exactly.

The xce-build skill now computes N live at invocation time from nproc,
/proc/meminfo's MemAvailable, and /proc/loadavg, rather than a fixed
constant, and passes it through.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DrxU1dSRYNMV9CDTXHFtcR
This commit is contained in:
Rongsong Shen 2026-09-07 10:21:23 +00:00
parent 099dbdabdc
commit 850d186dd8
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
2 changed files with 94 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

89
src/tools/xce-build.sh
View file

@ -22,15 +22,55 @@ set -euo pipefail
# want a sibling component alongside an already-built one in the same dir;
# rebuild without it (a real clean build) if anything looks off afterward.
INCREMENTAL=0
JOBS=""
ARGS=()
for arg in "$@"; do
case "$arg" in
--incremental) INCREMENTAL=1 ;;
*) ARGS+=("$arg") ;;
while [ $# -gt 0 ]; do
case "$1" in
--incremental) INCREMENTAL=1; shift ;;
--jobs) JOBS="${2:-}"; shift 2 ;;
--jobs=*) JOBS="${1#--jobs=}"; shift ;;
*) ARGS+=("$1"); shift ;;
esac
done
set -- "${ARGS[@]}"
# --jobs (run-queue-length limit): opt-in, no default -- when unset,
# meson/ninja/go fall back to their own auto-parallelism (typically nproc or
# nproc+2 workers), exactly as before this flag existed. Callers are expected
# to compute a live, invocation-time value from current CPU count, available
# memory, and current load average (this sandbox routinely runs multiple
# CONCURRENT sessions building/testing at once -- see prune_stale_tmp_fixtures
# below) rather than a fixed constant baked into this script; xce-build's own
# skill does that computation before invoking this script.
#
# For meson compile (the ninja backend, and the memory-heaviest step -- ASAN+
# debug C/C++ compiles), the value is applied ONLY as ninja's -l LOAD_AVERAGE:
# ninja re-checks the system's live load average -- literally its run-queue
# length -- before launching EACH new job, throughout the whole build, and
# holds off starting more while it's above this ceiling. That is deliberately
# NOT the same as also pinning -j to this number: -j is a one-time snapshot
# cap decided before the first job starts, while -l keeps adapting for the
# entire build, including backing off if some OTHER concurrent session's
# build drives load up partway through. -j is left at ninja's own default
# (~nproc+2) as a loose ceiling; -l is the actual throttle.
#
# meson test and go test have no load-average equivalent (meson test's -j/
# --num-processes and go test's -p are both flat process-count caps), so for
# those the same value is used as a job-count ceiling instead.
if [ -n "$JOBS" ] && ! [[ "$JOBS" =~ ^[1-9][0-9]*$ ]]; then
echo "ERROR: --jobs value must be a positive integer, got: '$JOBS'" >&2
exit 1
fi
MESON_COMPILE_JOB_OPTS=()
MESON_TEST_JOB_OPTS=()
GO_TEST_JOB_OPTS=()
if [ -n "$JOBS" ]; then
MESON_COMPILE_JOB_OPTS=(-l "$JOBS")
MESON_TEST_JOB_OPTS=(-j "$JOBS")
GO_TEST_JOB_OPTS=(-p "$JOBS")
echo "=== Run-queue-length limit: $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ==="
fi
COMPONENT="${1:-all}"
# Build PROFILE controls the meson buildtype + sanitizer (and the default build
@ -168,13 +208,20 @@ fi
MESON_OPTS+=(-Dcommit_id="$commit_id" -Dbuild_time="$build_time" -Ddirty_fingerprint="$dirty_fingerprint")
usage() {
echo "Usage: $0 [component] [profile|build_dir] [build_dir] [--incremental]"
echo "Usage: $0 [component] [profile|build_dir] [build_dir] [--incremental] [--jobs N]"
echo "Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin"
echo "Profiles: asan (default), debug (non-ASAN -O0 -g3, for gdb/cores), release"
echo "--incremental: reconfigure the existing build dir in place instead of a full"
echo " rm -rf + fresh setup -- preserves other already-built components in the same"
echo " dir (bug-1565). Not the default: see the comment at the top of this script"
echo " for the staleness tradeoff. Rebuild without it if anything looks wrong."
echo "--jobs N: cap this build's run-queue length at N -- meson compile's -l N"
echo " (ninja's load-average ceiling, re-checked live before every job launch for"
echo " the whole build), meson test's -j N, go test's -p N. No default -- omit to"
echo " let ninja/go auto-detect, as before this flag existed. N should be computed"
echo " live from current CPU count, available memory, and current load average"
echo " (see the xce-build skill), not a fixed constant, since this sandbox often"
echo " runs multiple concurrent build/test sessions at once."
exit 1
}
@ -224,25 +271,25 @@ case "$COMPONENT" in
lib)
setup_build
echo "=== Building lib + tests ==="
meson compile -C "$BUILD_DIR" xcelib_test
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcelib_test
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running lib tests ==="
meson test -C "$BUILD_DIR" "libxce test" --print-errorlogs
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" "libxce test" --print-errorlogs
fi
;;
controller)
setup_build
echo "=== Building lib + controller + tests ==="
meson compile -C "$BUILD_DIR" xce_controller controller_unit_test controller/xce_test
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xce_controller controller_unit_test controller/xce_test
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running controller tests ==="
meson test -C "$BUILD_DIR" --suite controller --print-errorlogs
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite controller --print-errorlogs
fi
;;
scheduler)
setup_build
echo "=== Building lib + scheduler + tests ==="
meson compile -C "$BUILD_DIR" xce_scheduler sched_test
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xce_scheduler sched_test
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running scheduler tests ==="
# bug-1559: run the whole 'scheduler' meson suite (meson test
@ -250,7 +297,7 @@ case "$COMPONENT" in
# by path -- that bypassed every other scheduler test target
# registered under the same suite tag (component-failover-
# adoption's 4 task-3.x binaries among them).
meson test -C "$BUILD_DIR" --suite scheduler --print-errorlogs
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite scheduler --print-errorlogs
fi
;;
worker)
@ -260,28 +307,28 @@ case "$COMPONENT" in
# picks whenever /run/systemd/system is absent, e.g. this sandbox) was
# missing from this target list -- /xce-build worker alone left it
# unbuilt, so deploy-ha.sh soft-skipped generating start-worker.sh.
meson compile -C "$BUILD_DIR" xcecd xcecd_worker xcecd_virtual xrun controller_connector_test
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcecd xcecd_worker xcecd_virtual xrun controller_connector_test
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running worker tests ==="
meson test -C "$BUILD_DIR" --suite worker --print-errorlogs
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --suite worker --print-errorlogs
fi
;;
apiserver)
setup_build
echo "=== Building lib + apiserver + xcecli ==="
meson compile -C "$BUILD_DIR" apiserver xcecli
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" apiserver xcecli
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running frontend Go unit tests ==="
cd frontend/apiserver && CGO_ENABLED=0 go test ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test ./... -count=1
cd frontend/apiserver && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1
fi
;;
xcecli)
setup_build
echo "=== Building xcecli ==="
meson compile -C "$BUILD_DIR" xcecli
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xcecli
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running xcecli Go unit tests ==="
cd frontend/xcecli && CGO_ENABLED=0 go test ./... -count=1
cd frontend/xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1
fi
;;
xceadmin)
@ -302,7 +349,7 @@ case "$COMPONENT" in
# steps this target runs).
setup_build
echo "=== Building + testing xceadmin (via meson) ==="
meson compile -C "$BUILD_DIR" xceadmin
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}" xceadmin
WHEEL_FILE=$(ls "$BUILD_DIR"/xceadmin/dist/xceadmin-*.whl 2>/dev/null || true)
if [ -n "$WHEEL_FILE" ]; then
echo "=== xceadmin wheel: $WHEEL_FILE ==="
@ -312,14 +359,14 @@ case "$COMPONENT" in
all)
setup_build
echo "=== Building all components ==="
meson compile -C "$BUILD_DIR"
meson compile -C "$BUILD_DIR" "${MESON_COMPILE_JOB_OPTS[@]}"
if [ "$RUN_TESTS" = 1 ]; then
echo "=== Running all C tests ==="
meson test -C "$BUILD_DIR" --print-errorlogs
meson test -C "$BUILD_DIR" "${MESON_TEST_JOB_OPTS[@]}" --print-errorlogs
echo "=== Running scheduler test (manual) ==="
"./$BUILD_DIR/scheduler/testsuite/sched_test"
echo "=== Running frontend Go unit tests ==="
cd frontend/apiserver && CGO_ENABLED=0 go test ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test ./... -count=1
cd frontend/apiserver && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./store/... ./utils/... ./controllers/... ./backend/... -count=1 && cd ../xcecli && CGO_ENABLED=0 go test "${GO_TEST_JOB_OPTS[@]}" ./... -count=1
fi
;;
*)