build: split compile -l ceiling from discounted test -j/-p caps

The single JOBS value was doing double duty: the PROCESS_FACTOR
forked-children discount baked into it made sense for meson test -j and
go test -p (flat process caps with no live check), but meson compile
consumes the same number as ninja's -l LOAD-AVERAGE ceiling -- and -l
already re-checks the live run-queue before every launch, so applying the
halved number there too just capped compile parallelism twice. On t1 the
recalibrated 512MB budget produced mem_jobs=9 but cpu_jobs=(8-0.4)*0.5=3,
leaving the memory recalibration without effect on the compile step.

Now: COMPILE_JOBS (raw nproc-minus-load, memory-capped) feeds -l; JOBS
(the same value discounted by PROCESS_FACTOR) feeds the flat test caps.
An explicit --jobs N still sets both, N and N, as before.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-21 15:41:25 +00:00
parent 28ccf8d35c
commit 84f8d6b76a

View file

@ -149,14 +149,14 @@ fi
# those the same value is used as a job-count ceiling instead. # those the same value is used as a job-count ceiling instead.
if [ -n "$JOBS" ]; then if [ -n "$JOBS" ]; then
echo "=== Run-queue-length limit (explicit --jobs): $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ===" echo "=== Run-queue-length limit (explicit --jobs): $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ==="
COMPILE_JOBS="$JOBS"
else else
# Live default -- same formula the /xce-build skill computes by hand: a # Live default -- same formula the /xce-build skill computes by hand: a
# per-profile memory budget per job (ASAN+debug's sanitizer # per-profile memory budget per job (ASAN+debug's sanitizer
# instrumentation and full debug info are the heaviest), reserving ~15% # instrumentation and full debug info are the heaviest), reserving ~15%
# of available memory as headroom, and a per-profile PROCESS_FACTOR # of available memory as headroom. The CPU term is raw nproc-minus-load;
# discounting nproc-minus-load-average for the compiler driver's forked # the forked-children discount is applied separately to the flat test
# cc1/cc1plus + as children (asan/debug keep those resident far longer # caps below (see the COMPILE_JOBS comment), not to the -l ceiling.
# than release's leaner ones).
# #
# asan's 512MB budget (build/de-nix): the two memory hogs the original # asan's 512MB budget (build/de-nix): the two memory hogs the original
# 1536MB figure was calibrated around are both gone -- the 293MB-peak # 1536MB figure was calibrated around are both gone -- the 293MB-peak
@ -179,11 +179,20 @@ else
release) default_per_job_mb=1024; default_process_factor=0.7 ;; release) default_per_job_mb=1024; default_process_factor=0.7 ;;
esac esac
# The memory term is where the per-job budget applies. The CPU term uses
# the RAW nproc-minus-load (no PROCESS_FACTOR discount) because the
# compile step consumes COMPILE_JOBS as ninja's -l LOAD-AVERAGE ceiling
# (see below), which re-checks the live run-queue before every launch and
# backs off by itself -- pre-emptively halving the ceiling for forked
# cc1/as children on top of that would roughly halve t1's compile
# parallelism for no added safety. The discount still matters for meson
# test -j / go test -p (flat caps with no live check), so those two get
# the discounted number.
JOBS="$(awk -v nproc="$default_nproc" -v mem="$default_mem_mb" \ JOBS="$(awk -v nproc="$default_nproc" -v mem="$default_mem_mb" \
-v loadavg="$default_load" -v per_job="$default_per_job_mb" \ -v loadavg="$default_load" -v per_job="$default_per_job_mb" \
-v factor="$default_process_factor" 'BEGIN { 'BEGIN {
mem_jobs = int(mem * 0.85 / per_job) mem_jobs = int(mem * 0.85 / per_job)
cpu_jobs = int((nproc - loadavg) * factor) cpu_jobs = int(nproc - loadavg)
if (cpu_jobs < 1) cpu_jobs = 1 if (cpu_jobs < 1) cpu_jobs = 1
jobs = nproc jobs = nproc
if (mem_jobs < jobs) jobs = mem_jobs if (mem_jobs < jobs) jobs = mem_jobs
@ -191,9 +200,18 @@ else
if (jobs < 1) jobs = 1 if (jobs < 1) jobs = 1
print jobs print jobs
}')" }')"
echo "=== --jobs not passed -- computed live default: nproc=$default_nproc available=${default_mem_mb}MB load=$default_load profile=$PROFILE factor=$default_process_factor -> --jobs $JOBS ===" COMPILE_JOBS="$JOBS"
echo "=== --jobs not passed -- computed live default: nproc=$default_nproc available=${default_mem_mb}MB load=$default_load profile=$PROFILE per_job=${default_per_job_mb}MB -> compile -l $COMPILE_JOBS, test/go -j/-p $JOBS ==="
fi fi
MESON_COMPILE_JOB_OPTS=(-l "$JOBS")
# meson compile gets ninja's -l LOAD-AVERAGE ceiling at COMPILE_JOBS, NOT a
# flat -j: -l re-checks the live run-queue before each launch and backs off
# on its own, so the compile path needs no pre-emptive discount for the
# compiler driver's forked cc1/as children -- ninja's own live check IS that
# safety. meson test / go test have no load-average equivalent, so they get
# the PROCESS_FACTOR-discounted number as flat process-count caps.
: "${COMPILE_JOBS:=$(awk -v j="$JOBS" -v f="${default_process_factor:-0.5}" 'BEGIN { d = int(j * f); if (d < 1) d = 1; print d }')}"
MESON_COMPILE_JOB_OPTS=(-l "$COMPILE_JOBS")
MESON_TEST_JOB_OPTS=(-j "$JOBS") MESON_TEST_JOB_OPTS=(-j "$JOBS")
GO_TEST_JOB_OPTS=(-p "$JOBS") GO_TEST_JOB_OPTS=(-p "$JOBS")