build: split compile -l ceiling from discounted test -j/-p caps
The single JOBS value was doing double duty: the PROCESS_FACTOR forked-children discount baked into it made sense for meson test -j and go test -p (flat process caps with no live check), but meson compile consumes the same number as ninja's -l LOAD-AVERAGE ceiling -- and -l already re-checks the live run-queue before every launch, so applying the halved number there too just capped compile parallelism twice. On t1 the recalibrated 512MB budget produced mem_jobs=9 but cpu_jobs=(8-0.4)*0.5=3, leaving the memory recalibration without effect on the compile step. Now: COMPILE_JOBS (raw nproc-minus-load, memory-capped) feeds -l; JOBS (the same value discounted by PROCESS_FACTOR) feeds the flat test caps. An explicit --jobs N still sets both, N and N, as before. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
parent
28ccf8d35c
commit
84f8d6b76a
1 changed files with 26 additions and 8 deletions
|
|
@ -149,14 +149,14 @@ fi
|
||||||
# those the same value is used as a job-count ceiling instead.
|
# those the same value is used as a job-count ceiling instead.
|
||||||
if [ -n "$JOBS" ]; then
|
if [ -n "$JOBS" ]; then
|
||||||
echo "=== Run-queue-length limit (explicit --jobs): $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ==="
|
echo "=== Run-queue-length limit (explicit --jobs): $JOBS (meson compile -l load-average ceiling, meson test -j, go test -p) ==="
|
||||||
|
COMPILE_JOBS="$JOBS"
|
||||||
else
|
else
|
||||||
# Live default -- same formula the /xce-build skill computes by hand: a
|
# Live default -- same formula the /xce-build skill computes by hand: a
|
||||||
# per-profile memory budget per job (ASAN+debug's sanitizer
|
# per-profile memory budget per job (ASAN+debug's sanitizer
|
||||||
# instrumentation and full debug info are the heaviest), reserving ~15%
|
# instrumentation and full debug info are the heaviest), reserving ~15%
|
||||||
# of available memory as headroom, and a per-profile PROCESS_FACTOR
|
# of available memory as headroom. The CPU term is raw nproc-minus-load;
|
||||||
# discounting nproc-minus-load-average for the compiler driver's forked
|
# the forked-children discount is applied separately to the flat test
|
||||||
# cc1/cc1plus + as children (asan/debug keep those resident far longer
|
# caps below (see the COMPILE_JOBS comment), not to the -l ceiling.
|
||||||
# than release's leaner ones).
|
|
||||||
#
|
#
|
||||||
# asan's 512MB budget (build/de-nix): the two memory hogs the original
|
# asan's 512MB budget (build/de-nix): the two memory hogs the original
|
||||||
# 1536MB figure was calibrated around are both gone -- the 293MB-peak
|
# 1536MB figure was calibrated around are both gone -- the 293MB-peak
|
||||||
|
|
@ -179,11 +179,20 @@ else
|
||||||
release) default_per_job_mb=1024; default_process_factor=0.7 ;;
|
release) default_per_job_mb=1024; default_process_factor=0.7 ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
# The memory term is where the per-job budget applies. The CPU term uses
|
||||||
|
# the RAW nproc-minus-load (no PROCESS_FACTOR discount) because the
|
||||||
|
# compile step consumes COMPILE_JOBS as ninja's -l LOAD-AVERAGE ceiling
|
||||||
|
# (see below), which re-checks the live run-queue before every launch and
|
||||||
|
# backs off by itself -- pre-emptively halving the ceiling for forked
|
||||||
|
# cc1/as children on top of that would roughly halve t1's compile
|
||||||
|
# parallelism for no added safety. The discount still matters for meson
|
||||||
|
# test -j / go test -p (flat caps with no live check), so those two get
|
||||||
|
# the discounted number.
|
||||||
JOBS="$(awk -v nproc="$default_nproc" -v mem="$default_mem_mb" \
|
JOBS="$(awk -v nproc="$default_nproc" -v mem="$default_mem_mb" \
|
||||||
-v loadavg="$default_load" -v per_job="$default_per_job_mb" \
|
-v loadavg="$default_load" -v per_job="$default_per_job_mb" \
|
||||||
-v factor="$default_process_factor" 'BEGIN {
|
'BEGIN {
|
||||||
mem_jobs = int(mem * 0.85 / per_job)
|
mem_jobs = int(mem * 0.85 / per_job)
|
||||||
cpu_jobs = int((nproc - loadavg) * factor)
|
cpu_jobs = int(nproc - loadavg)
|
||||||
if (cpu_jobs < 1) cpu_jobs = 1
|
if (cpu_jobs < 1) cpu_jobs = 1
|
||||||
jobs = nproc
|
jobs = nproc
|
||||||
if (mem_jobs < jobs) jobs = mem_jobs
|
if (mem_jobs < jobs) jobs = mem_jobs
|
||||||
|
|
@ -191,9 +200,18 @@ else
|
||||||
if (jobs < 1) jobs = 1
|
if (jobs < 1) jobs = 1
|
||||||
print jobs
|
print jobs
|
||||||
}')"
|
}')"
|
||||||
echo "=== --jobs not passed -- computed live default: nproc=$default_nproc available=${default_mem_mb}MB load=$default_load profile=$PROFILE factor=$default_process_factor -> --jobs $JOBS ==="
|
COMPILE_JOBS="$JOBS"
|
||||||
|
echo "=== --jobs not passed -- computed live default: nproc=$default_nproc available=${default_mem_mb}MB load=$default_load profile=$PROFILE per_job=${default_per_job_mb}MB -> compile -l $COMPILE_JOBS, test/go -j/-p $JOBS ==="
|
||||||
fi
|
fi
|
||||||
MESON_COMPILE_JOB_OPTS=(-l "$JOBS")
|
|
||||||
|
# meson compile gets ninja's -l LOAD-AVERAGE ceiling at COMPILE_JOBS, NOT a
|
||||||
|
# flat -j: -l re-checks the live run-queue before each launch and backs off
|
||||||
|
# on its own, so the compile path needs no pre-emptive discount for the
|
||||||
|
# compiler driver's forked cc1/as children -- ninja's own live check IS that
|
||||||
|
# safety. meson test / go test have no load-average equivalent, so they get
|
||||||
|
# the PROCESS_FACTOR-discounted number as flat process-count caps.
|
||||||
|
: "${COMPILE_JOBS:=$(awk -v j="$JOBS" -v f="${default_process_factor:-0.5}" 'BEGIN { d = int(j * f); if (d < 1) d = 1; print d }')}"
|
||||||
|
MESON_COMPILE_JOB_OPTS=(-l "$COMPILE_JOBS")
|
||||||
MESON_TEST_JOB_OPTS=(-j "$JOBS")
|
MESON_TEST_JOB_OPTS=(-j "$JOBS")
|
||||||
GO_TEST_JOB_OPTS=(-p "$JOBS")
|
GO_TEST_JOB_OPTS=(-p "$JOBS")
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue