fix(bug-924): prune stale /tmp test-fixture dirs in xce-build.sh

70+ testsuite/ files across controller/scheduler/lib call mkdtemp()
directly with no shared fixture helper, and none clean up after
themselves. Left unchecked this fills the sandbox disk (recurred three
times now: 2026-06-22, 2026-08-26, 2026-08-30, each larger than the
last -- 696GB at its worst), which then surfaces as unrelated
"Expected Non-NULL"-style cascade failures wherever the next test
happens to call mkdtemp()/persist_init(), easily mistaken for a real
regression.

Retrofitting cleanup into 70+ individual test files would be invasive
and risks breaking tests that inspect a fixture's contents across
multiple TEST_ functions in the same file. Every test run in this
project goes through xce-build.sh, so pruning once there -- age-gated
at 3 hours so a concurrent session's in-progress fixtures are never
touched -- is the lowest-risk single choke point.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
This commit is contained in:
Rongsong Shen 2026-08-30 15:07:49 +00:00
parent d8a37fa27a
commit ff9de2e004
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2

View file

@ -178,6 +178,35 @@ usage() {
exit 1 exit 1
} }
# bug-924 (recurred 2026-06-22, 2026-08-26, 2026-08-30 -- each time at a
# larger scale, most recently 16238 dirs / 696G / disk 100% full): every
# ASAN test file in this project's testsuite/ directories that needs a
# scratch directory calls mkdtemp() on its own (70+ files, no shared
# fixture helper to retrofit centrally -- confirmed by grep), and NONE of
# them clean up their own /tmp/xce_* dir afterward. Left unchecked this
# fills the sandbox disk to ENOSPC, which then manifests as unrelated
# "Expected Non-NULL"-style cascade failures in whatever test happens to
# call mkdtemp()/persist_init() next -- easily mistaken for a real
# regression (see bug-924's own history). Retrofitting cleanup into each
# of the 70+ test files individually would be invasive and risks breaking
# tests that inspect a fixture's on-disk contents across multiple TEST_
# functions in the same file. Pruning here instead, once per build
# invocation, is the single lowest-risk choke point: every test run in
# this project goes through this script. Age-gated (not "delete
# everything") so it never touches a fixture another CONCURRENT session's
# test run just created -- this sandbox routinely has multiple sessions
# building/testing at once.
prune_stale_tmp_fixtures() {
local stale
stale="$(find /tmp -maxdepth 1 -name 'xce_*' -mmin +180 2>/dev/null)"
if [ -n "$stale" ]; then
local count
count="$(printf '%s\n' "$stale" | wc -l)"
echo "=== Pruning $count stale (>3h old) /tmp/xce_* test-fixture dir(s) from prior runs (bug-924) ==="
printf '%s\n' "$stale" | xargs -r rm -rf -- 2>/dev/null || true
fi
}
setup_build() { setup_build() {
if [ "$INCREMENTAL" = 1 ] && [ -f "$BUILD_DIR/build.ninja" ]; then if [ "$INCREMENTAL" = 1 ] && [ -f "$BUILD_DIR/build.ninja" ]; then
echo "=== Incremental build: $BUILD_DIR (profile=$PROFILE) — reconfiguring in place, other components' binaries preserved. If anything looks wrong, rebuild without --incremental. ===" echo "=== Incremental build: $BUILD_DIR (profile=$PROFILE) — reconfiguring in place, other components' binaries preserved. If anything looks wrong, rebuild without --incremental. ==="
@ -189,6 +218,8 @@ setup_build() {
fi fi
} }
prune_stale_tmp_fixtures
case "$COMPONENT" in case "$COMPONENT" in
lib) lib)
setup_build setup_build