bug-1971: preserve flaky test-failure evidence across xce-build.sh runs
A failed test run's evidence (meson-logs/testlog.txt + the mkdtemp sweep's FAILED-case /tmp/xce_* fixture dirs) was destroyed by the next build: setup_build rm -rf's the build dir, and any subsequent meson test overwrites testlog.* in place. t2 2026-10-22: a controller case failed at ~05:46, the make clean + rerun at ~05:52 left 134/134 OK and zero trace of what failed. Two chokes in xce-build.sh: - finalize_failure_evidence (EXIT trap): on rc!=0 with RUN_TESTS=1, copy meson-logs to testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/ and drop the .meson_test_failed_rc marker; also releases the build lock (the old lock-release trap folded in). - archive_previous_failure (run start, pre-wipe): if the marker exists, move the previous run's meson-logs to prev-<stamp>-rc<N>/ -- outside the build dir, unreachable by the wipe; sweep aged (>5 min) /tmp/xce_* FAILED-case fixture dirs into prev-<stamp>-fixtures/ so they sit with the testlog that explains them; cap the archive at newest 10 per build dir. Also -type d on BOTH /tmp/xce_* sweeps (this one and bug-924's prune_stale_tmp_fixtures): mkdtemp fixtures are always directories, but the bare name also matches regular files -- the untyped find live-moved daemon debug logs (xce_scheduler.log, xce_strace*.log) into the archive, and the bug-924 variant rm -rf's its matches. Test: src/tools/test-failure-evidence-archive.sh, 6 scenarios / 11 assertions exercising the real function bodies (sourced with surgical stubbing) -- all green. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
e8cb6d5743
commit
68c3ccd773
3 changed files with 464 additions and 3 deletions
95
.wolf/bugs/bug-1971.md
Normal file
95
.wolf/bugs/bug-1971.md
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
---
|
||||
id: "bug-1971"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-22"
|
||||
status_note: "two-choke archive in xce-build.sh; test harness green"
|
||||
component: "build"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-22T07:50:00Z"
|
||||
last_seen: "2026-10-22T07:50:00Z"
|
||||
occurrences: 2
|
||||
files: ["src/tools/xce-build.sh"]
|
||||
tags: ["test", "flaky", "observability", "build"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1971: next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable
|
||||
|
||||
## Description
|
||||
**Observed:** (t2, 2026-10-22) a full `make build` (asan profile) failed on a controller test case; a following `make clean` + `make build` passed 134/134. The failing run's `meson-logs/testlog.{txt,json,junit.xml}` had been destroyed twice over: `setup_build`'s unconditional `rm -rf "$BUILD_DIR"` deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites `testlog.*` on every `meson test` invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (`/tmp/test_xcecd_resync.log` and rotations) showed no failure signature, and no `sched_test`/Go test output is retained anywhere.
|
||||
|
||||
Reconstruction from filesystem timestamps alone (`/tmp/test_*.log` rotations, `meson-log.txt` build options line, `/usr/local/go` ctime, bash history) established the two-run timeline but could not name the failed case.
|
||||
|
||||
The same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.
|
||||
|
||||
**Expected:** when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' `/tmp/xce_*` dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.
|
||||
|
||||
**Impact:** intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp "failed cases keep their dirs for post-mortem" contract, since the next `make build` never looks for those dirs and a clean build gives the appearance of none existing.
|
||||
|
||||
**Found by:** user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.
|
||||
|
||||
## Environment
|
||||
- OS: Ubuntu 24.04.2 LTS (t1 and t2); kernel Linux 6.8.0-139-generic x86_64
|
||||
- libc: glibc 2.39 (host ldd)
|
||||
- Toolchain: gcc 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1), meson 1.12.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: build/de-nix @ 5b9d7270 (t2 checkout), e8cb6d57 (t1)
|
||||
- Deployment: n/a — dev-loop build/test on t1 (workstation) and t2 (8-core/8GB test node)
|
||||
|
||||
## Reproduction
|
||||
1. On any machine: `make build` (or `bash src/tools/xce-build.sh`) and let the test phase fail (e.g. by patching a test to fail).
|
||||
2. Note `src/xce.build/meson-logs/testlog.txt` contains the failure.
|
||||
3. Run `make clean && make build` again: the previous testlog is gone (build dir was rm -rf'd), and even without `make clean`, a successful `meson test` run overwrites testlog.* in place.
|
||||
- Frequency: deterministic — every subsequent run destroys the prior run's evidence.
|
||||
- Evidence: t2 `/tmp/myxce` 2026-10-22: failing run ~05:46, passing rerun 05:52 (its testlog survives at `/tmp/myxce/src/xce.build/meson-logs/testlog.txt`, 134/134 OK); the failing run's testlog was deleted by the `make clean` at ~05:5x.
|
||||
|
||||
## Root cause
|
||||
`setup_build()` in `src/tools/xce-build.sh` unconditionally `rm -rf "$BUILD_DIR"` on every non-incremental invocation, and `meson test` overwrites `$BUILD_DIR/meson-logs/testlog.*` on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.
|
||||
|
||||
## Fix
|
||||
Two chokes in `src/tools/xce-build.sh` (this commit):
|
||||
|
||||
1. `archive_previous_failure()` — runs at script start, before `setup_build`'s
|
||||
wipe. If the marker `.meson_test_failed_rc` (written by the previous run's
|
||||
EXIT trap) exists, the previous run's `meson-logs` are MOVED to
|
||||
`$SRC_DIR/testlogs.archive/<BUILD_DIR>/prev-<stamp>-rc<N>/` — outside the
|
||||
build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e.
|
||||
not a concurrent session's) `/tmp/xce_*` FAILED-case fixture DIRS into
|
||||
`prev-<stamp>-fixtures/`, and caps the archive at the newest 10 entries
|
||||
per build dir.
|
||||
2. `finalize_failure_evidence()` — the EXIT trap (replaces the old
|
||||
lock-release trap, which becomes its last statement). On rc≠0 with
|
||||
RUN_TESTS=1, COPIES `meson-logs` to
|
||||
`testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/` and writes the marker.
|
||||
Copy-not-move: meson-logs still belongs to the live build dir for this
|
||||
run's own inspection; the next run moves the copy out at its start.
|
||||
Configure/compile failures (RUN_TESTS unset) archive nothing — those are
|
||||
deterministic and self-explanatory on rerun; it's the testlog-bearing
|
||||
flaky failures that need preservation.
|
||||
|
||||
Additionally `-type d` added to BOTH `/tmp/xce_*` sweeps (this one and
|
||||
bug-924's `prune_stale_tmp_fixtures`): mkdtemp fixtures are always
|
||||
directories (`/tmp/xce_<label>_XXXXXX`), but the bare name also matches
|
||||
regular files — the untyped find live-moved daemon debug logs
|
||||
(`/tmp/xce_scheduler.log`, `/tmp/xce_strace*.log`) into the archive on
|
||||
2026-10-22, and the bug-924 variant `rm -rf`s its matches.
|
||||
|
||||
## Verification
|
||||
- Test: `src/tools/test-failure-evidence-archive.sh` — 6 scenarios /
|
||||
11 assertions exercising the real function bodies (sourced with surgical
|
||||
stubbing): failed run archives as this-*-rc3 + marker; success archives
|
||||
nothing; next run moves prior failure to prev-*-rc3 + consumes marker;
|
||||
aged fixture dirs move / fresh stay / regular files never touched;
|
||||
retention caps at newest 10; RUN_TESTS=0 archives nothing. All green
|
||||
(11 passed, 0 failed, bash, 2026-10-22).
|
||||
- Suites: n/a — this bug lives in the build script, not in C code; no meson
|
||||
suite exercises it. `bash -n` syntax check green.
|
||||
- Real wipe path not exercised by the harness test (it stubs meson);
|
||||
validated by inspection: the archive root is `$SRC_DIR/testlogs.archive/`,
|
||||
which `rm -rf "$BUILD_DIR"` cannot reach.
|
||||
- Live: first t2/sandbox failing run after this commit leaves
|
||||
`testlogs.archive/xce.build/this-*-rc*` with the testlog — that is the
|
||||
evidence this bug asks for.
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
239
src/tools/test-failure-evidence-archive.sh
Normal file
239
src/tools/test-failure-evidence-archive.sh
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
#!/bin/bash
|
||||
# test-failure-evidence-archive.sh — bug-1971: a failed xce-build.sh run must
|
||||
# leave its meson-logs somewhere the NEXT run's wipe cannot destroy, and the
|
||||
# next run must move the previous failure's evidence (testlog + the mkdtemp
|
||||
# sweep's FAILED-case /tmp/xce_* fixture dirs) into the timestamped archive
|
||||
# before setup_build's rm -rf.
|
||||
#
|
||||
# Exercises src/tools/xce-build.sh's archive functions directly (sourced),
|
||||
# so no real build is needed:
|
||||
# 1. a failed run's EXIT trap archives meson-logs + writes the rc marker
|
||||
# 2. a successful run's EXIT trap archives nothing
|
||||
# 3. archive_previous_failure moves a previous failure's meson-logs out of
|
||||
# the build dir and consumes the marker (a later wipe can't destroy it)
|
||||
# 4. leftover /tmp/xce_* fixture DIRS (>5 min old) are moved into the
|
||||
# archive; fresh ones (a concurrent session's) are left alone; regular
|
||||
# files matching xce_* (daemon debug logs are not fixtures) are never
|
||||
# touched
|
||||
# 5. retention: only the newest 10 archive entries per build dir survive
|
||||
# 6. a non-test failure (RUN_TESTS=0, e.g. compile error) archives nothing
|
||||
#
|
||||
# Usage:
|
||||
# bash src/tools/test-failure-evidence-archive.sh
|
||||
set -u
|
||||
|
||||
TOOLS=$(cd "$(dirname "$0")" && pwd)
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
green() { printf '\033[32m%s\033[0m\n' "$*"; }
|
||||
red() { printf '\033[31m%s\033[0m\n' "$*"; }
|
||||
|
||||
pass() { green " PASS: $*"; PASS=$((PASS+1)); }
|
||||
fail() { red " FAIL: $*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
WORK=$(mktemp -d)
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
|
||||
# The extracted slice starts with xce-build.sh's `set -euo pipefail`, which
|
||||
# would make rc_probe's `return 3` abort THIS script mid-test. Neutralize it
|
||||
# right after sourcing: this test drives the failure paths itself and does
|
||||
# not want -e's early exit (the real script keeps its own set -e line --
|
||||
# only the test copy sheds it).
|
||||
STRICT_SAVED_OPTS="$-"
|
||||
|
||||
# Source the build script's functions without running it: xce-build.sh is a
|
||||
# straight-through script (no main guard), so sourcing would execute it.
|
||||
# Extract the slice that matters: from the top of the file through the
|
||||
# definitions of archive_root / archive_previous_failure /
|
||||
# finalize_failure_evidence, stopping at the bare `archive_previous_failure`
|
||||
# call (everything after that — the component case dispatch — is not under
|
||||
# test). The extracted slice still contains the arg parsing, the toolchain
|
||||
# preflight, the --jobs live-default block and the build-lock sourcing; stub
|
||||
# those out with targeted sed removals so the test exercises ONLY the real
|
||||
# function bodies (if someone edits them, this test sees the edit).
|
||||
CUT=$(grep -n '^archive_previous_failure$' "$TOOLS/xce-build.sh" | head -1 | cut -d: -f1)
|
||||
if [ -z "$CUT" ]; then
|
||||
fail "cannot locate archive_previous_failure call in xce-build.sh (structure changed?)"
|
||||
echo "Results: ${PASS} passed, ${FAIL} failed"
|
||||
exit 1
|
||||
fi
|
||||
head -n "$CUT" "$TOOLS/xce-build.sh" > "$WORK/functions.sh"
|
||||
|
||||
python3 - "$WORK/functions.sh" <<'PYEOF'
|
||||
import re, sys
|
||||
p = sys.argv[1]
|
||||
s = open(p).read()
|
||||
# 1. toolchain preflight: fake every tool as present
|
||||
s = re.sub(r"for t in meson ninja[^)]*\).*?^fi\n", "", s, flags=re.M | re.S)
|
||||
# 2. pkg-config system-library preflight block: drop entirely
|
||||
s = re.sub(r"missing_pkgs=\(\)\nwhile IFS= read -r line.*?^fi\n", "", s, flags=re.M | re.S)
|
||||
# 3. --jobs parsing/validation + live-default computation: not under test.
|
||||
# Cut from the INCREMENTAL=0 marker through the MESON_TEST_JOB_OPTS line
|
||||
# INCLUSIVE -- but no further: the archive function definitions below it
|
||||
# must survive (a `.*$` tail here would eat them). GO_TEST_JOB_OPTS (one
|
||||
# line after the cut point's sibling MESON_* assignment) still references
|
||||
# $JOBS -- stub it too; the test never passes Go opts anywhere.
|
||||
s = re.sub(r"INCREMENTAL=0\nJOBS=\"\"\nARGS=\(\).*?^MESON_TEST_JOB_OPTS=\([^\n]*\)\n", "", s, flags=re.M | re.S)
|
||||
s = re.sub(r"^GO_TEST_JOB_OPTS=\(-p \"\$JOBS\"\)\n", "GO_TEST_JOB_OPTS=()\n", s, flags=re.M)
|
||||
# 4. profile case: provide RUN_TESTS/BUILD_DIR assignment via test env instead
|
||||
s = re.sub(r"case \"\$\{2:-\}\" in.*?^esac\n", "", s, flags=re.M | re.S)
|
||||
s = re.sub(r"if \[ -z \"\$\{BUILD_DIR:-\}\" \]; then.*?^fi\n", "", s, flags=re.M | re.S)
|
||||
s = re.sub(r"# RUN_TESTS: the unit suites.*?^esac\n", "", s, flags=re.M | re.S)
|
||||
# 5. build-lock sourcing: the test stubs the lock itself
|
||||
s = s.replace('. "$SCRIPT_DIR/build-lock.sh"', ': # build-lock not under test')
|
||||
s = s.replace('build_lock_acquire "$BUILD_DIR"', ': # lock not under test')
|
||||
s = s.replace('build_lock_release', ': # lock not under test')
|
||||
s = s.replace('trap finalize_failure_evidence EXIT', ': # trap installed per-test')
|
||||
# 6. the bare pre-dispatch calls + component dispatch tail: drop
|
||||
s = s.replace('prune_stale_tmp_fixtures\n', ': # prune not under test\n')
|
||||
s = s.replace('archive_previous_failure\n', ': # called per-test\n')
|
||||
s = s.replace('cd "$SRC_DIR"\n', ': # cd per-test\n')
|
||||
open(p, "w").write(s)
|
||||
PYEOF
|
||||
|
||||
# Any stray execute-side reference left by the surgery would print here —
|
||||
# catch it loudly instead of silently passing with the real script never run.
|
||||
if bash -n "$WORK/functions.sh" 2>&1 | grep -q .; then
|
||||
fail "extracted functions.sh has a syntax error after stubbing"
|
||||
bash -n "$WORK/functions.sh"
|
||||
echo "Results: ${PASS} passed, ${FAIL} failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
. "$WORK/functions.sh"
|
||||
set +e # undo the sourced slice's `set -euo pipefail` (see note above)
|
||||
set +u
|
||||
set +o pipefail
|
||||
|
||||
SRC_DIR="$WORK/src"
|
||||
mkdir -p "$SRC_DIR"
|
||||
|
||||
echo "── 1. failed run (RUN_TESTS=1): EXIT trap archives meson-logs + writes marker ──"
|
||||
# Simulate what the EXIT trap sees when the script dies with rc=3: leave $? = 3
|
||||
# via a failing function, then call the finalizer (bash's trap semantics: the
|
||||
# trap sees the script's exit status in $?).
|
||||
BUILD_DIR="$WORK/b1"
|
||||
RUN_TESTS=1
|
||||
mkdir -p "$BUILD_DIR/meson-logs"
|
||||
echo "fake testlog with a failure" > "$BUILD_DIR/meson-logs/testlog.txt"
|
||||
echo "fake json" > "$BUILD_DIR/meson-logs/testlog.json"
|
||||
rc_probe() { return 3; }
|
||||
rc_probe # $? = 3
|
||||
finalize_failure_evidence
|
||||
arc="$(printf '%s/testlogs.archive/%s' "$SRC_DIR" "$BUILD_DIR")"
|
||||
entries="$(ls "$arc" 2>/dev/null | grep -c '^this-.*-rc3$')"
|
||||
if [ "$entries" = "1" ] && [ -f "$arc/this-"*"-rc3/meson-logs/testlog.txt" ] \
|
||||
&& [ "$(cat "$BUILD_DIR/.meson_test_failed_rc" 2>/dev/null)" = "3" ]; then
|
||||
pass "meson-logs archived as this-*-rc3 with marker .meson_test_failed_rc=3"
|
||||
else
|
||||
fail "expected this-*-rc3 archive entry + marker; got: $(ls "$arc" 2>/dev/null)"
|
||||
fi
|
||||
# original still in place (copy, not move)
|
||||
if [ -f "$BUILD_DIR/meson-logs/testlog.txt" ]; then
|
||||
pass "original meson-logs left in the build dir (copy semantics)"
|
||||
else
|
||||
fail "meson-logs was MOVED out of the build dir -- this run's own inspection loses it"
|
||||
fi
|
||||
|
||||
echo "── 2. successful run: EXIT trap archives nothing ──"
|
||||
arc2="$(printf '%s/testlogs.archive/%s' "$SRC_DIR" "$WORK/b2")"
|
||||
BUILD_DIR="$WORK/b2"
|
||||
RUN_TESTS=1
|
||||
mkdir -p "$BUILD_DIR/meson-logs"
|
||||
echo "all green" > "$BUILD_DIR/meson-logs/testlog.txt"
|
||||
true # $? = 0
|
||||
finalize_failure_evidence
|
||||
if [ ! -e "$arc2" ]; then
|
||||
pass "no archive entry on rc=0"
|
||||
else
|
||||
fail "archive entry created on a successful run: $(ls "$arc2")"
|
||||
fi
|
||||
if [ ! -f "$BUILD_DIR/.meson_test_failed_rc" ]; then
|
||||
pass "no marker written on rc=0"
|
||||
else
|
||||
fail "marker written on rc=0"
|
||||
fi
|
||||
|
||||
echo "── 3. next run's archive_previous_failure moves the previous failure out ──"
|
||||
BUILD_DIR="$WORK/b1" # still has meson-logs + marker from step 1
|
||||
mkdir -p "$BUILD_DIR"
|
||||
archive_previous_failure
|
||||
prev="$(ls "$arc" 2>/dev/null | grep -c '^prev-.*-rc3$')"
|
||||
if [ "$prev" = "1" ] && [ -f "$arc/prev-"*"-rc3/meson-logs/testlog.txt" ]; then
|
||||
pass "previous failure's meson-logs now at prev-*-rc3 (out of the wipe's path)"
|
||||
else
|
||||
fail "expected prev-*-rc3 entry with testlog; got: $(ls "$arc" 2>/dev/null)"
|
||||
fi
|
||||
if [ ! -f "$BUILD_DIR/.meson_test_failed_rc" ]; then
|
||||
pass "marker consumed after archiving (a later passing run won't re-archive)"
|
||||
else
|
||||
fail "marker not consumed"
|
||||
fi
|
||||
|
||||
echo "── 4. /tmp/xce_* fixtures: old dirs move, fresh stay, regular files never touched ──"
|
||||
fresh_fixture="/tmp/xce_test_evidence_fresh_$$"
|
||||
old_fixture="/tmp/xce_test_evidence_old_$$"
|
||||
plain_file="/tmp/xce_test_evidence_file_$$" # regular file, not a mkdtemp fixture
|
||||
mkdir -p "$fresh_fixture" "$old_fixture"
|
||||
: > "$plain_file"
|
||||
touch -d '30 minutes ago' "$old_fixture" "$plain_file" # aged: not a concurrent session's
|
||||
BUILD_DIR="$WORK/b3"
|
||||
archive_previous_failure
|
||||
arc3="$(printf '%s/testlogs.archive/%s' "$SRC_DIR" "$WORK/b3")"
|
||||
if [ -d "$arc3/prev-"*"-fixtures/xce_test_evidence_old_$$" ] && [ ! -e "$old_fixture" ]; then
|
||||
pass "aged /tmp/xce_* fixture moved into the archive"
|
||||
else
|
||||
fail "aged fixture not archived: $(ls -d "$arc3/prev-"*"-fixtures" 2>/dev/null)"
|
||||
fi
|
||||
if [ -d "$fresh_fixture" ]; then
|
||||
pass "fresh (<5 min) fixture left alone (concurrent session's)"
|
||||
else
|
||||
fail "fresh fixture was moved -- a concurrent session's run would lose it"
|
||||
fi
|
||||
if [ -f "$plain_file" ]; then
|
||||
pass "regular file xce_* NOT moved (only mkdtemp dirs are fixtures; xce_* also matches daemon debug logs)"
|
||||
else
|
||||
fail "regular file was moved -- the sweep is over-matching non-fixtures"
|
||||
fi
|
||||
rm -rf "$fresh_fixture"
|
||||
rm -f "$plain_file"
|
||||
|
||||
echo "── 5. retention: newest 10 survive ──"
|
||||
BUILD_DIR="$WORK/b5"
|
||||
arc5="$(printf '%s/testlogs.archive/%s' "$SRC_DIR" "$WORK/b5")"
|
||||
mkdir -p "$arc5"
|
||||
for i in $(seq -w 1 15); do mkdir -p "$arc5/prev-20261022-0000$i-rc1"; done
|
||||
archive_previous_failure
|
||||
count="$(find "$arc5" -mindepth 1 -maxdepth 1 -type d | wc -l)"
|
||||
if [ "$count" = "10" ] && [ -d "$arc5/prev-20261022-000015-rc1" ] && [ ! -d "$arc5/prev-20261022-000005-rc1" ]; then
|
||||
pass "archive capped at newest 10 (000015 kept, 000005 pruned)"
|
||||
else
|
||||
fail "retention wrong: $count entries"
|
||||
fi
|
||||
|
||||
echo "── 6. non-test failure (RUN_TESTS=0): nothing archived ──"
|
||||
BUILD_DIR="$WORK/b6"
|
||||
RUN_TESTS=0
|
||||
mkdir -p "$BUILD_DIR/meson-logs"
|
||||
echo "compile error log" > "$BUILD_DIR/meson-logs/meson-log.txt"
|
||||
rc_probe2() { return 2; }
|
||||
rc_probe2 # $? = 2
|
||||
finalize_failure_evidence
|
||||
arc6="$(printf '%s/testlogs.archive/%s' "$SRC_DIR" "$WORK/b6")"
|
||||
if [ ! -e "$arc6" ]; then
|
||||
pass "compile-error path archives nothing (deterministic, self-explanatory)"
|
||||
else
|
||||
fail "archive entry created for a non-test failure"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$FAIL" = 0 ]; then
|
||||
green "Results: ${PASS} passed, ${FAIL} failed"
|
||||
exit 0
|
||||
else
|
||||
red "Results: ${PASS} passed, ${FAIL} failed"
|
||||
exit 1
|
||||
fi
|
||||
|
|
@ -232,9 +232,14 @@ cd "$SRC_DIR"
|
|||
# against the same dir raced on setup_build's rm -rf ("Directory not empty")
|
||||
# or deleted a binary the other run had just built; it now waits here until
|
||||
# this run exits. Runs against different build dirs are not affected.
|
||||
#
|
||||
# The EXIT trap is finalize_failure_evidence (defined below, bug-1971): it
|
||||
# archives this run's meson-logs on a failed test run AND releases the
|
||||
# build lock -- so the lock release that used to be its own trap line is now
|
||||
# the finalizer's last statement.
|
||||
. "$SCRIPT_DIR/build-lock.sh"
|
||||
build_lock_acquire "$BUILD_DIR"
|
||||
trap build_lock_release EXIT
|
||||
trap finalize_failure_evidence EXIT
|
||||
|
||||
# build-time-versioning task 2.1: compute commit_id/build_time once, common to
|
||||
# every profile, and append to MESON_OPTS below (before either `meson setup`
|
||||
|
|
@ -363,8 +368,11 @@ usage() {
|
|||
# test run just created -- this sandbox routinely has multiple sessions
|
||||
# building/testing at once.
|
||||
prune_stale_tmp_fixtures() {
|
||||
# -type d for the same reason as archive_previous_failure below: mkdtemp
|
||||
# fixtures are directories; xce_* also matches regular files (daemon debug
|
||||
# logs), and this function rm -rf's its matches.
|
||||
local stale
|
||||
stale="$(find /tmp -maxdepth 1 -name 'xce_*' -mmin +180 2>/dev/null)"
|
||||
stale="$(find /tmp -maxdepth 1 -name 'xce_*' -type d -mmin +180 2>/dev/null)"
|
||||
if [ -n "$stale" ]; then
|
||||
local count
|
||||
count="$(printf '%s\n' "$stale" | wc -l)"
|
||||
|
|
@ -384,7 +392,126 @@ setup_build() {
|
|||
fi
|
||||
}
|
||||
|
||||
prune_stale_tmp_fixtures
|
||||
# bug-1971: this script destroys test-failure evidence by design of its own
|
||||
# default flow. setup_build() rm -rf's the whole build dir (meson-logs/
|
||||
# testlog.* with it), and even without the wipe meson overwrites testlog.* on
|
||||
# every `meson test`. A flaky failure (5b9d7270's two, bug-1521's random heap
|
||||
# corruption, t2 2026-10-22) therefore leaves NOTHING on disk: the user sees
|
||||
# "failed once, passes on rerun" and the failed case can never be named. Two
|
||||
# chokes fixes it, both here because every test run in this project goes
|
||||
# through this script:
|
||||
#
|
||||
# 1. BEFORE the wipe (archive_previous_failure, called just below): if the
|
||||
# PREVIOUS run in this build dir failed -- meson's own marker file
|
||||
# .meson_test_failed_rc (written by the EXIT trap, see
|
||||
# finalize_failure_evidence) or FAILED-case /tmp/xce_* fixture dirs the
|
||||
# mkdtemp shim deliberately preserved for post-mortem (see
|
||||
# testsuite_support/mkdtemp_clean.c: passed cases are swept, failed ones
|
||||
# keep their dirs) -- move that evidence into a timestamped,
|
||||
# never-wiped archive dir first.
|
||||
#
|
||||
# 2. ON FAILURE (finalize_failure_evidence, EXIT trap): if THIS run exits
|
||||
# nonzero after running tests, copy meson-logs/ out before the next
|
||||
# run's wipe. EXIT (not explicit calls) so every failure path is
|
||||
# covered: set -e aborts, meson test nonzero, Go test nonzero, Ctrl-C
|
||||
# (INT triggers EXIT in bash), the lot.
|
||||
#
|
||||
# Archives live under $SRC_DIR/testlogs.archive/<build_dir>/<timestamp>-rc<N>/
|
||||
# -- OUTSIDE the build dir so the wipe never touches them, inside the repo so
|
||||
# they sit next to the sources they belong to. Retention: keep the newest 10
|
||||
# per build dir, prune older ones (the archive is capped, not unbounded --
|
||||
# testlogs are a few MB each, fixtures can be bigger, so the cap matters).
|
||||
# The fixture sweep moves rather than copies dirs, and de-duplicates by
|
||||
# timestamp to the second so a same-second double invocation can't collide
|
||||
# (mktemp suffix if it somehow would).
|
||||
archive_root() {
|
||||
printf '%s/testlogs.archive/%s' "$SRC_DIR" "$BUILD_DIR"
|
||||
}
|
||||
|
||||
archive_previous_failure() {
|
||||
local stamp arc
|
||||
stamp="$(date +%Y%m%d-%H%M%S)"
|
||||
|
||||
# 1a. The previous run's own meson-logs, if it failed. finalize_failure_
|
||||
# evidence leaves .meson_test_failed_rc holding the exit code; no
|
||||
# marker = last run passed or never ran tests, nothing to do.
|
||||
local marker="$BUILD_DIR/.meson_test_failed_rc"
|
||||
if [ -f "$marker" ]; then
|
||||
local prev_rc
|
||||
prev_rc="$(cat "$marker" 2>/dev/null || echo '?')"
|
||||
arc="$(archive_root)/prev-$stamp-rc$prev_rc"
|
||||
if [ -d "$BUILD_DIR/meson-logs" ]; then
|
||||
mkdir -p "$arc"
|
||||
mv "$BUILD_DIR/meson-logs" "$arc/" 2>/dev/null || true
|
||||
echo "=== bug-1971: PREVIOUS run (rc=$prev_rc) failed -- its meson-logs archived to $arc (they would have been destroyed by this run's wipe) ==="
|
||||
fi
|
||||
rm -f "$marker"
|
||||
fi
|
||||
|
||||
# 1b. FAILED-case /tmp/xce_* fixture dirs (mkdtemp sweep contract: passed
|
||||
# cases are swept immediately, FAILED cases keep their dirs). They
|
||||
# survive this run's wipe on disk, but nothing else ever cleans or
|
||||
# curates them -- move them into the archive so they sit WITH the
|
||||
# testlog that explains them instead of leaking in /tmp until the
|
||||
# 3h prune (prune_stale_tmp_fixtures) destroys them unread. Skip the
|
||||
# ones a concurrent session's still-running test owns (mtime < 5 min
|
||||
# old): those are not from the previous run.
|
||||
# -type d: fixtures are always mkdtemp DIRECTORIES (/tmp/xce_<label>_XXXXXX,
|
||||
# e.g. test_bug857_regression.c:176, test_persist_wal.c:41); the bare
|
||||
# xce_* name also matches regular files -- daemon debug logs like
|
||||
# /tmp/xce_scheduler.log, /tmp/xce_strace*.log from cluster debugging
|
||||
# are NOT fixtures and must stay in /tmp (live-hit 2026-10-22: the
|
||||
# untyped find moved all five into this archive).
|
||||
local stale
|
||||
stale="$(find /tmp -maxdepth 1 -name 'xce_*' -type d -mmin +5 2>/dev/null || true)"
|
||||
if [ -n "$stale" ]; then
|
||||
arc="$(archive_root)/prev-$stamp-fixtures"
|
||||
mkdir -p "$arc"
|
||||
local moved=0
|
||||
while IFS= read -r d; do
|
||||
[ -z "$d" ] && continue
|
||||
mv "$d" "$arc/" 2>/dev/null && moved=$((moved + 1))
|
||||
done <<< "$stale"
|
||||
if [ "$moved" -gt 0 ]; then
|
||||
echo "=== bug-1971: moved $moved leftover /tmp/xce_* FAILED-case fixture dir(s) to $arc ==="
|
||||
else
|
||||
rmdir "$arc" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
|
||||
# 1c. Retention: cap each build dir's archive at its newest 10 entries.
|
||||
# -mindepth 1 so the archive root itself is never a rm candidate.
|
||||
local root
|
||||
root="$(archive_root)"
|
||||
if [ -d "$root" ]; then
|
||||
find "$root" -mindepth 1 -maxdepth 1 -type d | sort -r | tail -n +11 | xargs -r rm -rf -- 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
|
||||
# EXIT trap: this run failed after running tests -> copy meson-logs to the
|
||||
# archive and drop the marker for the NEXT run's archive_previous_failure.
|
||||
# Copy (not move): meson-logs still belongs to the live build dir for this
|
||||
# run's own inspection; the next run moves the copy out at ITS start (1a).
|
||||
# Guarded on RUN_TESTS=1 and meson-logs existing -- a configure/compile
|
||||
# failure has no testlog worth archiving (meson-log.txt IS in meson-logs,
|
||||
# but a compile error is deterministic and self-explanatory on rerun; it's
|
||||
# the testlog-bearing flaky failures that need this). INT/TERM fall through
|
||||
# to EXIT in bash, so Ctrl-C mid-test is covered too.
|
||||
finalize_failure_evidence() {
|
||||
local rc=$?
|
||||
if [ "$rc" -ne 0 ] && [ "${RUN_TESTS:-0}" = 1 ] && [ -d "$BUILD_DIR/meson-logs" ]; then
|
||||
local stamp arc
|
||||
stamp="$(date +%Y%m%d-%H%M%S)"
|
||||
arc="$(archive_root)/this-$stamp-rc$rc"
|
||||
mkdir -p "$arc"
|
||||
cp -r "$BUILD_DIR/meson-logs" "$arc/" 2>/dev/null || true
|
||||
printf '%s\n' "$rc" > "$BUILD_DIR/.meson_test_failed_rc" 2>/dev/null || true
|
||||
echo "=== bug-1971: this run FAILED (rc=$rc) -- meson-logs archived to $arc; the NEXT run will also move it out of the wipe's path ==="
|
||||
fi
|
||||
build_lock_release
|
||||
}
|
||||
|
||||
archive_previous_failure
|
||||
|
||||
case "$COMPONENT" in
|
||||
lib)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue