myxce/.wolf/bugs/bug-1971.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1971	fixed	2026-10-22	two-choke archive in xce-build.sh; test harness green	build	wrong-result	2026-10-22T07:50:00Z	2026-10-22T07:50:00Z	2	src/tools/xce-build.sh	test flaky observability build		68c3ccd7

bug-1971: next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable

Description

Observed: (t2, 2026-10-22) a full make build (asan profile) failed on a controller test case; a following make clean + make build passed 134/134. The failing run's meson-logs/testlog.{txt,json,junit.xml} had been destroyed twice over: setup_build's unconditional rm -rf "$BUILD_DIR" deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites testlog.* on every meson test invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (/tmp/test_xcecd_resync.log and rotations) showed no failure signature, and no sched_test/Go test output is retained anywhere.

Reconstruction from filesystem timestamps alone (/tmp/test_*.log rotations, meson-log.txt build options line, /usr/local/go ctime, bash history) established the two-run timeline but could not name the failed case.

The same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.

Expected: when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' /tmp/xce_* dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.

Impact: intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp "failed cases keep their dirs for post-mortem" contract, since the next make build never looks for those dirs and a clean build gives the appearance of none existing.

Found by: user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.

Environment

• OS: Ubuntu 24.04.2 LTS (t1 and t2); kernel Linux 6.8.0-139-generic x86_64
• libc: glibc 2.39 (host ldd)
• Toolchain: gcc 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1), meson 1.12.1
• Build: asan (xce.build)
• Code: build/de-nix @ 5b9d7270 (t2 checkout), e8cb6d57 (t1)
• Deployment: n/a — dev-loop build/test on t1 (workstation) and t2 (8-core/8GB test node)

Reproduction

1. On any machine: make build (or bash src/tools/xce-build.sh) and let the test phase fail (e.g. by patching a test to fail).
2. Note src/xce.build/meson-logs/testlog.txt contains the failure.
3. Run make clean && make build again: the previous testlog is gone (build dir was rm -rf'd), and even without make clean, a successful meson test run overwrites testlog.* in place.

• Frequency: deterministic — every subsequent run destroys the prior run's evidence.
• Evidence: t2 /tmp/myxce 2026-10-22: failing run ~05:46, passing rerun 05:52 (its testlog survives at /tmp/myxce/src/xce.build/meson-logs/testlog.txt, 134/134 OK); the failing run's testlog was deleted by the make clean at ~05:5x.

Root cause

setup_build() in src/tools/xce-build.sh unconditionally rm -rf "$BUILD_DIR" on every non-incremental invocation, and meson test overwrites $BUILD_DIR/meson-logs/testlog.* on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.

Fix

Two chokes in src/tools/xce-build.sh (this commit):

1. archive_previous_failure() — runs at script start, before setup_build's wipe. If the marker .meson_test_failed_rc (written by the previous run's EXIT trap) exists, the previous run's meson-logs are MOVED to $SRC_DIR/testlogs.archive/<BUILD_DIR>/prev-<stamp>-rc<N>/ — outside the build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e. not a concurrent session's) /tmp/xce_* FAILED-case fixture DIRS into prev-<stamp>-fixtures/, and caps the archive at the newest 10 entries per build dir.
2. finalize_failure_evidence() — the EXIT trap (replaces the old lock-release trap, which becomes its last statement). On rc≠0 with RUN_TESTS=1, COPIES meson-logs to testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/ and writes the marker. Copy-not-move: meson-logs still belongs to the live build dir for this run's own inspection; the next run moves the copy out at its start. Configure/compile failures (RUN_TESTS unset) archive nothing — those are deterministic and self-explanatory on rerun; it's the testlog-bearing flaky failures that need preservation.

Additionally -type d added to BOTH /tmp/xce_* sweeps (this one and bug-924's prune_stale_tmp_fixtures): mkdtemp fixtures are always directories (/tmp/xce_<label>_XXXXXX), but the bare name also matches regular files — the untyped find live-moved daemon debug logs (/tmp/xce_scheduler.log, /tmp/xce_strace*.log) into the archive on 2026-10-22, and the bug-924 variant rm -rfs its matches.

Verification

• Test: src/tools/test-failure-evidence-archive.sh — 6 scenarios / 11 assertions exercising the real function bodies (sourced with surgical stubbing): failed run archives as this--rc3 + marker; success archives nothing; next run moves prior failure to prev--rc3 + consumes marker; aged fixture dirs move / fresh stay / regular files never touched; retention caps at newest 10; RUN_TESTS=0 archives nothing. All green (11 passed, 0 failed, bash, 2026-10-22).
• Suites: n/a — this bug lives in the build script, not in C code; no meson suite exercises it. bash -n syntax check green.
• Real wipe path not exercised by the harness test (it stubs meson); validated by inspection: the archive root is $SRC_DIR/testlogs.archive/, which rm -rf "$BUILD_DIR" cannot reach.
• Live: first t2/sandbox failing run after this commit leaves testlogs.archive/xce.build/this-*-rc* with the testlog — that is the evidence this bug asks for.

References

• Related: none