96 lines
6.4 KiB
Markdown
96 lines
6.4 KiB
Markdown
---
|
|
id: "bug-1971"
|
|
status: "fixed"
|
|
status_updated: "2026-10-22"
|
|
status_note: "two-choke archive in xce-build.sh; test harness green"
|
|
component: "build"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-22T07:50:00Z"
|
|
last_seen: "2026-10-22T07:50:00Z"
|
|
occurrences: 2
|
|
files: ["src/tools/xce-build.sh"]
|
|
tags: ["test", "flaky", "observability", "build"]
|
|
related: []
|
|
fix_commits: ["68c3ccd7"]
|
|
---
|
|
|
|
# bug-1971: next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable
|
|
|
|
## Description
|
|
**Observed:** (t2, 2026-10-22) a full `make build` (asan profile) failed on a controller test case; a following `make clean` + `make build` passed 134/134. The failing run's `meson-logs/testlog.{txt,json,junit.xml}` had been destroyed twice over: `setup_build`'s unconditional `rm -rf "$BUILD_DIR"` deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites `testlog.*` on every `meson test` invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (`/tmp/test_xcecd_resync.log` and rotations) showed no failure signature, and no `sched_test`/Go test output is retained anywhere.
|
|
|
|
Reconstruction from filesystem timestamps alone (`/tmp/test_*.log` rotations, `meson-log.txt` build options line, `/usr/local/go` ctime, bash history) established the two-run timeline but could not name the failed case.
|
|
|
|
The same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.
|
|
|
|
**Expected:** when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' `/tmp/xce_*` dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.
|
|
|
|
**Impact:** intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp "failed cases keep their dirs for post-mortem" contract, since the next `make build` never looks for those dirs and a clean build gives the appearance of none existing.
|
|
|
|
**Found by:** user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.
|
|
|
|
## Environment
|
|
- OS: Ubuntu 24.04.2 LTS (t1 and t2); kernel Linux 6.8.0-139-generic x86_64
|
|
- libc: glibc 2.39 (host ldd)
|
|
- Toolchain: gcc 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1), meson 1.12.1
|
|
- Build: asan (xce.build)
|
|
- Code: build/de-nix @ 5b9d7270 (t2 checkout), e8cb6d57 (t1)
|
|
- Deployment: n/a — dev-loop build/test on t1 (workstation) and t2 (8-core/8GB test node)
|
|
|
|
## Reproduction
|
|
1. On any machine: `make build` (or `bash src/tools/xce-build.sh`) and let the test phase fail (e.g. by patching a test to fail).
|
|
2. Note `src/xce.build/meson-logs/testlog.txt` contains the failure.
|
|
3. Run `make clean && make build` again: the previous testlog is gone (build dir was rm -rf'd), and even without `make clean`, a successful `meson test` run overwrites testlog.* in place.
|
|
- Frequency: deterministic — every subsequent run destroys the prior run's evidence.
|
|
- Evidence: t2 `/tmp/myxce` 2026-10-22: failing run ~05:46, passing rerun 05:52 (its testlog survives at `/tmp/myxce/src/xce.build/meson-logs/testlog.txt`, 134/134 OK); the failing run's testlog was deleted by the `make clean` at ~05:5x.
|
|
|
|
## Root cause
|
|
`setup_build()` in `src/tools/xce-build.sh` unconditionally `rm -rf "$BUILD_DIR"` on every non-incremental invocation, and `meson test` overwrites `$BUILD_DIR/meson-logs/testlog.*` on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.
|
|
|
|
## Fix
|
|
Two chokes in `src/tools/xce-build.sh` (this commit):
|
|
|
|
1. `archive_previous_failure()` — runs at script start, before `setup_build`'s
|
|
wipe. If the marker `.meson_test_failed_rc` (written by the previous run's
|
|
EXIT trap) exists, the previous run's `meson-logs` are MOVED to
|
|
`$SRC_DIR/testlogs.archive/<BUILD_DIR>/prev-<stamp>-rc<N>/` — outside the
|
|
build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e.
|
|
not a concurrent session's) `/tmp/xce_*` FAILED-case fixture DIRS into
|
|
`prev-<stamp>-fixtures/`, and caps the archive at the newest 10 entries
|
|
per build dir.
|
|
2. `finalize_failure_evidence()` — the EXIT trap (replaces the old
|
|
lock-release trap, which becomes its last statement). On rc≠0 with
|
|
RUN_TESTS=1, COPIES `meson-logs` to
|
|
`testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/` and writes the marker.
|
|
Copy-not-move: meson-logs still belongs to the live build dir for this
|
|
run's own inspection; the next run moves the copy out at its start.
|
|
Configure/compile failures (RUN_TESTS unset) archive nothing — those are
|
|
deterministic and self-explanatory on rerun; it's the testlog-bearing
|
|
flaky failures that need preservation.
|
|
|
|
Additionally `-type d` added to BOTH `/tmp/xce_*` sweeps (this one and
|
|
bug-924's `prune_stale_tmp_fixtures`): mkdtemp fixtures are always
|
|
directories (`/tmp/xce_<label>_XXXXXX`), but the bare name also matches
|
|
regular files — the untyped find live-moved daemon debug logs
|
|
(`/tmp/xce_scheduler.log`, `/tmp/xce_strace*.log`) into the archive on
|
|
2026-10-22, and the bug-924 variant `rm -rf`s its matches.
|
|
|
|
## Verification
|
|
- Test: `src/tools/test-failure-evidence-archive.sh` — 6 scenarios /
|
|
11 assertions exercising the real function bodies (sourced with surgical
|
|
stubbing): failed run archives as this-*-rc3 + marker; success archives
|
|
nothing; next run moves prior failure to prev-*-rc3 + consumes marker;
|
|
aged fixture dirs move / fresh stay / regular files never touched;
|
|
retention caps at newest 10; RUN_TESTS=0 archives nothing. All green
|
|
(11 passed, 0 failed, bash, 2026-10-22).
|
|
- Suites: n/a — this bug lives in the build script, not in C code; no meson
|
|
suite exercises it. `bash -n` syntax check green.
|
|
- Real wipe path not exercised by the harness test (it stubs meson);
|
|
validated by inspection: the archive root is `$SRC_DIR/testlogs.archive/`,
|
|
which `rm -rf "$BUILD_DIR"` cannot reach.
|
|
- Live: first t2/sandbox failing run after this commit leaves
|
|
`testlogs.archive/xce.build/this-*-rc*` with the testlog — that is the
|
|
evidence this bug asks for.
|
|
|
|
## References
|
|
- Related: none
|