diff --git a/.wolf/buglog.json b/.wolf/buglog.json index e79997c0..d4f2c157 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -3,19 +3,19 @@ "generated": "by scripts/buglog.py from .wolf/bugs/*.md -- do not edit", "bugs": [ { - "id": "bug-001", - "status": "open", + "id": "bug-1971", + "status": "fixed", "status_updated": "2026-10-22", - "status_note": "newly logged", + "status_note": "", "component": "build", "severity": "wrong-result", - "timestamp": "2026-10-22T07:48:38Z", - "last_seen": "2026-10-22T07:48:38Z", - "occurrences": 1, - "error_message": "test: xce-build.sh wipes meson-logs and fixtures on the next run — a failing test's evidence is unrecoverable\n\n**Observed:**\nTODO: the exact error, log line or sanitizer text, in a code block\n\n**Expected:** TODO\n**Impact:** TODO\n**Found by:** TODO: user report | soak | ASAN | audit | review | test | spec task", + "timestamp": "2026-10-22T07:50:00Z", + "last_seen": "2026-10-22T07:50:00Z", + "occurrences": 2, + "error_message": "next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable\n\n**Observed:** (t2, 2026-10-22) a full `make build` (asan profile) failed on a controller test case; a following `make clean` + `make build` passed 134/134. The failing run's `meson-logs/testlog.{txt,json,junit.xml}` had been destroyed twice over: `setup_build`'s unconditional `rm -rf \"$BUILD_DIR\"` deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites `testlog.*` on every `meson test` invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (`/tmp/test_xcecd_resync.log` and rotations) showed no failure signature, and no `sched_test`/Go test output is retained anywhere.\n\nReconstruction from filesystem timestamps alone (`/tmp/test_*.log` rotations, `meson-log.txt` build options line, `/usr/local/go` ctime, bash history) established the two-run timeline but could not name the failed case.\n\nThe same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.\n\n**Expected:** when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' `/tmp/xce_*` dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.\n\n**Impact:** intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp \"failed cases keep their dirs for post-mortem\" contract, since the next `make build` never looks for those dirs and a clean build gives the appearance of none existing.\n\n**Found by:** user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.", "file": "src/tools/xce-build.sh", - "root_cause": "TODO: required for fixed/partial", - "fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why", + "root_cause": "`setup_build()` in `src/tools/xce-build.sh` unconditionally `rm -rf \"$BUILD_DIR\"` on every non-incremental invocation, and `meson test` overwrites `$BUILD_DIR/meson-logs/testlog.*` on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.", + "fix": "Two chokes in `src/tools/xce-build.sh` (this commit):\n\n1. `archive_previous_failure()` — runs at script start, before `setup_build`'s\n wipe. If the marker `.meson_test_failed_rc` (written by the previous run's\n EXIT trap) exists, the previous run's `meson-logs` are MOVED to\n `$SRC_DIR/testlogs.archive//prev--rc/` — outside the\n build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e.\n not a concurrent session's) `/tmp/xce_*` FAILED-case fixture DIRS into\n `prev--fixtures/`, and caps the archive at the newest 10 entries\n per build dir.\n2. `finalize_failure_evidence()` — the EXIT trap (replaces the old\n lock-release trap, which becomes its last statement). On rc≠0 with\n RUN_TESTS=1, COPIES `meson-logs` to\n `testlogs.archive//this--rc/` and writes the marker.\n Copy-not-move: meson-logs still belongs to the live build dir for this\n run's own inspection; the next run moves the copy out at its start.\n Configure/compile failures (RUN_TESTS unset) archive nothing — those are\n deterministic and self-explanatory on rerun; it's the testlog-bearing\n flaky failures that need preservation.\n\nAdditionally `-type d` added to BOTH `/tmp/xce_*` sweeps (this one and\nbug-924's `prune_stale_tmp_fixtures`): mkdtemp fixtures are always\ndirectories (`/tmp/xce_