bug-1971: fix_commits field via buglog.py status
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
parent
d8f738b08a
commit
993daa8f95
2 changed files with 15 additions and 12 deletions
|
|
@ -3,19 +3,19 @@
|
|||
"generated": "by scripts/buglog.py from .wolf/bugs/*.md -- do not edit",
|
||||
"bugs": [
|
||||
{
|
||||
"id": "bug-001",
|
||||
"status": "open",
|
||||
"id": "bug-1971",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-22",
|
||||
"status_note": "newly logged",
|
||||
"status_note": "",
|
||||
"component": "build",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-22T07:48:38Z",
|
||||
"last_seen": "2026-10-22T07:48:38Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "test: xce-build.sh wipes meson-logs and fixtures on the next run — a failing test's evidence is unrecoverable\n\n**Observed:**\nTODO: the exact error, log line or sanitizer text, in a code block\n\n**Expected:** TODO\n**Impact:** TODO\n**Found by:** TODO: user report | soak | ASAN | audit | review | test | spec task",
|
||||
"timestamp": "2026-10-22T07:50:00Z",
|
||||
"last_seen": "2026-10-22T07:50:00Z",
|
||||
"occurrences": 2,
|
||||
"error_message": "next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable\n\n**Observed:** (t2, 2026-10-22) a full `make build` (asan profile) failed on a controller test case; a following `make clean` + `make build` passed 134/134. The failing run's `meson-logs/testlog.{txt,json,junit.xml}` had been destroyed twice over: `setup_build`'s unconditional `rm -rf \"$BUILD_DIR\"` deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites `testlog.*` on every `meson test` invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (`/tmp/test_xcecd_resync.log` and rotations) showed no failure signature, and no `sched_test`/Go test output is retained anywhere.\n\nReconstruction from filesystem timestamps alone (`/tmp/test_*.log` rotations, `meson-log.txt` build options line, `/usr/local/go` ctime, bash history) established the two-run timeline but could not name the failed case.\n\nThe same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.\n\n**Expected:** when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' `/tmp/xce_*` dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.\n\n**Impact:** intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp \"failed cases keep their dirs for post-mortem\" contract, since the next `make build` never looks for those dirs and a clean build gives the appearance of none existing.\n\n**Found by:** user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.",
|
||||
"file": "src/tools/xce-build.sh",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"root_cause": "`setup_build()` in `src/tools/xce-build.sh` unconditionally `rm -rf \"$BUILD_DIR\"` on every non-incremental invocation, and `meson test` overwrites `$BUILD_DIR/meson-logs/testlog.*` on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.",
|
||||
"fix": "Two chokes in `src/tools/xce-build.sh` (this commit):\n\n1. `archive_previous_failure()` — runs at script start, before `setup_build`'s\n wipe. If the marker `.meson_test_failed_rc` (written by the previous run's\n EXIT trap) exists, the previous run's `meson-logs` are MOVED to\n `$SRC_DIR/testlogs.archive/<BUILD_DIR>/prev-<stamp>-rc<N>/` — outside the\n build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e.\n not a concurrent session's) `/tmp/xce_*` FAILED-case fixture DIRS into\n `prev-<stamp>-fixtures/`, and caps the archive at the newest 10 entries\n per build dir.\n2. `finalize_failure_evidence()` — the EXIT trap (replaces the old\n lock-release trap, which becomes its last statement). On rc≠0 with\n RUN_TESTS=1, COPIES `meson-logs` to\n `testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/` and writes the marker.\n Copy-not-move: meson-logs still belongs to the live build dir for this\n run's own inspection; the next run moves the copy out at its start.\n Configure/compile failures (RUN_TESTS unset) archive nothing — those are\n deterministic and self-explanatory on rerun; it's the testlog-bearing\n flaky failures that need preservation.\n\nAdditionally `-type d` added to BOTH `/tmp/xce_*` sweeps (this one and\nbug-924's `prune_stale_tmp_fixtures`): mkdtemp fixtures are always\ndirectories (`/tmp/xce_<label>_XXXXXX`), but the bare name also matches\nregular files — the untyped find live-moved daemon debug logs\n(`/tmp/xce_scheduler.log`, `/tmp/xce_strace*.log`) into the archive on\n2026-10-22, and the bug-924 variant `rm -rf`s its matches.\n\nVerification:\n- Test: `src/tools/test-failure-evidence-archive.sh` — 6 scenarios /\n 11 assertions exercising the real function bodies (sourced with surgical\n stubbing): failed run archives as this-*-rc3 + marker; success archives\n nothing; next run moves prior failure to prev-*-rc3 + consumes marker;\n aged fixture dirs move / fresh stay / regular files never touched;\n retention caps at newest 10; RUN_TESTS=0 archives nothing. All green\n (11 passed, 0 failed, bash, 2026-10-22).\n- Suites: n/a — this bug lives in the build script, not in C code; no meson\n suite exercises it. `bash -n` syntax check green.\n- Real wipe path not exercised by the harness test (it stubs meson);\n validated by inspection: the archive root is `$SRC_DIR/testlogs.archive/`,\n which `rm -rf \"$BUILD_DIR\"` cannot reach.\n- Live: first t2/sandbox failing run after this commit leaves\n `testlogs.archive/xce.build/this-*-rc*` with the testlog — that is the\n evidence this bug asks for.",
|
||||
"tags": [
|
||||
"test",
|
||||
"flaky",
|
||||
|
|
@ -23,8 +23,10 @@
|
|||
"build"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-001.md"
|
||||
"fix_commits": [
|
||||
"68c3ccd7"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1971.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
id: "bug-1971"
|
||||
status: "fixed"
|
||||
status_updated: "2026-10-22"
|
||||
status_note: "two-choke archive in xce-build.sh; test harness green; fix 68c3ccd7"
|
||||
status_note: "two-choke archive in xce-build.sh; test harness green"
|
||||
component: "build"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-22T07:50:00Z"
|
||||
|
|
@ -11,6 +11,7 @@ occurrences: 2
|
|||
files: ["src/tools/xce-build.sh"]
|
||||
tags: ["test", "flaky", "observability", "build"]
|
||||
related: []
|
||||
fix_commits: ["68c3ccd7"]
|
||||
---
|
||||
|
||||
# bug-1971: next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue