chore(buglog): bug-1604 fixed (980a0054)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
980a0054d1
commit
18b9602204
2 changed files with 31 additions and 17 deletions
|
|
@ -8480,18 +8480,18 @@
|
|||
},
|
||||
{
|
||||
"id": "bug-1604",
|
||||
"status": "open",
|
||||
"status_updated": "2026-09-24",
|
||||
"status_note": "concurrent xce-build.sh runs race on rm -rf of shared BUILD_DIR (no lock)",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-09-30",
|
||||
"status_note": "",
|
||||
"component": "build",
|
||||
"severity": "unknown",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-08-20T10:24:39Z",
|
||||
"last_seen": "2026-09-19T09:00:00Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh…\n\nrm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh setup_build()'s rm -rf \\$BUILD_DIR step aborted the script under set -euo pipefail before meson setup ran)",
|
||||
"file": "src/tools/xce-build.sh",
|
||||
"file": "src/tools/xce-build.sh (setup_build), src/tools/build-lock.sh",
|
||||
"root_cause": "CORRECTED after further evidence (see fix note): initial overlayfs-flakiness theory was WRONG. The real cause is xce-build.sh defaulting BUILD_DIR to the same \"xce.build\" for every component and every invocation, with setup_build() doing an unconditional rm -rf \"$BUILD_DIR\" + fresh meson setup on every non-incremental run. When two xce-build.sh invocations run concurrently in the SAME repo working directory against DIFFERENT components (observed live: this session's \"xce-build.sh scheduler asan\" raced against a second, independently-running \"xce-build.sh controller asan\" process, PID 655158, started ~1 min later in the same cwd), they contend for the identical xce.build/ path -- one process's rm -rf can hit a directory the other process is actively writing into (\"Directory not empty\"), and later, a second clean-build pass can silently delete and replace a just-verified binary out from under a caller who believes their build is done. Confirmed live: xce_scheduler at src/xce.build/scheduler/xce_scheduler, verified present and fresh via ls -la and 'xce_scheduler --version' immediately after this session's scheduler build/test run, was gone minutes later -- the concurrent controller build's rm -rf had removed and was in the process of regenerating xce.build/ for its own component.",
|
||||
"fix": "No code fix applied (out of scope for the build-verification task this session was doing) -- flagging as a real, reproducible hazard rather than env flakiness. Practical mitigation for future sessions/agents: never assume xce.build/ (or any non-incremental xce-build.sh output) is stable once another concurrent xce-build.sh process might be running in the same working directory/worktree -- check 'ps aux | grep xce-build.sh' (or pass a distinct BUILD_DIR / use --incremental) before relying on a binary's presence or mtime surviving past the moment it was checked. Corrects this entry's own original overlayfs-race theory, which was a plausible but incorrect guess made before the concurrent process was discovered.",
|
||||
"fix": "src/tools/xce-build.sh takes an exclusive per-build-dir lock right after `cd \"$SRC_DIR\"` (before any setup_build) and releases it on EXIT, so a second run against the same build dir waits for the first to finish (printing `Waiting for another xce-build.sh run (PID N) to finish with <dir>`); runs against different build dirs never block each other. New src/tools/build-lock.sh: build_lock_acquire/build_lock_release. The lock is an mkdir-created directory holding the owner PID (atomic, coreutils only -- the nix dev shell has no flock(1)), placed outside the build dir (the wipe would delete it) under $XCE_BUILD_LOCK_ROOT or $TMPDIR, named by a cksum of the build dir's absolute path; a lock whose owner PID is gone (build killed with SIGKILL) is taken over. It does not change the wipe itself: a single non-incremental run still starts clean (bug-1719's behavior is unchanged).\n\nVerification:\n- Test: src/tools/test-build-lock.sh -- (1) a second acquire of the same dir waits until the first releases, (2) a dead owner's lock is taken over, (3) different dirs do not block, (4) xce-build.sh acquires the lock before its first setup_build call; RED before (build-lock.sh missing, then case 4 failing until xce-build.sh was wired), 4/4 GREEN after\n- Suites: n/a for C suites (script-only change); `bash -n src/tools/xce-build.sh` clean\n- Live: two real `xce-build.sh lib asan <same dir>` runs started 5 s apart, 2026-09-29: with the pre-fix script run 1 FAILED (dependency file deleted mid-compile) and run 2 passed; with the fix run 2 logged `Waiting for another xce-build.sh run (PID 1240637) to finish` and both runs completed (`=== Done: lib`)",
|
||||
"tags": [
|
||||
"asan",
|
||||
"concurrency",
|
||||
|
|
@ -8506,7 +8506,9 @@
|
|||
"related_bugs": [
|
||||
"bug-1561"
|
||||
],
|
||||
"fix_commits": [],
|
||||
"fix_commits": [
|
||||
"980a0054"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1604.md"
|
||||
},
|
||||
{
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue