lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore(buglog): bug-1604 fixed (980a0054)

Browse source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-30 03:44:09 +00:00
parent 980a0054d1
commit 18b9602204
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
2 changed files with 31 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

16
.wolf/buglog.json
View file

@ -8480,18 +8480,18 @@
},
{
"id": "bug-1604",
"status": "open",
"status_updated": "2026-09-24",
"status_note": "concurrent xce-build.sh runs race on rm -rf of shared BUILD_DIR (no lock)",
"status": "fixed",
"status_updated": "2026-09-30",
"status_note": "",
"component": "build",
"severity": "unknown",
"severity": "wrong-result",
"timestamp": "2026-08-20T10:24:39Z",
"last_seen": "2026-09-19T09:00:00Z",
"occurrences": 1,
"error_message": "rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh…\n\nrm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh setup_build()'s rm -rf \\$BUILD_DIR step aborted the script under set -euo pipefail before meson setup ran)",
"file": "src/tools/xce-build.sh",
"file": "src/tools/xce-build.sh (setup_build), src/tools/build-lock.sh",
"root_cause": "CORRECTED after further evidence (see fix note): initial overlayfs-flakiness theory was WRONG. The real cause is xce-build.sh defaulting BUILD_DIR to the same \"xce.build\" for every component and every invocation, with setup_build() doing an unconditional rm -rf \"$BUILD_DIR\" + fresh meson setup on every non-incremental run. When two xce-build.sh invocations run concurrently in the SAME repo working directory against DIFFERENT components (observed live: this session's \"xce-build.sh scheduler asan\" raced against a second, independently-running \"xce-build.sh controller asan\" process, PID 655158, started ~1 min later in the same cwd), they contend for the identical xce.build/ path -- one process's rm -rf can hit a directory the other process is actively writing into (\"Directory not empty\"), and later, a second clean-build pass can silently delete and replace a just-verified binary out from under a caller who believes their build is done. Confirmed live: xce_scheduler at src/xce.build/scheduler/xce_scheduler, verified present and fresh via ls -la and 'xce_scheduler --version' immediately after this session's scheduler build/test run, was gone minutes later -- the concurrent controller build's rm -rf had removed and was in the process of regenerating xce.build/ for its own component.",
"fix": "No code fix applied (out of scope for the build-verification task this session was doing) -- flagging as a real, reproducible hazard rather than env flakiness. Practical mitigation for future sessions/agents: never assume xce.build/ (or any non-incremental xce-build.sh output) is stable once another concurrent xce-build.sh process might be running in the same working directory/worktree -- check 'ps aux | grep xce-build.sh' (or pass a distinct BUILD_DIR / use --incremental) before relying on a binary's presence or mtime surviving past the moment it was checked. Corrects this entry's own original overlayfs-race theory, which was a plausible but incorrect guess made before the concurrent process was discovered.",
"fix": "src/tools/xce-build.sh takes an exclusive per-build-dir lock right after `cd \"$SRC_DIR\"` (before any setup_build) and releases it on EXIT, so a second run against the same build dir waits for the first to finish (printing `Waiting for another xce-build.sh run (PID N) to finish with <dir>`); runs against different build dirs never block each other. New src/tools/build-lock.sh: build_lock_acquire/build_lock_release. The lock is an mkdir-created directory holding the owner PID (atomic, coreutils only -- the nix dev shell has no flock(1)), placed outside the build dir (the wipe would delete it) under $XCE_BUILD_LOCK_ROOT or $TMPDIR, named by a cksum of the build dir's absolute path; a lock whose owner PID is gone (build killed with SIGKILL) is taken over. It does not change the wipe itself: a single non-incremental run still starts clean (bug-1719's behavior is unchanged).\n\nVerification:\n- Test: src/tools/test-build-lock.sh -- (1) a second acquire of the same dir waits until the first releases, (2) a dead owner's lock is taken over, (3) different dirs do not block, (4) xce-build.sh acquires the lock before its first setup_build call; RED before (build-lock.sh missing, then case 4 failing until xce-build.sh was wired), 4/4 GREEN after\n- Suites: n/a for C suites (script-only change); `bash -n src/tools/xce-build.sh` clean\n- Live: two real `xce-build.sh lib asan <same dir>` runs started 5 s apart, 2026-09-29: with the pre-fix script run 1 FAILED (dependency file deleted mid-compile) and run 2 passed; with the fix run 2 logged `Waiting for another xce-build.sh run (PID 1240637) to finish` and both runs completed (`=== Done: lib`)",
"tags": [
"asan",
"concurrency",
@ -8506,7 +8506,9 @@
"related_bugs": [
"bug-1561"
],
"fix_commits": [],
"fix_commits": [
"980a0054"
],
"record": ".wolf/bugs/bug-1604.md"
},
{

32
.wolf/bugs/bug-1604.md
View file

@ -1,16 +1,17 @@
---
id: "bug-1604"
status: "open"
status_updated: "2026-09-24"
status_note: "concurrent xce-build.sh runs race on rm -rf of shared BUILD_DIR (no lock)"
status: "fixed"
status_updated: "2026-09-30"
status_note: ""
component: "build"
severity: "unknown"
severity: "wrong-result"
timestamp: "2026-08-20T10:24:39Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/tools/xce-build.sh"]
files: ["src/tools/xce-build.sh (setup_build)", "src/tools/build-lock.sh"]
tags: ["asan", "concurrency", "multi-agent-race", "overlayfs", "rm-rf", "sandbox-flake", "scheduler", "shared-build-dir", "xce-build.sh"]
related: ["bug-1561"]
fix_commits: ["980a0054"]
legacy: true
---
@ -20,19 +21,30 @@ legacy: true
rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh setup_build()'s rm -rf \$BUILD_DIR step aborted the script under set -euo pipefail before meson setup ran)
## Environment
_Not recorded (migrated from buglog.json)._
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: n/a -- a bash build-orchestration script (runs inside the nix dev shell; glibc 2.42-47 there)
- Toolchain: bash 5.3, meson 1.9.1, gcc (GCC) 15.2.0 (nix dev shell)
- Build: any profile; the race needs two runs against the same build dir, at least one without --incremental
- Code: original sighting Aug 2026 (commit not recorded); reproduced 2026-09-29 on 16b7aa4b's xce-build.sh
- Deployment: n/a -- local build; two sessions/processes sharing one repo checkout
## Reproduction
_Not recorded (migrated from buglog.json)._
1. From src/: `bash tools/xce-build.sh lib asan xce.build.race & sleep 5; bash tools/xce-build.sh lib asan xce.build.race`
2. The second run's setup_build `rm -rf xce.build.race` deletes the first run's in-progress build tree.
- Frequency: 1/1 runs, 2026-09-29 (pre-fix script): run 1 failed with `lib/controller.capnp.c:26384:1: fatal error: opening dependency file lib/libxce.a.p/meson-generated_.._controller.capnp.c.o.d: No such file or directory`, run 2 succeeded; originally seen as `rm: cannot remove ...: Directory not empty` and as a just-verified xce_scheduler binary vanishing
- Evidence: not kept (scratch logs); the steps above reproduce it with the pre-fix script (`git show 16b7aa4b:src/tools/xce-build.sh`)
## Root cause
CORRECTED after further evidence (see fix note): initial overlayfs-flakiness theory was WRONG. The real cause is xce-build.sh defaulting BUILD_DIR to the same "xce.build" for every component and every invocation, with setup_build() doing an unconditional rm -rf "$BUILD_DIR" + fresh meson setup on every non-incremental run. When two xce-build.sh invocations run concurrently in the SAME repo working directory against DIFFERENT components (observed live: this session's "xce-build.sh scheduler asan" raced against a second, independently-running "xce-build.sh controller asan" process, PID 655158, started ~1 min later in the same cwd), they contend for the identical xce.build/ path -- one process's rm -rf can hit a directory the other process is actively writing into ("Directory not empty"), and later, a second clean-build pass can silently delete and replace a just-verified binary out from under a caller who believes their build is done. Confirmed live: xce_scheduler at src/xce.build/scheduler/xce_scheduler, verified present and fresh via ls -la and 'xce_scheduler --version' immediately after this session's scheduler build/test run, was gone minutes later -- the concurrent controller build's rm -rf had removed and was in the process of regenerating xce.build/ for its own component.
## Fix
No code fix applied (out of scope for the build-verification task this session was doing) -- flagging as a real, reproducible hazard rather than env flakiness. Practical mitigation for future sessions/agents: never assume xce.build/ (or any non-incremental xce-build.sh output) is stable once another concurrent xce-build.sh process might be running in the same working directory/worktree -- check 'ps aux | grep xce-build.sh' (or pass a distinct BUILD_DIR / use --incremental) before relying on a binary's presence or mtime surviving past the moment it was checked. Corrects this entry's own original overlayfs-race theory, which was a plausible but incorrect guess made before the concurrent process was discovered.
src/tools/xce-build.sh takes an exclusive per-build-dir lock right after `cd "$SRC_DIR"` (before any setup_build) and releases it on EXIT, so a second run against the same build dir waits for the first to finish (printing `Waiting for another xce-build.sh run (PID N) to finish with <dir>`); runs against different build dirs never block each other. New src/tools/build-lock.sh: build_lock_acquire/build_lock_release. The lock is an mkdir-created directory holding the owner PID (atomic, coreutils only -- the nix dev shell has no flock(1)), placed outside the build dir (the wipe would delete it) under $XCE_BUILD_LOCK_ROOT or $TMPDIR, named by a cksum of the build dir's absolute path; a lock whose owner PID is gone (build killed with SIGKILL) is taken over. It does not change the wipe itself: a single non-incremental run still starts clean (bug-1719's behavior is unchanged).
## Verification
_Not recorded (migrated from buglog.json)._
- Test: src/tools/test-build-lock.sh -- (1) a second acquire of the same dir waits until the first releases, (2) a dead owner's lock is taken over, (3) different dirs do not block, (4) xce-build.sh acquires the lock before its first setup_build call; RED before (build-lock.sh missing, then case 4 failing until xce-build.sh was wired), 4/4 GREEN after
- Suites: n/a for C suites (script-only change); `bash -n src/tools/xce-build.sh` clean
- Live: two real `xce-build.sh lib asan <same dir>` runs started 5 s apart, 2026-09-29: with the pre-fix script run 1 FAILED (dependency file deleted mid-compile) and run 2 passed; with the fix run 2 logged `Waiting for another xce-build.sh run (PID 1240637) to finish` and both runs completed (`=== Done: lib`)
## References
- Related: bug-1561
- Fix commit: 980a0054
- Related: bug-1719 (single-component clean build wipes sibling binaries -- unchanged, use --incremental)