---
id: "bug-1604"
status: "fixed"
status_updated: "2026-09-30"
status_note: ""
component: "build"
severity: "wrong-result"
timestamp: "2026-08-20T10:24:39Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/tools/xce-build.sh (setup_build)", "src/tools/build-lock.sh"]
tags: ["asan", "concurrency", "multi-agent-race", "overlayfs", "rm-rf", "sandbox-flake", "scheduler", "shared-build-dir", "xce-build.sh"]
related: ["bug-1561"]
fix_commits: ["980a0054"]
legacy: true
---

# bug-1604: rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh…

## Description
rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh setup_build()'s rm -rf \$BUILD_DIR step aborted the script under set -euo pipefail before meson setup ran)

## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: n/a -- a bash build-orchestration script (runs inside the nix dev shell; glibc 2.42-47 there)
- Toolchain: bash 5.3, meson 1.9.1, gcc (GCC) 15.2.0 (nix dev shell)
- Build: any profile; the race needs two runs against the same build dir, at least one without --incremental
- Code: original sighting Aug 2026 (commit not recorded); reproduced 2026-09-29 on 16b7aa4b's xce-build.sh
- Deployment: n/a -- local build; two sessions/processes sharing one repo checkout

## Reproduction
1. From src/: `bash tools/xce-build.sh lib asan xce.build.race & sleep 5; bash tools/xce-build.sh lib asan xce.build.race`
2. The second run's setup_build `rm -rf xce.build.race` deletes the first run's in-progress build tree.
- Frequency: 1/1 runs, 2026-09-29 (pre-fix script): run 1 failed with `lib/controller.capnp.c:26384:1: fatal error: opening dependency file lib/libxce.a.p/meson-generated_.._controller.capnp.c.o.d: No such file or directory`, run 2 succeeded; originally seen as `rm: cannot remove ...: Directory not empty` and as a just-verified xce_scheduler binary vanishing
- Evidence: not kept (scratch logs); the steps above reproduce it with the pre-fix script (`git show 16b7aa4b:src/tools/xce-build.sh`)

## Root cause
CORRECTED after further evidence (see fix note): initial overlayfs-flakiness theory was WRONG. The real cause is xce-build.sh defaulting BUILD_DIR to the same "xce.build" for every component and every invocation, with setup_build() doing an unconditional rm -rf "$BUILD_DIR" + fresh meson setup on every non-incremental run. When two xce-build.sh invocations run concurrently in the SAME repo working directory against DIFFERENT components (observed live: this session's "xce-build.sh scheduler asan" raced against a second, independently-running "xce-build.sh controller asan" process, PID 655158, started ~1 min later in the same cwd), they contend for the identical xce.build/ path -- one process's rm -rf can hit a directory the other process is actively writing into ("Directory not empty"), and later, a second clean-build pass can silently delete and replace a just-verified binary out from under a caller who believes their build is done. Confirmed live: xce_scheduler at src/xce.build/scheduler/xce_scheduler, verified present and fresh via ls -la and 'xce_scheduler --version' immediately after this session's scheduler build/test run, was gone minutes later -- the concurrent controller build's rm -rf had removed and was in the process of regenerating xce.build/ for its own component.

## Fix
src/tools/xce-build.sh takes an exclusive per-build-dir lock right after `cd "$SRC_DIR"` (before any setup_build) and releases it on EXIT, so a second run against the same build dir waits for the first to finish (printing `Waiting for another xce-build.sh run (PID N) to finish with <dir>`); runs against different build dirs never block each other. New src/tools/build-lock.sh: build_lock_acquire/build_lock_release. The lock is an mkdir-created directory holding the owner PID (atomic, coreutils only -- the nix dev shell has no flock(1)), placed outside the build dir (the wipe would delete it) under $XCE_BUILD_LOCK_ROOT or $TMPDIR, named by a cksum of the build dir's absolute path; a lock whose owner PID is gone (build killed with SIGKILL) is taken over. It does not change the wipe itself: a single non-incremental run still starts clean (bug-1719's behavior is unchanged).

## Verification
- Test: src/tools/test-build-lock.sh -- (1) a second acquire of the same dir waits until the first releases, (2) a dead owner's lock is taken over, (3) different dirs do not block, (4) xce-build.sh acquires the lock before its first setup_build call; RED before (build-lock.sh missing, then case 4 failing until xce-build.sh was wired), 4/4 GREEN after
- Suites: n/a for C suites (script-only change); `bash -n src/tools/xce-build.sh` clean
- Live: two real `xce-build.sh lib asan <same dir>` runs started 5 s apart, 2026-09-29: with the pre-fix script run 1 FAILED (dependency file deleted mid-compile) and run 2 passed; with the fix run 2 logged `Waiting for another xce-build.sh run (PID 1240637) to finish` and both runs completed (`=== Done: lib`)

## References
- Fix commit: 980a0054
- Related: bug-1719 (single-component clean build wipes sibling binaries -- unchanged, use --incremental)