myxce/.wolf/bugs/bug-1604.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	legacy
bug-1604	fixed	2026-09-30		build	wrong-result	2026-08-20T10:24:39Z	2026-09-19T09:00:00Z	1	src/tools/xce-build.sh (setup_build) src/tools/build-lock.sh	asan concurrency multi-agent-race overlayfs rm-rf sandbox-flake scheduler shared-build-dir xce-build.sh	bug-1561	980a0054	true

bug-1604: rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh…

Description

rm: cannot remove 'xce.build/controller/controller_unit_test.p': Directory not empty (xce-build.sh setup_build()'s rm -rf $BUILD_DIR step aborted the script under set -euo pipefail before meson setup ran)

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: n/a -- a bash build-orchestration script (runs inside the nix dev shell; glibc 2.42-47 there)
• Toolchain: bash 5.3, meson 1.9.1, gcc (GCC) 15.2.0 (nix dev shell)
• Build: any profile; the race needs two runs against the same build dir, at least one without --incremental
• Code: original sighting Aug 2026 (commit not recorded); reproduced 2026-09-29 on 16b7aa4b's xce-build.sh
• Deployment: n/a -- local build; two sessions/processes sharing one repo checkout

Reproduction

1. From src/: bash tools/xce-build.sh lib asan xce.build.race & sleep 5; bash tools/xce-build.sh lib asan xce.build.race
2. The second run's setup_build rm -rf xce.build.race deletes the first run's in-progress build tree.

• Frequency: 1/1 runs, 2026-09-29 (pre-fix script): run 1 failed with lib/controller.capnp.c:26384:1: fatal error: opening dependency file lib/libxce.a.p/meson-generated_.._controller.capnp.c.o.d: No such file or directory, run 2 succeeded; originally seen as rm: cannot remove ...: Directory not empty and as a just-verified xce_scheduler binary vanishing
• Evidence: not kept (scratch logs); the steps above reproduce it with the pre-fix script (git show 16b7aa4b:src/tools/xce-build.sh)

Root cause

CORRECTED after further evidence (see fix note): initial overlayfs-flakiness theory was WRONG. The real cause is xce-build.sh defaulting BUILD_DIR to the same "xce.build" for every component and every invocation, with setup_build() doing an unconditional rm -rf "$BUILD_DIR" + fresh meson setup on every non-incremental run. When two xce-build.sh invocations run concurrently in the SAME repo working directory against DIFFERENT components (observed live: this session's "xce-build.sh scheduler asan" raced against a second, independently-running "xce-build.sh controller asan" process, PID 655158, started ~1 min later in the same cwd), they contend for the identical xce.build/ path -- one process's rm -rf can hit a directory the other process is actively writing into ("Directory not empty"), and later, a second clean-build pass can silently delete and replace a just-verified binary out from under a caller who believes their build is done. Confirmed live: xce_scheduler at src/xce.build/scheduler/xce_scheduler, verified present and fresh via ls -la and 'xce_scheduler --version' immediately after this session's scheduler build/test run, was gone minutes later -- the concurrent controller build's rm -rf had removed and was in the process of regenerating xce.build/ for its own component.

Fix

src/tools/xce-build.sh takes an exclusive per-build-dir lock right after cd "$SRC_DIR" (before any setup_build) and releases it on EXIT, so a second run against the same build dir waits for the first to finish (printing Waiting for another xce-build.sh run (PID N) to finish with <dir>); runs against different build dirs never block each other. New src/tools/build-lock.sh: build_lock_acquire/build_lock_release. The lock is an mkdir-created directory holding the owner PID (atomic, coreutils only -- the nix dev shell has no flock(1)), placed outside the build dir (the wipe would delete it) under $XCE_BUILD_LOCK_ROOT or $TMPDIR, named by a cksum of the build dir's absolute path; a lock whose owner PID is gone (build killed with SIGKILL) is taken over. It does not change the wipe itself: a single non-incremental run still starts clean (bug-1719's behavior is unchanged).

Verification

• Test: src/tools/test-build-lock.sh -- (1) a second acquire of the same dir waits until the first releases, (2) a dead owner's lock is taken over, (3) different dirs do not block, (4) xce-build.sh acquires the lock before its first setup_build call; RED before (build-lock.sh missing, then case 4 failing until xce-build.sh was wired), 4/4 GREEN after
• Suites: n/a for C suites (script-only change); bash -n src/tools/xce-build.sh clean
• Live: two real xce-build.sh lib asan <same dir> runs started 5 s apart, 2026-09-29: with the pre-fix script run 1 FAILED (dependency file deleted mid-compile) and run 2 passed; with the fix run 2 logged Waiting for another xce-build.sh run (PID 1240637) to finish and both runs completed (=== Done: lib)

References

• Fix commit: 980a0054
• Related: bug-1719 (single-component clean build wipes sibling binaries -- unchanged, use --incremental)