The original 1536MB/job figure was calibrated around two memory hogs that
the build/de-nix work has since removed: the 293MB-peak serial capnp
compile (now compiled once via libxce_dep) and the 17 scheduler test
executables each statically linking all of scheduler_srcs (now one shared
sched_srcs_lib archive; link peaks ~125-150MB with -g1). Measured per-job
compile RSS ceiling post-dedup is ~125-150MB; 512MB keeps ~3x headroom
while letting an 8GB box run ~8 concurrent compiles instead of 2-3
(t1: mem_jobs goes from ~2-3 to ~9, so the -l ceiling becomes CPU-bound
rather than memory-bound).
SKILL.md's hand-computed formula is updated to match.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The build no longer self-wraps into `nix develop`: src/tools/xce-build.sh now
runs meson+ninja directly on system packages, with a dependency preflight
that fails early naming the missing tool/pkg-config package (DEPS.md).
Dependencies (see DEPS.md for the full tables):
- 11 C libraries from apt (libuv, openssl, libedit, sqlite3, hwloc, libcgroup,
libtirpc, libcyaml, libyaml, zlib, libsystemd) with version floors in
meson.build mirroring the Ubuntu 24.04 baseline.
- sd-bus: default libsystemd, -Duse_elogind=true opt-back for nix-era hosts.
- 6 libraries/tools unavailable in apt become committed meson wrap
subprojects under src/subprojects/ (source closure, ~500 files): the
upstream trees plus packagefiles/ overlays and diff_files patches
registering dependency/program overrides the nix .pc files used to
provide (incl. the '<name>-raw' variable deps the Go CGO path reads).
Fresh-clone reproducibility verified: wiped trees + `meson subprojects
download` reproduce the committed state exactly (fixed librtti.wrap's
dangling patch_directory and c-capnproto's empty packagefiles overlay).
- xceadmin: self-contained venv built from requirements-build.txt (pinned,
offline-seedable via PIP_FIND_LINKS) instead of --system-site-packages.
Build/packaging:
- Makefile facade over xce-build.sh; scripts/ci-ubuntu.sh drives a bare
Ubuntu 24.04 container through deps + build + tests.
- src/tools/xce-package.sh: release-tree tarball packager. Collects
non-base DT_NEEDED libs from the BUILD-tree binaries (staged ldd lies:
the dangling build RPATH makes ldd drop the very lines the naive filter
needs), sets $ORIGIN RPATHs, rebases the xceadmin venv at install time
(kernel does not parse quotes in shebangs), ships systemd unit samples +
install.sh. Local smoke test: all 8 binaries resolve, xceadmin works from
both bin/ wrapper and venv script.
Test fixes found on the way (all environmental, not build bugs):
- harness_test.c setUp(): pin XCE_CONTROLLER_PEERS to a bound-then-closed
loopback port so the CLI-error tests assert the no-controller failure
path even on a dev box running a live cluster on localhost:9990.
- test_test_cli.c: setpgid(0,0) in the spawned-xce_test child. libedit's
sig_handler re-raises received signals via kill(0, sig) -- process GROUP
wide (strace -k evidence) -- and the child previously shared the
harness's pgrp, so an interrupted SSL handshake killed the whole suite
(exit 130).
Also fixed while re-baselining on the de-nix toolchain (pre-existing, now
deterministic): raft_init zeroes pre-vote tally state (pre_vote_in_progress/
pre_votes_granted/pre_vote_replies were left at stack garbage);
push_config_to_standbys clears stale config acks before fan-out
(repl_reset_config_acks) -- SHA-256-derived tags are not monotonic, so a
previous push's ack numerically above the new tag fabricated a barrier
quorum; capnpc-c's generated encode_* memsets the destination first
(skipped Data fields fed garbage into capn_setp).
Validation: asan profile 161/161 meson tests green (one known p2_integration
-110 timing flake on first run, green on re-run); debug + release full
builds OK; tarball smoke-tested on the build host. Cluster rollout on
t1/t2/t3 follows.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Design constraint (tech.md "Process Creation"): the controller, scheduler,
xcecd and anything that starts libxce threads never call fork, vfork,
forkpty, daemon, system, popen or uv_spawn; children start with posix_spawn()
or a systemd unit. A process proven single-threaded may fork, marked
/* xce-fork-ok: <why> */. Tests, harness, autoscaler (not finalized; bug-1969)
and the Go frontend are out of scope.
- scripts/check-no-fork.py scans src/{controller,scheduler,lib,worker}
(comment/string/#if 0 aware); runs as meson test "no-fork rule" in the
lib/controller/scheduler/worker suites and in the pre-commit hook.
- worker: drop xcecd's unused fork-based worker start paths
(process_manager_start_process, process_manager_start_worker; xcecd
starts xcecd_worker over D-Bus) and a dead #if 0 daemonize fork.
- xcecd_worker's fork/uv_spawn/system calls are marked: its call graph
from main(), callbacks included, reaches no pthread_create or libuv
thread-pool call.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: the apiserver and all components ran go test on a fixed list
(./store/... ./utils/... ./controllers/... ./backend/...), so the
middleware, router, cmd and models tests (several security tests among
them) never ran in /xce-build; a deliberately broken middleware test left
the build green.
Fix: run ./... (vendor/ is excluded by the pattern) in both places.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The apiserver and all components ran the frontend Go tests as one list,
'cd frontend/apiserver && go test ... && cd ../xcecli && go test ...'.
set -e ignores a failure of any command in an && list except the last,
so a failing apiserver go test only short-circuited the list: the build
reported success and the xcecli tests never ran. Run each suite as its
own command in a subshell (which also keeps the cd from leaking).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two xce-build.sh runs against the same build dir raced: one run's
setup_build rm -rf hit files the other was writing ('Directory not
empty'), or deleted a binary the other had just built and tested.
xce-build.sh now takes an exclusive per-build-dir lock right after cd-ing
into src/ and holds it until it exits; a second run against the same dir
waits, runs against different dirs are unaffected. The lock
(src/tools/build-lock.sh) is an mkdir lock holding the owner PID, outside
the build dir, keyed by a hash of its absolute path; it needs only
coreutils (the nix dev shell has no flock) and takes over a lock whose
owner died. src/tools/test-build-lock.sh tests it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add a harness) case to src/tools/xce-build.sh following the same
three-part template (setup_build / meson compile / conditional meson
test) already used by scheduler)/worker), building all 6 harness
targets (xce_test plus its 5 test binaries) and running suite harness.
Remove controller/xce_test from the controller) case's compile target
list -- it no longer exists as a controller target. Add harness to the
script's usage/component-list text.
Verified both directions: `/xce-build harness` alone produces xce_test
and all 5 test binaries (4/20/5/12/132 tests, 0 failures); `/xce-build
controller` alone no longer produces controller/xce_test while
controller_unit_test stays green at 1199 tests.
Task 3.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
meson/ninja/capnp/Go only exist inside the hermetic nix dev shell, but
xce-build.sh never entered it itself -- the /xce-build skill documented
wrapping every invocation in `nix develop ./src --command ...` by hand,
so a bare `bash src/tools/xce-build.sh ...` (run directly instead of
through the skill) failed opaquely with "meson: command not found". This
was a recurring, previously-logged gap that resurfaced again today mid
bug-1789 verification.
The script now checks IN_NIX_SHELL at the top and re-execs itself under
`nix develop` if unset, so every caller -- the skill, a bare invocation,
CI, a human at a terminal -- gets it automatically. A nested call (already
inside the shell) is a no-op past that check. README.md's daily-develop
section updated to match; the old "or use nix develop --command" example
is now redundant and removed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Add --jobs N to xce-build.sh: for meson compile it's passed as ninja's -l N
load-average ceiling (re-checked before every job launch for the whole
build, so it adapts if a concurrent session in this shared sandbox loads
the system mid-build), and as a job-count ceiling (-j/--num-processes,
-p) for meson test/go test, which have no load-average equivalent. No
default -- omitting it preserves prior auto-parallelism behavior exactly.
The xce-build skill now computes N live at invocation time from nproc,
/proc/meminfo's MemAvailable, and /proc/loadavg, rather than a fixed
constant, and passes it through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DrxU1dSRYNMV9CDTXHFtcR
Replaces both `cd frontend && make test` call sites (apiserver and
all cases) with direct `go test` invocations covering the identical
package set/flags, mirroring the xcecli case's already-Makefile-free
style. Verified: apiserver case end-to-end, matching the captured
pre-change baseline exactly; all case verified by proxy (byte-
identical replacement line, apiserver instance fully verified) since
a full /xce-build all run is currently blocked earlier in the
pipeline by two pre-existing, unrelated C-test bugs (bug-1723,
bug-1758) -- noted in Implementation Notes for task 4's awareness.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
70+ testsuite/ files across controller/scheduler/lib call mkdtemp()
directly with no shared fixture helper, and none clean up after
themselves. Left unchecked this fills the sandbox disk (recurred three
times now: 2026-06-22, 2026-08-26, 2026-08-30, each larger than the
last -- 696GB at its worst), which then surfaces as unrelated
"Expected Non-NULL"-style cascade failures wherever the next test
happens to call mkdtemp()/persist_init(), easily mistaken for a real
regression.
Retrofitting cleanup into 70+ individual test files would be invasive
and risks breaking tests that inspect a fixture's contents across
multiple TEST_ functions in the same file. Every test run in this
project goes through xce-build.sh, so pruning once there -- age-gated
at 3 hours so a concurrent session's in-progress fixtures are never
touched -- is the lowest-risk single choke point.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
The `worker)` case's meson compile target list omitted xcecd_virtual --
the no-root worker binary deploy-ha.sh/the xce-deploy-ha skill needs
whenever no real systemd manager is present (the common sandbox case).
/xce-build worker alone left it unbuilt, so deploy-ha.sh silently
soft-skipped generating start-worker.sh. Verified: xcecd_virtual now
builds with a fresh mtime, full worker suite still 5/5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds an independent dirty_fingerprint computation to xce-build.sh: sha256
over git diff HEAD's output plus the sorted content of untracked,
non-ignored files (git ls-files --others --exclude-standard), truncated to
8 hex chars. Resolves to empty on a clean tree or if the hash computation
itself fails -- commit_id's own computation and -dirty suffix are left
completely untouched. Zero working-tree/index mutation.
Verified independently by review with real builds (HEAD pinned to control
for concurrent background commits): distinguishes different uncommitted
content, reproduces identical fingerprints for identical content, excludes
.gitignore'd files, degrades gracefully under 3 independently-injected
failure modes without aborting the build, applies to every profile, and
leaves the original feature's regression script passing unmodified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dev-loop build path (/xce-build, the one behind the motivating
incident) never set -Dcommit_id= at all, so every locally-built binary
carried an empty commit id. xce-build.sh now computes commit_id (git
rev-parse --short HEAD, -dirty suffixed on uncommitted changes, unknown
fallback on a genuine git failure) and build_time (UTC date), appends
both to MESON_OPTS for every profile before meson setup, and warns
loudly to stderr if either ever resolves empty anyway.
Task 2.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setup_build() unconditionally rm -rf'd the whole profile build
directory on every single-component invocation -- running
/xce-build xceadmin after /xce-build controller silently deleted
xce_controller/xce_test/etc. from the same build dir.
Added an opt-in --incremental flag (meson setup --reconfigure in
place instead of wiping) rather than changing the default: this
project's capnp-generation custom_target has known missing-
dependency-edge races that a full wipe currently papers over as a
loud, retry-fixable failure. Making incremental the default risked
turning that same gap into silent stale-generated-code instead, so
the safer full-wipe behavior is preserved unless explicitly opted
into.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
Reopens the 2026-07-21 design decision to keep xceadmin outside meson.
Standard onboarding consistency (README 3.3 `nix build .#xcecore` and
3.4 `meson compile` working for xceadmin like every other component)
outweighs the earlier concern about meson's single-artifact model not
fitting a multi-file pip install.
- src/xceadmin/meson.build: custom_target() wrapping install+test,
stamp-file output, build_always_stale (matches this repo's existing
clean-rebuild-every-invocation convention).
- src/xceadmin/meson-build.sh: out-of-tree pinned install + pytest;
stages a repo-mirror/ symlink tree so tests' fixed-relative-path
assumptions (src/configs/, src/schema/, ha/data/) still resolve from
a copied location, without touching their deliberate fail-loud
behavior.
- src/meson.build: subdir('xceadmin') under component != 'xcecli'.
- src/tools/xce-build.sh: xceadmin case now delegates to
`meson compile -C "$BUILD_DIR" xceadmin` instead of its own
pip/venv implementation, like every other component.
Verified both build paths: meson compile (165/165 passed) and
nix build --option sandbox false .#xcecore_debug (128 passed / 37
skipped — cert-dependent tests correctly skip under nix's
git-tracked-only hermetic source). AppImage bundling (README 3.5)
is explicitly deferred to future work, not part of this change.
Add python3 + pycapnp/pyyaml/jsonschema/click/pytest/ruff (+pip) to the
nix dev shell, wire a new `xceadmin` case into xce-build.sh (schema
sync from src/schema/ and src/configs/schema/, pip install by profile,
pytest), package-data config so bundled schemas survive a non-editable
install, xce-build SKILL.md docs, and ruff format/check hooks matching
this repo's stdin-JSON hook convention (not brief.md's env-var sketch,
which would have silently no-op'd).
Verified live end-to-end inside `nix develop ./src` for both the
default (editable) and release (pinned) profiles; found and fixed 3
real bugs along the way (missing pip, PEP 668 externally-managed-
environment, missing package-data for release-profile bundling) —
logged as bug-1458/1459/1460.
Requirements: 5.5, 8.1, 8.2, 8.6
- xce-build.sh: `controller` component now always builds `controller/xce_test`
alongside `xce_controller`, in every profile — xce-test-ha needs both
binaries and previously had to build xce_test as a separate step.
- xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this
skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked
subagent never receives a usable task and silently does nothing. Documented
that this skill should instead always be dispatched via the plain Agent
tool with a fully-resolved, self-contained prompt (not `context: fork`,
which failed) — proven to work end-to-end.
- xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never
invoke via subagent" warning with the precise finding — `context: fork`
(Skill-tool isolation) is the confirmed-broken mechanism (its own teardown
SIGHUPs the whole controller process group mid-soak/mid-deploy), but
Agent-tool subagent dispatch is a different mechanism and is safe
(empirically verified: a `setsid nohup ... & disown` process launched from
inside an Agent subagent survives that subagent returning, reparented to
init). Documented both the foreground-blocking and fire-and-forget
subagent patterns for each skill; xce-deploy-ha additionally flagged that
its `start.sh &` should gain `setsid` to match the soak driver's stronger
detach pattern before relying on unattended subagent dispatch.
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
