meson/ninja/capnp/Go only exist inside the hermetic nix dev shell, but
xce-build.sh never entered it itself -- the /xce-build skill documented
wrapping every invocation in `nix develop ./src --command ...` by hand,
so a bare `bash src/tools/xce-build.sh ...` (run directly instead of
through the skill) failed opaquely with "meson: command not found". This
was a recurring, previously-logged gap that resurfaced again today mid
bug-1789 verification.
The script now checks IN_NIX_SHELL at the top and re-execs itself under
`nix develop` if unset, so every caller -- the skill, a bare invocation,
CI, a human at a terminal -- gets it automatically. A nested call (already
inside the shell) is a no-op past that check. README.md's daily-develop
section updated to match; the old "or use nix develop --command" example
is now redundant and removed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Add --jobs N to xce-build.sh: for meson compile it's passed as ninja's -l N
load-average ceiling (re-checked before every job launch for the whole
build, so it adapts if a concurrent session in this shared sandbox loads
the system mid-build), and as a job-count ceiling (-j/--num-processes,
-p) for meson test/go test, which have no load-average equivalent. No
default -- omitting it preserves prior auto-parallelism behavior exactly.
The xce-build skill now computes N live at invocation time from nproc,
/proc/meminfo's MemAvailable, and /proc/loadavg, rather than a fixed
constant, and passes it through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DrxU1dSRYNMV9CDTXHFtcR
Replaces both `cd frontend && make test` call sites (apiserver and
all cases) with direct `go test` invocations covering the identical
package set/flags, mirroring the xcecli case's already-Makefile-free
style. Verified: apiserver case end-to-end, matching the captured
pre-change baseline exactly; all case verified by proxy (byte-
identical replacement line, apiserver instance fully verified) since
a full /xce-build all run is currently blocked earlier in the
pipeline by two pre-existing, unrelated C-test bugs (bug-1723,
bug-1758) -- noted in Implementation Notes for task 4's awareness.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
70+ testsuite/ files across controller/scheduler/lib call mkdtemp()
directly with no shared fixture helper, and none clean up after
themselves. Left unchecked this fills the sandbox disk (recurred three
times now: 2026-06-22, 2026-08-26, 2026-08-30, each larger than the
last -- 696GB at its worst), which then surfaces as unrelated
"Expected Non-NULL"-style cascade failures wherever the next test
happens to call mkdtemp()/persist_init(), easily mistaken for a real
regression.
Retrofitting cleanup into 70+ individual test files would be invasive
and risks breaking tests that inspect a fixture's contents across
multiple TEST_ functions in the same file. Every test run in this
project goes through xce-build.sh, so pruning once there -- age-gated
at 3 hours so a concurrent session's in-progress fixtures are never
touched -- is the lowest-risk single choke point.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
The `worker)` case's meson compile target list omitted xcecd_virtual --
the no-root worker binary deploy-ha.sh/the xce-deploy-ha skill needs
whenever no real systemd manager is present (the common sandbox case).
/xce-build worker alone left it unbuilt, so deploy-ha.sh silently
soft-skipped generating start-worker.sh. Verified: xcecd_virtual now
builds with a fresh mtime, full worker suite still 5/5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds an independent dirty_fingerprint computation to xce-build.sh: sha256
over git diff HEAD's output plus the sorted content of untracked,
non-ignored files (git ls-files --others --exclude-standard), truncated to
8 hex chars. Resolves to empty on a clean tree or if the hash computation
itself fails -- commit_id's own computation and -dirty suffix are left
completely untouched. Zero working-tree/index mutation.
Verified independently by review with real builds (HEAD pinned to control
for concurrent background commits): distinguishes different uncommitted
content, reproduces identical fingerprints for identical content, excludes
.gitignore'd files, degrades gracefully under 3 independently-injected
failure modes without aborting the build, applies to every profile, and
leaves the original feature's regression script passing unmodified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dev-loop build path (/xce-build, the one behind the motivating
incident) never set -Dcommit_id= at all, so every locally-built binary
carried an empty commit id. xce-build.sh now computes commit_id (git
rev-parse --short HEAD, -dirty suffixed on uncommitted changes, unknown
fallback on a genuine git failure) and build_time (UTC date), appends
both to MESON_OPTS for every profile before meson setup, and warns
loudly to stderr if either ever resolves empty anyway.
Task 2.1/13 of build-time-versioning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
setup_build() unconditionally rm -rf'd the whole profile build
directory on every single-component invocation -- running
/xce-build xceadmin after /xce-build controller silently deleted
xce_controller/xce_test/etc. from the same build dir.
Added an opt-in --incremental flag (meson setup --reconfigure in
place instead of wiping) rather than changing the default: this
project's capnp-generation custom_target has known missing-
dependency-edge races that a full wipe currently papers over as a
loud, retry-fixable failure. Making incremental the default risked
turning that same gap into silent stale-generated-code instead, so
the safer full-wipe behavior is preserved unless explicitly opted
into.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
Reopens the 2026-07-21 design decision to keep xceadmin outside meson.
Standard onboarding consistency (README 3.3 `nix build .#xcecore` and
3.4 `meson compile` working for xceadmin like every other component)
outweighs the earlier concern about meson's single-artifact model not
fitting a multi-file pip install.
- src/xceadmin/meson.build: custom_target() wrapping install+test,
stamp-file output, build_always_stale (matches this repo's existing
clean-rebuild-every-invocation convention).
- src/xceadmin/meson-build.sh: out-of-tree pinned install + pytest;
stages a repo-mirror/ symlink tree so tests' fixed-relative-path
assumptions (src/configs/, src/schema/, ha/data/) still resolve from
a copied location, without touching their deliberate fail-loud
behavior.
- src/meson.build: subdir('xceadmin') under component != 'xcecli'.
- src/tools/xce-build.sh: xceadmin case now delegates to
`meson compile -C "$BUILD_DIR" xceadmin` instead of its own
pip/venv implementation, like every other component.
Verified both build paths: meson compile (165/165 passed) and
nix build --option sandbox false .#xcecore_debug (128 passed / 37
skipped — cert-dependent tests correctly skip under nix's
git-tracked-only hermetic source). AppImage bundling (README 3.5)
is explicitly deferred to future work, not part of this change.
Add python3 + pycapnp/pyyaml/jsonschema/click/pytest/ruff (+pip) to the
nix dev shell, wire a new `xceadmin` case into xce-build.sh (schema
sync from src/schema/ and src/configs/schema/, pip install by profile,
pytest), package-data config so bundled schemas survive a non-editable
install, xce-build SKILL.md docs, and ruff format/check hooks matching
this repo's stdin-JSON hook convention (not brief.md's env-var sketch,
which would have silently no-op'd).
Verified live end-to-end inside `nix develop ./src` for both the
default (editable) and release (pinned) profiles; found and fixed 3
real bugs along the way (missing pip, PEP 668 externally-managed-
environment, missing package-data for release-profile bundling) —
logged as bug-1458/1459/1460.
Requirements: 5.5, 8.1, 8.2, 8.6
- xce-build.sh: `controller` component now always builds `controller/xce_test`
alongside `xce_controller`, in every profile — xce-test-ha needs both
binaries and previously had to build xce_test as a separate step.
- xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this
skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked
subagent never receives a usable task and silently does nothing. Documented
that this skill should instead always be dispatched via the plain Agent
tool with a fully-resolved, self-contained prompt (not `context: fork`,
which failed) — proven to work end-to-end.
- xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never
invoke via subagent" warning with the precise finding — `context: fork`
(Skill-tool isolation) is the confirmed-broken mechanism (its own teardown
SIGHUPs the whole controller process group mid-soak/mid-deploy), but
Agent-tool subagent dispatch is a different mechanism and is safe
(empirically verified: a `setsid nohup ... & disown` process launched from
inside an Agent subagent survives that subagent returning, reparented to
init). Documented both the foreground-blocking and fire-and-forget
subagent patterns for each skill; xce-deploy-ha additionally flagged that
its `start.sh &` should gain `setsid` to match the soak driver's stronger
detach pattern before relying on unattended subagent dispatch.
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
