myxce/.kiro/steering/build-deployment.md

Build & Deployment

Philosophy

A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the asan profile (the default for /xce-build) includes both debug symbols and ASAN; debug builds symbols without ASAN, and release builds (including the nix xcecore production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.

Build Commands

All builds use /xce-build (skill at .claude/skills/xce-build/):

/xce-build              # all components + all tests
/xce-build lib          # libxce + lib tests
/xce-build controller   # lib + controller + controller tests + xce_test driver
/xce-build scheduler    # lib + scheduler + scheduler tests
/xce-build worker       # lib + worker (daemon only, no unit tests)
/xce-build apiserver    # lib + apiserver + xcecli + Go unit tests
/xce-build xcecli       # xcecli + Go tests
/xce-build xceadmin     # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)

xceadmin ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of asan/debug/release). Beyond the build-dir venv, it also produces a real wheel at <build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl (built via pip wheel from the same schema-synced tree the tests just ran against, only after they pass) — pip install it anywhere a compatible Python is available, no nix/meson needed at install time. pyproject.toml declares all 4 real runtime deps (click, pycapnp, PyYAML, jsonschema — verified against every actual import in src/xceadmin/src/xcecommon); the one caveat is pycapnp, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime python3Packages.pycapnp gets.

controller always builds xce_test alongside xce_controller — the driver /xce-test-ha needs, in every profile below. There is no separate build step for it.

Each invocation:

1. nix develop into the hermetic shell
2. meson setup <dir> --buildtype=<type> -Db_sanitize=<mode> (fresh clean build — rm -rf <dir> then reconfigure, unconditionally, for EVERY component including single-component invocations)
3. meson compile -C <dir> [targets...] — per-component invocations pass an explicit target list scoped to just that component (e.g. controller compiles only xce_controller controller_unit_test controller/xce_test); only /xce-build all compiles with no target args, building everything
4. Run registered meson tests (skipped for non-asan profiles — see below)

Consequence (bug-1565) and fix: because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running /xce-build xceadmin (or any other single component) after /xce-build controller in the SAME build dir silently deletes the already-built xce_controller/xce_test/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run /xce-build all once instead of separate single-component invocations, or pass --incremental to a single-component invocation (/xce-build xceadmin --incremental) to reconfigure the existing dir in place (meson setup --reconfigure) instead of wiping it — preserves whatever else was already built there. --incremental is opt-in, not the default: this project's capnp-generation custom_target has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.

Three profiles, three build dirs (/xce-build [component] [profile], second arg optional): asan (default, debug+ASAN, xce.build/, runs the meson test suite) for everyday development and CI-style verification; debug (-O0 -g3 no sanitizer, build.dbg/, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; release (-O3 no sanitizer, build.release/, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a debug rebuild doesn't invalidate an asan binary already in use.

ASAN Integration

ASAN is enabled in every build by default (-Db_sanitize=address). Controller tests set detect_leaks=0 (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.

Meson Structure

src/
├── meson.build                    # top-level: subdir() order, project()
├── lib/meson.build                # libxce shared library + libxce_test
├── controller/meson.build         # xce_controller daemon + controller_unit_test
├── worker/meson.build             # xcecd daemon
├── scheduler/meson.build          # scheduler daemon + test
├── autoscaler/meson.build         # libscaler static lib + provider tools (no daemon binary)
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
├── frontend/xcecli/meson.build    # Go xcecli via tools/go_build
└── xceadmin/meson.build           # xceadmin config-build tool (used by /xce-deploy-ha)

Dependency flow: lib → {controller, worker, scheduler, apiserver}

Test registration:

# controller/meson.build
test('controller test', controller_unit_test,
     suite: 'controller',
     timeout: 300,
     env: ['ASAN_OPTIONS=detect_leaks=0'])

XCE_TESTING flag: The controller_unit_test executable compiles with -DXCE_TESTING to enable test-only seams (hook variables, reset functions).

Nix Flake

Located at src/flake.nix. Provides:

nix develop          # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
nix build .#xcecore   # C daemons + apiserver (release)
nix build .#xcecore_debug  # default package: debug + ASAN
nix build .#xcecli    # CLI only

AppImage: nix build .#xcecore/.#xcecore_debug do not produce an AppImage — they only install xce (a copy of tools/appimage/entry.sh, see src/meson.build) as a plain shell entry-point binary. The xce flake package (appImageEntryPackage in src/flake.nix) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual .AppImage file is produced separately with nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce (see README.md, "3.5 build AppImage package").

Local HA Cluster Deployment

/xce-deploy-ha (skill at .claude/skills/xce-deploy-ha/):

/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

This creates under /tmp/xce-deploy/:

certs/                          # TLS certs for all peers
controller/
  ctrl-1/
    conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
    data/                        # WAL + checkpoint storage
  ctrl-2/
    conf/                        # receives config via OPCode_replConfigPush
    data/
  ctrl-3/ (same as ctrl-2)
xceadmin-site/                  # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
logs/                           # per-node + worker/scheduler log files
scripts/
  start.sh ctrl-N               # launch a single node
  start-worker.sh [name]        # launch xcecd_virtual against the SAME --peers (if worker built)
  start-scheduler.sh            # launch xce_scheduler against the SAME --peers (if scheduler built)
  xceadmin ...                  # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
  cleanup.sh                    # wipe data/standby-conf/logs, keep ctrl-1 config + certs

start-worker.sh/start-scheduler.sh/xceadmin are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run /xce-build worker//xce-build scheduler//xce-build xceadmin first if you need them. scripts/xceadmin is a genuinely standalone install: the wheel is extracted (not installed via a --system-site-packages venv — this nix devShell injects click/pycapnp/PyYAML/jsonschema via $PYTHONPATH, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved $PYTHONPATH at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.

Startup (from deploy dir, via the bash resolved in the deploy environment — see below):

BASH_BIN=$(nix develop ./src --command which bash)   # never a bare `bash`; see below
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv &   # primary first
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv &   # standby (0.5s stagger for election ordering)
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &

# Optional, once at least one controller is up:
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
"$BASH_BIN" scripts/start-scheduler.sh -v &

(-vvv is a positional arg to start.sh, not a VERBOSE env var — the generated script only forwards $*.)

Why resolve $BASH_BIN explicitly: every script this deploy generates is bash (arrays, +=, <<<), and deploy-ha.sh's own shebang has been found locally altered to #!/bin/sh while still using bash-only syntax — it only kept working because this environment's /bin/sh happens to be symlinked to bash. Resolving which bash inside nix develop once and using it explicitly for every invocation (instead of a bare bash or trusting a file's shebang) sidesteps that class of mismatch entirely; deploy-ha.sh also bakes this same resolved path into the shebang of every script it generates.

Clean restart (keeps certs + ctrl-1 config, wipes all data):

kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
"$BASH_BIN" scripts/cleanup.sh
# Then re-start all nodes (and worker/scheduler, if you had them running)

Go Frontend Build

Go builds use src/frontend/Makefile for development, Meson for production:

make build-server     # → /tmp/apiserver
make build-cli        # → /tmp/xce_static (CGO_ENABLED=0)
make test             # unit tests (CGO_ENABLED=0)
make test-backend     # CGO-backed backend tests (needs nix dev shell)
make test-integration # full regression suite (needs XCE_BIN + SRV_URL)

Go modules are independent — apiserver and xcecli have separate go.mod files, no go.work.

Production Artifacts

Artifact	Build command	Output
Controller daemon	nix build .#xcecore	xce_controller binary
Worker daemon	same	xcecd binary
Scheduler daemon	same	xce_scheduler binary
REST API server	same	xce_apiserver binary
CLI tool	nix build .#xcecli	xcecli binary
Admin config CLI (installable)	/xce-build xceadmin	xceadmin-<version>-py3-none-any.whl at <build_dir>/xceadmin/dist/ — pip install it directly
AppImage	nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce	.AppImage bundle (see README.md 3.5)
Debug build	nix build .#xcecore_debug	full debug + ASAN binaries