myxce/.kiro/steering/build-deployment.md

Build & Deployment

Philosophy

A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the asan profile (the default for /xce-build) includes both debug symbols and ASAN; debug builds symbols without ASAN, and release builds (including the nix xcecore production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.

Build Commands

All builds use /xce-build (skill at .claude/skills/xce-build/):

/xce-build              # all components + all tests
/xce-build lib          # libxce + lib tests
/xce-build controller   # lib + controller + controller tests
/xce-build harness      # lib + harness + harness tests + xce_test driver
/xce-build scheduler    # lib + scheduler + scheduler tests
/xce-build worker       # lib + worker (daemon only, no unit tests)
/xce-build apiserver    # lib + apiserver + xcecli + Go unit tests
/xce-build xcecli       # xcecli + Go tests
/xce-build xceadmin     # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)

xceadmin ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of asan/debug/release). Beyond the build-dir venv, it also produces a real wheel at <build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl (built via pip wheel from the same schema-synced tree the tests just ran against, only after they pass) — pip install it anywhere a compatible Python is available, no nix/meson needed at install time. pyproject.toml declares all 4 real runtime deps (click, pycapnp, PyYAML, jsonschema — verified against every actual import in src/xceadmin/src/xcecommon); the one caveat is pycapnp, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime python3Packages.pycapnp gets.

harness builds the xce_test driver that /xce-test-ha and the multi-host/resource-value verification workflows need, independently of controller — building controller alone no longer produces xce_test as a side effect. Run /xce-build harness (or /xce-build all) explicitly to get it.

Each invocation:

1. nix develop into the hermetic shell
2. meson setup <dir> --buildtype=<type> -Db_sanitize=<mode> (fresh clean build — rm -rf <dir> then reconfigure, unconditionally, for EVERY component including single-component invocations)
3. meson compile -C <dir> [targets...] — per-component invocations pass an explicit target list scoped to just that component (e.g. controller compiles only xce_controller controller_unit_test; harness compiles xce_test harness_unit_test plus its four standalone test binaries); only /xce-build all compiles with no target args, building everything
4. Run registered meson tests (skipped for non-asan profiles — see below)

Consequence (bug-1565) and fix: because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running /xce-build xceadmin (or any other single component) after /xce-build controller in the SAME build dir silently deletes the already-built binaries for that component (e.g. xce_controller, controller_unit_test) — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run /xce-build all once instead of separate single-component invocations, or pass --incremental to a single-component invocation (/xce-build xceadmin --incremental) to reconfigure the existing dir in place (meson setup --reconfigure) instead of wiping it — preserves whatever else was already built there. --incremental is opt-in, not the default: this project's capnp-generation custom_target has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.

Three profiles, three build dirs (/xce-build [component] [profile], second arg optional): asan (default, debug+ASAN, xce.build/, runs the meson test suite) for everyday development and CI-style verification; debug (-O0 -g3 no sanitizer, build.dbg/, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; release (-O3 no sanitizer, build.release/, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a debug rebuild doesn't invalidate an asan binary already in use.

ASAN Integration

ASAN is enabled in every build by default (-Db_sanitize=address). Controller tests set detect_leaks=0 (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.

Meson Structure

src/
├── meson.build                    # top-level: subdir() order, project()
├── lib/meson.build                # libxce shared library + libxce_test
├── controller/meson.build         # xce_controller daemon + controller_unit_test
├── harness/meson.build            # xce_test driver + harness_unit_test + 4 standalone test binaries
├── worker/meson.build             # xcecd daemon
├── scheduler/meson.build          # scheduler daemon + test
├── autoscaler/meson.build         # libscaler static lib + provider tools (no daemon binary)
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
├── frontend/xcecli/meson.build    # Go xcecli via tools/go_build
└── xceadmin/meson.build           # xceadmin config-build tool (used by /xce-deploy-ha)

Dependency flow: lib → {controller, harness, worker, scheduler, apiserver}

Test registration:

# controller/meson.build
test('controller test', controller_unit_test,
     suite: 'controller',
     timeout: 300,
     env: ['ASAN_OPTIONS=detect_leaks=0'])

XCE_TESTING flag: The controller_unit_test executable compiles with -DXCE_TESTING to enable test-only seams (hook variables, reset functions).

Nix Flake

Located at src/flake.nix. Provides:

nix develop          # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
nix build .#xcecore   # C daemons + apiserver (release)
nix build .#xcecore_debug  # default package: debug + ASAN
nix build .#xcecli    # CLI only

AppImage: nix build .#xcecore/.#xcecore_debug do not produce an AppImage — they only install xce (a copy of tools/appimage/entry.sh, see src/meson.build) as a plain shell entry-point binary. The xce flake package (appImageEntryPackage in src/flake.nix) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual .AppImage file is produced by the xce.appImage flake output (nix build .#xce.appImage, via the nix-appimage flake input's mkAppImage — no external nix bundle --bundler invocation, see README.md 3.5).

Local HA Cluster Deployment

/xce-deploy-ha (skill at .claude/skills/xce-deploy-ha/):

/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

This creates under /tmp/xce-deploy/:

certs/                          # TLS certs for all peers
controller/
  ctrl-1/
    conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
    data/                        # WAL + checkpoint storage
  ctrl-2/
    conf/                        # receives config via OPCode_replConfigPush
    data/
  ctrl-3/ (same as ctrl-2)
xceadmin-site/                  # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
logs/                           # per-node + worker/scheduler log files
scripts/
  start.sh ctrl-N               # launch a single node
  start-worker.sh [name]        # launch xcecd_virtual against the SAME --peers (if worker built)
  start-scheduler.sh            # launch xce_scheduler against the SAME --peers (if scheduler built)
  xceadmin ...                  # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
  cleanup.sh                    # wipe data/standby-conf/logs, keep ctrl-1 config + certs

start-worker.sh/start-scheduler.sh/xceadmin are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run /xce-build worker//xce-build scheduler//xce-build xceadmin first if you need them. scripts/xceadmin is a genuinely standalone install: the wheel is extracted (not installed via a --system-site-packages venv — this nix devShell injects click/pycapnp/PyYAML/jsonschema via $PYTHONPATH, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved $PYTHONPATH at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.

Startup (from deploy dir, via the bash resolved in the deploy environment — see below):

BASH_BIN=$(nix develop ./src --command which bash)   # never a bare `bash`; see below
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv &   # primary first
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv &   # standby (0.5s stagger for election ordering)
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &

# Optional, once at least one controller is up:
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
"$BASH_BIN" scripts/start-scheduler.sh -v &

(-vvv is a positional arg to start.sh, not a VERBOSE env var — the generated script only forwards $*.)

Why resolve $BASH_BIN explicitly: every script this deploy generates is bash (arrays, +=, <<<), and deploy-ha.sh's own shebang has been found locally altered to #!/bin/sh while still using bash-only syntax — it only kept working because this environment's /bin/sh happens to be symlinked to bash. Resolving which bash inside nix develop once and using it explicitly for every invocation (instead of a bare bash or trusting a file's shebang) sidesteps that class of mismatch entirely; deploy-ha.sh also bakes this same resolved path into the shebang of every script it generates.

Clean restart (keeps certs + ctrl-1 config, wipes all data):

kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
"$BASH_BIN" scripts/cleanup.sh
# Then re-start all nodes (and worker/scheduler, if you had them running)

Controller Peer List from the Environment (XCE_CONTROLLER_PEERS)

XCE_CONTROLLER_PEERS supplies the controller candidate list to any component started without an explicit one, so it does not have to be repeated on every command line.

Value format: [name=]host:port,..., the same as --peers (e.g. ctrl-1=node1:9991,ctrl-2=node2:9991,ctrl-3=node3:9991). An empty value is the same as unset for every component.

Who reads it, and what wins:

Component	Explicit value (always wins)	Used when the explicit value is absent	Neither present
xce_controller	--peers	XCE_CONTROLLER_PEERS	standalone single-node primary
xce_scheduler	--peers / -P	XCE_CONTROLLER_PEERS (not read in --sim mode)	single-controller path
xcecd / xcecd_virtual	--peers	XCE_CONTROLLER_PEERS	single-controller path (localhost:9990)
xce_test (harness)	set masters (typed, scripted, or in the rc file)	XCE_CONTROLLER_PEERS as the masters default, read once at startup	localhost:9990
xceadmin (--controller options)	--controller	XCE_CONTROLLER_PEERS, then the legacy XCEADMIN_CONTROLLER	localhost:9991

xceadmin strips any name= prefix from each entry and treats the list as failover candidates. When a list came from the variable, the daemons' "HA mode"/"HA cluster config" log lines end with (from XCE_CONTROLLER_PEERS), and parse/validation errors name XCE_CONTROLLER_PEERS instead of --peers. xceadmin's controller errors end with (from $XCE_CONTROLLER_PEERS) (or (from $XCEADMIN_CONTROLLER)) when the value came from that variable.

Opting out for one invocation: unset or empty it on that command, e.g. XCE_CONTROLLER_PEERS= xce_controller ... (same for xce_scheduler, xcecd, xce_test). Passing an explicit --peers/set masters/--controller also overrides it.

Controller own-name rule: a list taken from the variable must contain a candidate whose name (or, for an entry without name=, whose host) equals the controller's --myname, which defaults to the host name. Otherwise the controller refuses to start, before any network activity:

xce_controller: XCE_CONTROLLER_PEERS (<value>) has no candidate named '<myname>'; pass --myname <one of: ...> or unset XCE_CONTROLLER_PEERS to run standalone

This stops a stray variable from joining a controller to a cluster it is not part of. An explicit --peers keeps its existing handling. Even-count and parse-error checks apply to both sources.

sudo and systemd do not pass it on: sudo drops it under the default env_reset, and a systemd service never sees the login shell's environment. Pass it explicitly:

sudo --preserve-env=XCE_CONTROLLER_PEERS xcecd ...

# in the unit's [Service] section
Environment=XCE_CONTROLLER_PEERS=ctrl-1=node1:9991,ctrl-2=node2:9991,ctrl-3=node3:9991

This matters most for the production worker (xcecd), which runs as root and so is usually started through sudo or a unit. If the variable was dropped, the component logs No --peers flag and XCE_CONTROLLER_PEERS unset or empty; ... at startup and takes the standalone/single-controller path. That line means it never received the list.

deploy-ha.sh / /xce-deploy-ha: the generated start.sh, start-worker.sh and start-scheduler.sh still pass --peers explicitly, so the variable has no effect on them. The generated scripts/xceadmin wrapper exports XCE_CONTROLLER_PEERS set to the deployment's own full peer list (plus XCEADMIN_CONTROLLER set to its first candidate), so a value left in the caller's shell for another cluster cannot redirect it.

Regression check: src/tools/verify-controller-peers-env.sh covers the variable's handling in the controller, scheduler, worker and harness, plus the deploy-ha.sh xceadmin wrapper (xceadmin's own precedence is covered by its pytest suite).

Go Frontend Build

Go builds use Meson for both development and production — apiserver and xcecli have no separate build system:

/xce-build apiserver                              # → <build_dir>/frontend/apiserver/xce_apiserver, xcecli; runs unit tests (CGO_ENABLED=0)
meson test -C <build_dir> --suite apiserver       # CGO-backed backend tests (needs nix dev shell)
XCE=<build_dir>/frontend/xcecli/xcecli URL=<url> \
    src/frontend/tests/regression/run.sh          # full regression suite

Go modules are independent — apiserver and xcecli have separate go.mod files, no go.work.

Production Artifacts

Artifact	Build command	Output
Controller daemon	nix build .#xcecore	xce_controller binary
Worker daemon	same	xcecd binary
Scheduler daemon	same	xce_scheduler binary
REST API server	same	xce_apiserver binary
CLI tool	nix build .#xcecli	xcecli binary
Admin config CLI (installable)	/xce-build xceadmin	xceadmin-<version>-py3-none-any.whl at <build_dir>/xceadmin/dist/ — pip install it directly
AppImage	nix build .#xce.appImage	xce.AppImage bundle at ./result (see README.md 3.5)
Debug build	nix build .#xcecore_debug	full debug + ASAN binaries