# Build & Deployment

## Philosophy

A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.

## Build Commands

All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):

```bash
/xce-build              # all components + all tests
/xce-build lib          # libxce + lib tests
/xce-build controller   # lib + controller + controller tests
/xce-build harness      # lib + harness + harness tests + xce_test driver
/xce-build scheduler    # lib + scheduler + scheduler tests
/xce-build worker       # lib + worker (daemon only, no unit tests)
/xce-build apiserver    # lib + apiserver + xcecli + Go unit tests
/xce-build xcecli       # xcecli + Go tests
/xce-build xceadmin     # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)
```

`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets.

`harness` builds the `xce_test` driver that `/xce-test-ha` and the multi-host/resource-value verification workflows need, independently of `controller` — building `controller` alone no longer produces `xce_test` as a side effect. Run `/xce-build harness` (or `/xce-build all`) explicitly to get it.

Each invocation:
1. `nix develop` into the hermetic shell
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations)
3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test`; `harness` compiles `xce_test harness_unit_test` plus its four standalone test binaries); only `/xce-build all` compiles with no target args, building everything
4. Run registered meson tests (skipped for non-`asan` profiles — see below)

**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built binaries for that component (e.g. `xce_controller`, `controller_unit_test`) — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.

**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.

## ASAN Integration

ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.

## Meson Structure

```
src/
├── meson.build                    # top-level: subdir() order, project()
├── lib/meson.build                # libxce shared library + libxce_test
├── controller/meson.build         # xce_controller daemon + controller_unit_test
├── harness/meson.build            # xce_test driver + harness_unit_test + 4 standalone test binaries
├── worker/meson.build             # xcecd daemon
├── scheduler/meson.build          # scheduler daemon + test
├── autoscaler/meson.build         # libscaler static lib + provider tools (no daemon binary)
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
├── frontend/xcecli/meson.build    # Go xcecli via tools/go_build
└── xceadmin/meson.build           # xceadmin config-build tool (used by /xce-deploy-ha)
```

**Dependency flow**: `lib → {controller, harness, worker, scheduler, apiserver}`

**Test registration**:
```python
# controller/meson.build
test('controller test', controller_unit_test,
     suite: 'controller',
     timeout: 300,
     env: ['ASAN_OPTIONS=detect_leaks=0'])
```

**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions).

## Nix Flake

Located at `src/flake.nix`. Provides:

```bash
nix develop          # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
nix build .#xcecore   # C daemons + apiserver (release)
nix build .#xcecore_debug  # default package: debug + ASAN
nix build .#xcecli    # CLI only
```

**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced by the `xce.appImage` flake output (`nix build .#xce.appImage`, via the `nix-appimage` flake input's `mkAppImage` — no external `nix bundle --bundler` invocation, see README.md 3.5).

## Local HA Cluster Deployment

`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`):

```bash
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
```

This creates under `/tmp/xce-deploy/`:
```
certs/                          # TLS certs for all peers
controller/
  ctrl-1/
    conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
    data/                        # WAL + checkpoint storage
  ctrl-2/
    conf/                        # receives config via OPCode_replConfigPush
    data/
  ctrl-3/ (same as ctrl-2)
xceadmin-site/                  # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
logs/                           # per-node + worker/scheduler log files
scripts/
  start.sh ctrl-N               # launch a single node
  start-worker.sh [name]        # launch xcecd_virtual against the SAME --peers (if worker built)
  start-scheduler.sh            # launch xce_scheduler against the SAME --peers (if scheduler built)
  xceadmin ...                  # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
  cleanup.sh                    # wipe data/standby-conf/logs, keep ctrl-1 config + certs
```

`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.

**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below):
```bash
BASH_BIN=$(nix develop ./src --command which bash)   # never a bare `bash`; see below
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv &   # primary first
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv &   # standby (0.5s stagger for election ordering)
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &

# Optional, once at least one controller is up:
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
"$BASH_BIN" scripts/start-scheduler.sh -v &
```
(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.)

**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates.

**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
"$BASH_BIN" scripts/cleanup.sh
# Then re-start all nodes (and worker/scheduler, if you had them running)
```

## Controller Peer List from the Environment (`XCE_CONTROLLER_PEERS`)

`XCE_CONTROLLER_PEERS` supplies the controller candidate list to any component started without an explicit one, so it does not have to be repeated on every command line.

**Value format**: `[name=]host:port,...`, the same as `--peers` (e.g. `ctrl-1=node1:9991,ctrl-2=node2:9991,ctrl-3=node3:9991`). An empty value is the same as unset for every component.

**Who reads it, and what wins**:

| Component | Explicit value (always wins) | Used when the explicit value is absent | Neither present |
|-----------|------------------------------|----------------------------------------|-----------------|
| `xce_controller` | `--peers` | `XCE_CONTROLLER_PEERS` | standalone single-node primary |
| `xce_scheduler` | `--peers` / `-P` | `XCE_CONTROLLER_PEERS` (not read in `--sim` mode) | single-controller path |
| `xcecd` / `xcecd_virtual` | `--peers` | `XCE_CONTROLLER_PEERS` | single-controller path (`localhost:9990`) |
| `xce_test` (harness) | `set masters` (typed, scripted, or in the rc file) | `XCE_CONTROLLER_PEERS` as the `masters` default, read once at startup | `localhost:9990` |
| `xceadmin` (`--controller` options) | `--controller` | `XCE_CONTROLLER_PEERS`, then the legacy `XCEADMIN_CONTROLLER` | `localhost:9991` |

xceadmin strips any `name=` prefix from each entry and treats the list as failover candidates. When a list came from the variable, the daemons' "HA mode"/"HA cluster config" log lines end with ` (from XCE_CONTROLLER_PEERS)`, and parse/validation errors name `XCE_CONTROLLER_PEERS` instead of `--peers`. xceadmin's controller errors end with ` (from $XCE_CONTROLLER_PEERS)` (or ` (from $XCEADMIN_CONTROLLER)`) when the value came from that variable.

**Opting out for one invocation**: unset or empty it on that command, e.g. `XCE_CONTROLLER_PEERS= xce_controller ...` (same for `xce_scheduler`, `xcecd`, `xce_test`). Passing an explicit `--peers`/`set masters`/`--controller` also overrides it.

**Controller own-name rule**: a list taken from the variable must contain a candidate whose name (or, for an entry without `name=`, whose host) equals the controller's `--myname`, which defaults to the host name. Otherwise the controller refuses to start, before any network activity:
```
xce_controller: XCE_CONTROLLER_PEERS (<value>) has no candidate named '<myname>'; pass --myname <one of: ...> or unset XCE_CONTROLLER_PEERS to run standalone
```
This stops a stray variable from joining a controller to a cluster it is not part of. An explicit `--peers` keeps its existing handling. Even-count and parse-error checks apply to both sources.

**`sudo` and systemd do not pass it on**: `sudo` drops it under the default `env_reset`, and a systemd service never sees the login shell's environment. Pass it explicitly:
```bash
sudo --preserve-env=XCE_CONTROLLER_PEERS xcecd ...
```
```ini
# in the unit's [Service] section
Environment=XCE_CONTROLLER_PEERS=ctrl-1=node1:9991,ctrl-2=node2:9991,ctrl-3=node3:9991
```
This matters most for the production worker (`xcecd`), which runs as root and so is usually started through `sudo` or a unit. If the variable was dropped, the component logs `No --peers flag and XCE_CONTROLLER_PEERS unset or empty; ...` at startup and takes the standalone/single-controller path. That line means it never received the list.

**`deploy-ha.sh` / `/xce-deploy-ha`**: the generated `start.sh`, `start-worker.sh` and `start-scheduler.sh` still pass `--peers` explicitly, so the variable has no effect on them. The generated `scripts/xceadmin` wrapper exports `XCE_CONTROLLER_PEERS` set to the deployment's own full peer list (plus `XCEADMIN_CONTROLLER` set to its first candidate), so a value left in the caller's shell for another cluster cannot redirect it.

**Regression check**: `src/tools/verify-controller-peers-env.sh` covers the variable's handling in the controller, scheduler, worker and harness, plus the `deploy-ha.sh` xceadmin wrapper (xceadmin's own precedence is covered by its pytest suite).

## Go Frontend Build

Go builds use Meson for both development and production — apiserver and xcecli have no separate build system:

```bash
/xce-build apiserver                              # → <build_dir>/frontend/apiserver/xce_apiserver, xcecli; runs unit tests (CGO_ENABLED=0)
meson test -C <build_dir> --suite apiserver       # CGO-backed backend tests (needs nix dev shell)
XCE=<build_dir>/frontend/xcecli/xcecli URL=<url> \
    src/frontend/tests/regression/run.sh          # full regression suite
```

Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.

## Production Artifacts

| Artifact | Build command | Output |
|----------|-------------|--------|
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
| Worker daemon | same | `xcecd` binary |
| Scheduler daemon | same | `xce_scheduler` binary |
| REST API server | same | `xce_apiserver` binary |
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly |
| AppImage | `nix build .#xce.appImage` | `xce.AppImage` bundle at `./result` (see README.md 3.5) |
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |