myxce/.kiro/steering/build-deployment.md

# Build & Deployment

## Philosophy

A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.

## Build Commands

All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):

```bash
/xce-build              # all components + all tests
/xce-build lib          # libxce + lib tests
/xce-build controller   # lib + controller + controller tests + xce_test driver
/xce-build scheduler    # lib + scheduler + scheduler tests
/xce-build worker       # lib + worker (daemon only, no unit tests)
/xce-build apiserver    # lib + apiserver + xcecli + Go unit tests
/xce-build xcecli       # xcecli + Go tests
/xce-build xceadmin     # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)
```

`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets.

`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.

Each invocation:
1. `nix develop` into the hermetic shell
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations)
3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything
4. Run registered meson tests (skipped for non-`asan` profiles — see below)

**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built `xce_controller`/`xce_test`/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.

**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.

## ASAN Integration

ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.

## Meson Structure

```
src/
├── meson.build                    # top-level: subdir() order, project()
├── lib/meson.build                # libxce shared library + libxce_test
├── controller/meson.build         # xce_controller daemon + controller_unit_test
├── worker/meson.build             # xcecd daemon
├── scheduler/meson.build          # scheduler daemon + test
├── autoscaler/meson.build         # libscaler static lib + provider tools (no daemon binary)
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
├── frontend/xcecli/meson.build    # Go xcecli via tools/go_build
└── xceadmin/meson.build           # xceadmin config-build tool (used by /xce-deploy-ha)
```

**Dependency flow**: `lib → {controller, worker, scheduler, apiserver}`

**Test registration**:
```python
# controller/meson.build
test('controller test', controller_unit_test,
     suite: 'controller',
     timeout: 300,
     env: ['ASAN_OPTIONS=detect_leaks=0'])
```

**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions).

## Nix Flake

Located at `src/flake.nix`. Provides:

```bash
nix develop          # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
nix build .#xcecore   # C daemons + apiserver (release)
nix build .#xcecore_debug  # default package: debug + ASAN
nix build .#xcecli    # CLI only
```

**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced separately with `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` (see README.md, "3.5 build AppImage package").

## Local HA Cluster Deployment

`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`):

```bash
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
```

This creates under `/tmp/xce-deploy/`:
```
certs/                          # TLS certs for all peers
controller/
  ctrl-1/
    conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
    data/                        # WAL + checkpoint storage
  ctrl-2/
    conf/                        # receives config via OPCode_replConfigPush
    data/
  ctrl-3/ (same as ctrl-2)
xceadmin-site/                  # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
logs/                           # per-node + worker/scheduler log files
scripts/
  start.sh ctrl-N               # launch a single node
  start-worker.sh [name]        # launch xcecd_virtual against the SAME --peers (if worker built)
  start-scheduler.sh            # launch xce_scheduler against the SAME --peers (if scheduler built)
  xceadmin ...                  # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
  cleanup.sh                    # wipe data/standby-conf/logs, keep ctrl-1 config + certs
```

`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.

**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below):
```bash
BASH_BIN=$(nix develop ./src --command which bash)   # never a bare `bash`; see below
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv &   # primary first
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv &   # standby (0.5s stagger for election ordering)
sleep 0.5
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &

# Optional, once at least one controller is up:
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
"$BASH_BIN" scripts/start-scheduler.sh -v &
```
(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.)

**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates.

**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
```bash
kill -9 $(pgrep xce_controller) 2>/dev/null || true
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
"$BASH_BIN" scripts/cleanup.sh
# Then re-start all nodes (and worker/scheduler, if you had them running)
```

## Go Frontend Build

Go builds use `src/frontend/Makefile` for development, Meson for production:

```bash
make build-server     # → /tmp/apiserver
make build-cli        # → /tmp/xce_static (CGO_ENABLED=0)
make test             # unit tests (CGO_ENABLED=0)
make test-backend     # CGO-backed backend tests (needs nix dev shell)
make test-integration # full regression suite (needs XCE_BIN + SRV_URL)
```

Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.

## Production Artifacts

| Artifact | Build command | Output |
|----------|-------------|--------|
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
| Worker daemon | same | `xcecd` binary |
| Scheduler daemon | same | `xce_scheduler` binary |
| REST API server | same | `xce_apiserver` binary |
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly |
| AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) |
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |