204 lines
16 KiB
Markdown
204 lines
16 KiB
Markdown
# Build & Deployment
|
|
|
|
## Philosophy
|
|
|
|
A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.
|
|
|
|
## Build Commands
|
|
|
|
All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
|
|
|
```bash
|
|
/xce-build # all components + all tests
|
|
/xce-build lib # libxce + lib tests
|
|
/xce-build controller # lib + controller + controller tests
|
|
/xce-build harness # lib + harness + harness tests + xce_test driver
|
|
/xce-build scheduler # lib + scheduler + scheduler tests
|
|
/xce-build worker # lib + worker (daemon only, no unit tests)
|
|
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
|
/xce-build xcecli # xcecli + Go tests
|
|
/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)
|
|
```
|
|
|
|
`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets.
|
|
|
|
`harness` builds the `xce_test` driver that `/xce-test-ha` and the multi-host/resource-value verification workflows need, independently of `controller` — building `controller` alone no longer produces `xce_test` as a side effect. Run `/xce-build harness` (or `/xce-build all`) explicitly to get it.
|
|
|
|
Each invocation:
|
|
1. `nix develop` into the hermetic shell
|
|
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations)
|
|
3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test`; `harness` compiles `xce_test harness_unit_test` plus its four standalone test binaries); only `/xce-build all` compiles with no target args, building everything
|
|
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
|
|
|
|
**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built binaries for that component (e.g. `xce_controller`, `controller_unit_test`) — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.
|
|
|
|
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
|
|
|
|
## ASAN Integration
|
|
|
|
ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.
|
|
|
|
## Meson Structure
|
|
|
|
```
|
|
src/
|
|
├── meson.build # top-level: subdir() order, project()
|
|
├── lib/meson.build # libxce shared library + libxce_test
|
|
├── controller/meson.build # xce_controller daemon + controller_unit_test
|
|
├── harness/meson.build # xce_test driver + harness_unit_test + 4 standalone test binaries
|
|
├── worker/meson.build # xcecd daemon
|
|
├── scheduler/meson.build # scheduler daemon + test
|
|
├── autoscaler/meson.build # libscaler static lib + provider tools (no daemon binary)
|
|
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
|
|
├── frontend/xcecli/meson.build # Go xcecli via tools/go_build
|
|
└── xceadmin/meson.build # xceadmin config-build tool (used by /xce-deploy-ha)
|
|
```
|
|
|
|
**Dependency flow**: `lib → {controller, harness, worker, scheduler, apiserver}`
|
|
|
|
**Test registration**:
|
|
```python
|
|
# controller/meson.build
|
|
test('controller test', controller_unit_test,
|
|
suite: 'controller',
|
|
timeout: 300,
|
|
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
|
```
|
|
|
|
**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions).
|
|
|
|
## Nix Flake
|
|
|
|
Located at `src/flake.nix`. Provides:
|
|
|
|
```bash
|
|
nix develop # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
|
|
nix build .#xcecore # C daemons + apiserver (release)
|
|
nix build .#xcecore_debug # default package: debug + ASAN
|
|
nix build .#xcecli # CLI only
|
|
```
|
|
|
|
**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced by the `xce.appImage` flake output (`nix build .#xce.appImage`, via the `nix-appimage` flake input's `mkAppImage` — no external `nix bundle --bundler` invocation, see README.md 3.5).
|
|
|
|
## Local HA Cluster Deployment
|
|
|
|
`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`):
|
|
|
|
```bash
|
|
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
|
```
|
|
|
|
This creates under `/tmp/xce-deploy/`:
|
|
```
|
|
certs/ # TLS certs for all peers
|
|
controller/
|
|
ctrl-1/
|
|
conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
|
|
data/ # WAL + checkpoint storage
|
|
ctrl-2/
|
|
conf/ # receives config via OPCode_replConfigPush
|
|
data/
|
|
ctrl-3/ (same as ctrl-2)
|
|
xceadmin-site/ # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
|
|
logs/ # per-node + worker/scheduler log files
|
|
scripts/
|
|
start.sh ctrl-N # launch a single node
|
|
start-worker.sh [name] # launch xcecd_virtual against the SAME --peers (if worker built)
|
|
start-scheduler.sh # launch xce_scheduler against the SAME --peers (if scheduler built)
|
|
xceadmin ... # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
|
|
cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs
|
|
```
|
|
|
|
`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.
|
|
|
|
**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below):
|
|
```bash
|
|
BASH_BIN=$(nix develop ./src --command which bash) # never a bare `bash`; see below
|
|
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv & # primary first
|
|
sleep 0.5
|
|
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv & # standby (0.5s stagger for election ordering)
|
|
sleep 0.5
|
|
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &
|
|
|
|
# Optional, once at least one controller is up:
|
|
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
|
|
"$BASH_BIN" scripts/start-scheduler.sh -v &
|
|
```
|
|
(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.)
|
|
|
|
**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates.
|
|
|
|
**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
|
|
```bash
|
|
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
|
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
|
|
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
|
|
"$BASH_BIN" scripts/cleanup.sh
|
|
# Then re-start all nodes (and worker/scheduler, if you had them running)
|
|
```
|
|
|
|
## Controller Peer List from the Environment (`XCE_CONTROLLER_PEERS`)
|
|
|
|
`XCE_CONTROLLER_PEERS` supplies the controller candidate list to any component started without an explicit one, so it does not have to be repeated on every command line.
|
|
|
|
**Value format**: `[name=]host:port,...`, the same as `--peers` (e.g. `ctrl-1=node1:9991,ctrl-2=node2:9991,ctrl-3=node3:9991`). An empty value is the same as unset for every component.
|
|
|
|
**Who reads it, and what wins**:
|
|
|
|
| Component | Explicit value (always wins) | Used when the explicit value is absent | Neither present |
|
|
|-----------|------------------------------|----------------------------------------|-----------------|
|
|
| `xce_controller` | `--peers` | `XCE_CONTROLLER_PEERS` | standalone single-node primary |
|
|
| `xce_scheduler` | `--peers` / `-P` | `XCE_CONTROLLER_PEERS` (not read in `--sim` mode) | single-controller path |
|
|
| `xcecd` / `xcecd_virtual` | `--peers` | `XCE_CONTROLLER_PEERS` | single-controller path (`localhost:9990`) |
|
|
| `xce_test` (harness) | `set masters` (typed, scripted, or in the rc file) | `XCE_CONTROLLER_PEERS` as the `masters` default, read once at startup | `localhost:9990` |
|
|
| `xceadmin` (`--controller` options) | `--controller` | `XCE_CONTROLLER_PEERS`, then the legacy `XCEADMIN_CONTROLLER` | `localhost:9991` |
|
|
|
|
xceadmin strips any `name=` prefix from each entry and treats the list as failover candidates. When a list came from the variable, the daemons' "HA mode"/"HA cluster config" log lines end with ` (from XCE_CONTROLLER_PEERS)`, and parse/validation errors name `XCE_CONTROLLER_PEERS` instead of `--peers`. xceadmin's controller errors end with ` (from $XCE_CONTROLLER_PEERS)` (or ` (from $XCEADMIN_CONTROLLER)`) when the value came from that variable.
|
|
|
|
**Opting out for one invocation**: unset or empty it on that command, e.g. `XCE_CONTROLLER_PEERS= xce_controller ...` (same for `xce_scheduler`, `xcecd`, `xce_test`). Passing an explicit `--peers`/`set masters`/`--controller` also overrides it.
|
|
|
|
**Controller own-name rule**: a list taken from the variable must contain a candidate whose name (or, for an entry without `name=`, whose host) equals the controller's `--myname`, which defaults to the host name. Otherwise the controller refuses to start, before any network activity:
|
|
```
|
|
xce_controller: XCE_CONTROLLER_PEERS (<value>) has no candidate named '<myname>'; pass --myname <one of: ...> or unset XCE_CONTROLLER_PEERS to run standalone
|
|
```
|
|
This stops a stray variable from joining a controller to a cluster it is not part of. An explicit `--peers` keeps its existing handling. Even-count and parse-error checks apply to both sources.
|
|
|
|
**`sudo` and systemd do not pass it on**: `sudo` drops it under the default `env_reset`, and a systemd service never sees the login shell's environment. Pass it explicitly:
|
|
```bash
|
|
sudo --preserve-env=XCE_CONTROLLER_PEERS xcecd ...
|
|
```
|
|
```ini
|
|
# in the unit's [Service] section
|
|
Environment=XCE_CONTROLLER_PEERS=ctrl-1=node1:9991,ctrl-2=node2:9991,ctrl-3=node3:9991
|
|
```
|
|
This matters most for the production worker (`xcecd`), which runs as root and so is usually started through `sudo` or a unit. If the variable was dropped, the component logs `No --peers flag and XCE_CONTROLLER_PEERS unset or empty; ...` at startup and takes the standalone/single-controller path. That line means it never received the list.
|
|
|
|
**`deploy-ha.sh` / `/xce-deploy-ha`**: the generated `start.sh`, `start-worker.sh` and `start-scheduler.sh` still pass `--peers` explicitly, so the variable has no effect on them. The generated `scripts/xceadmin` wrapper exports `XCE_CONTROLLER_PEERS` set to the deployment's own full peer list (plus `XCEADMIN_CONTROLLER` set to its first candidate), so a value left in the caller's shell for another cluster cannot redirect it.
|
|
|
|
**Regression check**: `src/tools/verify-controller-peers-env.sh` covers the variable's handling in the controller, scheduler, worker and harness, plus the `deploy-ha.sh` xceadmin wrapper (xceadmin's own precedence is covered by its pytest suite).
|
|
|
|
## Go Frontend Build
|
|
|
|
Go builds use Meson for both development and production — apiserver and xcecli have no separate build system:
|
|
|
|
```bash
|
|
/xce-build apiserver # → <build_dir>/frontend/apiserver/xce_apiserver, xcecli; runs unit tests (CGO_ENABLED=0)
|
|
meson test -C <build_dir> --suite apiserver # CGO-backed backend tests (needs nix dev shell)
|
|
XCE=<build_dir>/frontend/xcecli/xcecli URL=<url> \
|
|
src/frontend/tests/regression/run.sh # full regression suite
|
|
```
|
|
|
|
Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.
|
|
|
|
## Production Artifacts
|
|
|
|
| Artifact | Build command | Output |
|
|
|----------|-------------|--------|
|
|
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
|
|
| Worker daemon | same | `xcecd` binary |
|
|
| Scheduler daemon | same | `xce_scheduler` binary |
|
|
| REST API server | same | `xce_apiserver` binary |
|
|
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
|
|
| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly |
|
|
| AppImage | `nix build .#xce.appImage` | `xce.AppImage` bundle at `./result` (see README.md 3.5) |
|
|
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |
|