myxce/src/docs/deployment.md

XCE Deployment Guide

Covers building and running all four daemons (controller, worker, scheduler, apiserver) and using xceadmin to author and push cluster configuration. Local dev/test deployment is fully automated; production process supervision (systemd/Docker/etc.) is not shipped in this repo — see Production Gaps before treating anything here as a production runbook.

Component Overview

xceadmin (Python CLI)  --config push (TLS)-->  controller (C, HA cluster, 1..N nodes)
                                                     ^              ^
                                    TLS join         |              | TLS join
                                                     |              |
                                                 worker (C)     scheduler (C)
                                                     ^
                                    CGO-linked libxce client (TCP, TLS, HA-aware)
                                                     |
                                                apiserver (Go, REST) <-- xcecli (Go CLI, REST)

• controller (xce_controller) — the only stateful daemon. Runs as 1 or 3+ nodes for HA (Raft-style election + replication). All other components are its clients.
• worker (xcecd) — one per compute host, runs jobs, joins a controller.
• scheduler (xce_scheduler) — job-to-node placement engine, joins a controller.
• apiserver (xce_apiserver) — Go REST API, joins the controller cluster as a client role (HostRole_apiserver) via a CGO-linked libxce client, not a plain HTTP backend call.
• xcecli — Go CLI, talks to apiserver over REST, never touches the controller directly.
• xceadmin — Python CLI, the only tool that compiles and pushes cluster configuration (cluster.bin) to the controller. Separate concern from the four daemons above.

Every connection into the controller is TLS-encrypted — controller↔controller HA replication, and every libxce-client connection (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller). The controller runs a single iomgr bound to one server SSL_CTX (myapp.c); endpoint_accept()/endpoint_connect() in src/lib/netio.c have no plaintext code path at all — every accepted or outgoing socket is wrapped in SSL_new() before use. What's actually missing is client authentication and identity verification, not encryption: TLS verification is SSL_VERIFY_NONE everywhere in this codebase (server and client contexts alike, no client certificates), so any of these connections buys you encryption-in-transit, not authentication of who's on the other end.

Building

Two different Nix entry points, for two different jobs — don't conflate them:

• nix develop --command — single-component dev builds. /xce-build is not a separate build system; it's nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> under the hood, i.e. the hermetic dev shell driving meson/ninja for exactly one component. This is the reproducible way to build just controller, just worker, etc.
• nix build — whole-stack production packaging. A packaged Nix derivation, all-or-nothing (see Production Gaps item 6) — it always produces every daemon in one result/, never a single one.

# Dev loop, per component (debug + ASAN, runs tests) — wraps `nix develop ./src --command ...`
/xce-build lib           # libxce only
/xce-build controller
/xce-build worker
/xce-build scheduler
/xce-build apiserver      # also builds xcecli in the same pass (same meson step)
/xce-build xcecli         # xcecli alone, if you don't need apiserver
/xce-build xceadmin
/xce-build              # all components

# Profiles: /xce-build [component] [profile]
#   asan (default)  -> xce.build/       debug+ASAN, runs tests
#   debug            -> build.dbg/       -O0 -g3, no sanitizer, gdb-friendly
#   release          -> build.release/   -O3, no sanitizer, build-only

# Production packaging (Nix, whole stack only)
nix build .#xcecore          # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation
nix build .#xcecore_debug    # same, debug + ASAN (this flake's default package)
nix build .#xcecli           # xcecli, as its own separate Nix *package* derivation (Go-only, no C toolchain) — `/xce-build xcecli` above gets you the same binary without going through `nix build` at all

nix build .#xcecore is all-or-nothing — there is no per-daemon Nix target. It produces one result/bin/ containing xce_controller, xcecd (+ xcecd_worker, xcecd_virtual), xce_scheduler, xce_apiserver, the xceadmin venv wrapper, and xce (the AppImage dispatcher script, see below). nix build .#xcecli is the one genuinely separate target.

AppImage (xce-x86_64.AppImage, from nix build .#xcecore) wraps src/tools/appimage/entry.sh, a thin per-daemon dispatcher — it runs exactly one daemon per invocation, never the whole stack:

./xce-x86_64.AppImage controller --myname ctrl-1 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990,ctrl-3=h3:9990" ...
./xce-x86_64.AppImage worker -c /etc/xce/worker.yaml
./xce-x86_64.AppImage scheduler --peers node1=h1:9990
./xce-x86_64.AppImage apiserver -d /opt/xce/apiserver
./xce-x86_64.AppImage shell     # interactive bash with all binaries + haproxy/glow/less on PATH

Run it once per daemon (likely on different hosts) to stand up a cluster.

xceadmin — Cluster Configuration

xceadmin is the only supported way to turn operator-authored YAML into the cluster.bin binary the controller loads, and to push it to a running cluster. It speaks the same raw TLS + Cap'n Proto wire protocol controllers use with each other (no client certificate — verify_mode=ssl.CERT_NONE, matching the controller's own outbound posture).

Install

# Dev, editable, fast iteration (must run inside `nix develop ./src`)
cd src/xceadmin && pip install -e .

# Production / CI (what /xce-build and `nix build .#xcecore` actually use)
/xce-build xceadmin

/xce-build xceadmin always does a clean, non-editable pip install into an out-of-tree venv (--system-site-packages, inheriting nix-provided pycapnp/pyyaml/jsonschema), runs the full pytest suite, syncs the bundled schemas fresh from src/schema/, and produces a standalone xceadmin wrapper script — this is what ships in the AppImage and nix build output.

This venv only works from inside nix develop ./src (or the AppImage's bundled env) — it is not a portable, standalone venv. --system-site-packages means pip install sees click/pycapnp/pyyaml/jsonschema already satisfied by nix's site-packages and skips vendoring its own copies — deliberately, since pycapnp is ABI-sensitive to the exact capnproto C library version, and letting pip resolve/build its own would risk a version mismatch that crashes at runtime. If you source .venv/bin/activate (or run the xceadmin binary directly) outside nix develop, none of those packages are on sys.path and you'll hit ModuleNotFoundError: No module named 'click' (or similar) immediately. Either stay inside nix develop ./src when invoking the built xceadmin, or use the AppImage, which bundles a working environment on PATH already.

Usage

# Compile YAML config into a binary, print its SHA-256 to stderr
xceadmin config build /path/to/configs -o cluster.bin
xceadmin config build /path/to/configs -o -            # to stdout instead of a file

# Push a compiled config to a running controller (any node — primary or standby; redirects to primary automatically, up to 3 hops)
xceadmin config push cluster.bin --controller 127.0.0.1:9991     # host:port; defaults to localhost:9991

# Check whether the controller would accept it, without committing
xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991

config build reads a fixed set of source files from the given directory — xce.yaml, hosts.yaml, queues.yaml, resources.yaml, users.yaml, limits.yaml, scaler.yaml (a missing file → empty section, not an error) — validates each against a bundled JSON schema, runs hard-fail business checks (reserved names, duplicates) and warn-only checks (undefined cross-references), then maps everything into a ClusterConfig Cap'n Proto message. Building twice from identical input is byte-for-byte deterministic (no wall-clock/hostname ever read). Fields that are never settable from YAML: port, hash-table-size tuning, xstrings, candidates (peer membership), configGeneration (controller-assigned).

config push exits non-zero unless the controller's reply reports ackedCount > 0 and no error string; --dry-run exits non-zero on any rejection reason the controller returns.

Known limitations (as of this writing)

• config push against a real controller fails with a client-side decode crash — Fixed 2026-07-30. decode_reply() (src/xceadmin/src/xceadmin/discovery.py:46-77) now unwraps the outer Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}} envelope before decoding the inner RequestReply. Both _redirect_target() and push_config() go through decode_reply(); the old bare-RequestReply decode path is gone. decode_reply() validates body.which() == "replymsg" and raises ReplyDecodeError on anything else. Live-verified against a real deployed controller.
• config push --dry-run's "OK" result is not meaningful evidence of anything — Fixed 2026-08-09 (bug-1506/1507). The controller gained a real OPCode_probeConfigTag handler (config_repl.c:handle_probe_config_tag) that returns the current config_version_tag through the standard Request/RequestReply envelope. probe_config() now wraps the probe in a Request envelope and goes through follow_redirects → decode_reply (same as push_config), so accepted=True means the controller actually replied with a valid tag, and --dry-run reports the real generation number from the cluster.
• No config diff, no rollback, no version history command. xceadmin today has exactly two subcommands: config build and config push [--dry-run].
• No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification.

Deploying the Controller

The controller is the only daemon with real config-file/CLI complexity for cluster topology.

xce_controller \
  --myname ctrl-1 \
  --peers "ctrl-1=host1:9990,ctrl-2=host2:9990,ctrl-3=host3:9990" \
  --workdir /var/lib/xce/controller \
  --certdir /var/lib/xce/certs \
  --logfile /var/log/xce/ctrl-1.log \
  --vcpus 4

• Omit --peers for a single-node, non-HA controller.
• --workdir holds per-node conf/ (cluster config) and data/ (WAL + checkpoint) subdirectories.
• --certdir must contain xce.crt/xce.key — this is the only TLS material in the whole stack, and the controller loads it into the single server SSL_CTX used for every inbound connection it accepts: controller↔controller HA traffic and every worker/scheduler/apiserver/xceadmin client join. Generate a self-signed pair for local/dev use:

  openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
    -keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce"
  

  TLS verification is SSL_VERIFY_NONE everywhere in this codebase — no hostname/identity check is enforced regardless of CA, so a CA-signed cert buys you encryption, not authentication.
• Exactly one node (conventionally the first started) needs its conf/ seeded with an initial cluster.bin (via xceadmin config build) before first boot; other nodes receive config over the wire via config-push once joined.
• WAL and snapshot retention in data/ is fully automatic — no manual purging needed. Every checkpoint trims WAL segments below a replication-safe floor (a segment is never deleted while any standby still needs it to catch up) and rotates old snapshots down to snap_keep_n most recent (default 2 — current + one previous, kept as a last-known-good fallback if a checkpoint is later found corrupt). snap_keep_n is tunable via the cluster config's PersistConfig.snapKeepN field, not a CLI flag.

Local dev cluster (fully automated)

/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

This is the xce-deploy-ha skill wrapping src/tools/deploy-ha.sh, which does everything above automatically: generates certs, builds an initial config from src/configs/*.yaml via xceadmin, and writes scripts/start.sh ctrl-N [extra flags] / scripts/cleanup.sh under the deploy directory. Start nodes with a short stagger for clean election ordering:

scripts/start.sh ctrl-1 -vvv &
sleep 0.5
scripts/start.sh ctrl-2 -vvv &
sleep 0.5
scripts/start.sh ctrl-3 -vvv &

(-vvv is a positional arg to start.sh, not VERBOSE=-vvv — the skill's own docs have this wrong.)

scripts/cleanup.sh wipes logs and every node's data/, plus conf/ for nodes 2..N — ctrl-1's config and all certs survive a cleanup, giving a clean restart without regenerating TLS material or re-authoring config.

Deploying the Worker

xcecd -c /opt/xce/conf/worker.yaml
# or, without a config file:
xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var/log/xce

• Must run as root — check_user() exits immediately otherwise.
• Default config path is /opt/xce/conf/worker.yaml if -c is omitted (and it must exist, or the daemon exits).
• -c, if given, wins over -m/-p (those flags are ignored with a warning).
• Minimal worker.yaml:

  workerconf:
    management_list: ["ctrl-1-host"]
    controller_port: 9990
    log_level: "info"
    log_dir: "/var/log/xce"
  

• --peers "[name=]host:port,..." enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join.
• Worker↔controller traffic is TLS-encrypted (via the shared libxce client, xcecli_ctx_init() → xce_ctx_init()), same posture as everything else in the stack: no client certificate, SSL_VERIFY_NONE.

Deploying the Scheduler

xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.log -v

• No config file at all — pure CLI (argp).
• Gotcha: the default controller target is hard-coded to localhost:9990. There is no -m/-p equivalent — the only way to point the scheduler at any other host or port, even a single non-HA controller, is --peers. Forgetting --peers silently targets localhost:9990 regardless of where your controller actually is.
• -v is repeatable for more verbose logging; default log file is /tmp/xce_scheduler.log.
• TLS-encrypted to the controller, same as the worker (same shared libxce client path).
• Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary through the picker plugin interface (builtin_fcfs is the registered default), not dynamically loaded.

Session recording (live cluster)

Every normal-mode scheduler can record its scheduling sessions to a .replay file for later offline replay/debugging — off by default, no overhead when disabled:

xce_scheduler --peers "ctrl-1=controller-host:9990" --record-dir /var/log/xce/scheduler-recordings -v

• --record-dir DIR sets where recordings are written (default /var/log/xce/). This is the only way record_dir is provisioned — there is no per-request override.
• Recording itself is controlled at runtime, not via a startup flag, through the controller:

  xceadmin scheduler record start [FILENAME] --controller 127.0.0.1:9991   # filename optional, auto-generated if omitted
  xceadmin scheduler record status --controller 127.0.0.1:9991
  xceadmin scheduler record stop --controller 127.0.0.1:9991
  

  These go through a new OPCode_schedAdmin request the controller pass-through-forwards to its attached scheduler (xceadmin → controller → scheduler → reply, same primary-redirect behavior as config push). The controller holds no recording state itself.
• FILENAME (if given) must be a bare filename — no / and no .. segment. The scheduler resolves it under --record-dir and rejects anything that would escape that directory ("invalid record file name").
• A .replay file is a single self-contained Cap'n Proto binary stream (header + snapshot + interleaved events/decisions + periodic checkpoints) — inspect one offline, no running cluster needed:

  xceadmin scheduler dump /var/log/xce/scheduler-recordings/some-session.replay
  

Simulation mode (offline debugging, no controller connection)

--sim runs the scheduler as a standalone replay tool against a previously recorded (or hand-built) .replay file — zero connection to any controller, safe to run against a production recording on a laptop:

xce_scheduler --sim /path/to/session.replay --sim-speed 0        # replay as fast as possible
xce_scheduler --sim /path/to/session.replay --sim-verify         # replay + diff against the recorded decisions
xce_scheduler --sim /path/to/session.replay --sim-output /tmp/resim.replay --sim-alt-events /path/to/edited-events.replay   # "what-if": substitute events, record the new decisions
xce_scheduler --sim /path/to/session.replay --interactive        # step through it interactively (REPL)

• --sim-speed: 0 = as-fast-as-possible, 1 = real-time (default), N>1 = Nx real-time.
• --sim-verify re-derives decisions from the recorded events and asserts they match the decisions the file already recorded — the feature's core "deterministic replay" regression check, usable ad hoc against any recording.
• --sim-alt-events FILE swaps in a different event stream after the initial snapshot for what-if branching; --sim-output FILE records the resulting decisions as a new .replay file.
• --interactive opens a REPL (step, continue, breakpoints, state inspection, snapshot capture) driving the same simulation run a batch --sim would — REPL and batch produce byte-identical decisions for the same input, only the pacing differs.
• Simulation mode never opens a socket to any controller and never touches --peers/--record-dir — see .kiro/specs/scheduler-core/design.md's Boundary Commitments for the full isolation contract.

Deploying apiserver

apiserver -d /opt/xce/apiserver

-d <top_dir> is mandatory despite the --help text implying a default — omitting it exits immediately. <top_dir>/config/ must already exist and contain:

File	Purpose
app.yaml	HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, and the backend: block (address, port, from, key, optional peers for HA) pointing at the controller.
auth.yaml	Auth provider selection (config_file | pam | ldap | oauth2); for config_file, embeds usernames/passwords/roles directly.
rbac.yaml	Permission catalog + role→permission mappings (domains: self, self_ug, etc.). User→role binding comes from auth.yaml, not this file.
logger.yaml	zap logger level/encoding/rotation.

Both app.yaml and logger.yaml are JSON-Schema-validated at startup (fail-fast on malformed config).

Build gotcha — use the Meson/Nix build, not make build-server

make build-server (src/frontend/Makefile) runs a bare go build with no CGO_CFLAGS/CGO_LDFLAGS set. The real controller-connection code (backend/clib.go) is CGO and needs those flags to find libxce's headers. Depending on your environment this either fails to compile, or — if CGO_ENABLED=0 is ambient — silently links backend/stub.go instead, producing a binary where every backend call returns errNoCGO: backend requires CGO. There is no startup warning; the HTTP server comes up fine and only fails at request time.

Use /xce-build apiserver (or nix build .#xcecore) for anything that needs to actually talk to a controller — it computes the correct CGO_CFLAGS/CGO_LDFLAGS from pkg-config and produces xce_apiserver under src/xce.build/frontend/apiserver/ (or the equivalent path for other profiles). Reserve make build-server/make test for the pure-Go unit-test loop (CGO_ENABLED=0 is intentional there).

Relationship to the controller

apiserver is a separate OS process, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as HostRole_apiserver via a CGO-linked libxce client (the same join mechanism workers/schedulers use), configured entirely by app.yaml's backend: block. If backend.address is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. This connection is TLS-encrypted, same as every other libxce client (no client certificate, SSL_VERIFY_NONE) — do not confuse it with apiserver's own separate HTTPS termination for its REST API, covered next.

TLS

apiserver can self-terminate HTTPS (app.yaml's security.https.enabled + cert_file/key_file, default a checked-in self-signed dev cert). A reverse-proxy pattern (sample config/nginx.https.conf, HAProxy mentioned in steering docs) is the intended production direction but is not wired into any startup automation — you'd have to set it up yourself.

Putting It Together: A Full Local Cluster

# 1. Build everything
/xce-build all

# 2. Bring up a 3-node HA controller cluster
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
# (start ctrl-1, then ctrl-2/ctrl-3 with a 0.5s stagger, per the deploy script)

# 3. Start a worker pointed at the cluster
xcecd -m localhost -p 9991 --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -l /tmp/xce-worker

# 4. Start a scheduler pointed at the cluster
xce_scheduler --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -v

# 5. Start apiserver (needs its own config/ dir prepared first — copy from src/frontend/apiserver/config/,
#    at minimum set backend.address/port to match the controller and use auth.yaml.example, not the checked-in auth.yaml)
apiserver -d /path/to/apiserver-topdir

# 6. Use xcecli against apiserver, or xceadmin against the controller directly
xcecli job submit myjob.json
xceadmin config push cluster.bin --controller localhost:9991

Production Gaps

Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features:

1. No process supervision is shipped. No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit sketch exists in src/docs/config-replication-design.md but has a wrong binary name (xcecontroller vs the real xce_controller) and specifies Type=notify, which the controller does not support (it never calls sd_notify) — using it as written will hang at startup and time out. If you adapt it, switch to Type=simple and fix the binary name.
2. src/frontend/apiserver/config/auth.yaml ships plaintext dev credentials checked into git: four accounts total — admin/admin and op7x2k9/CHANGE_ME_dev_only (both role cluster_admin), plus user1/user1 and user2/user2 (role basic_user). Treat all of them as exposed by default — don't ship the checked-in file as-is; copy auth.yaml.example and set real credentials/an external auth provider instead.
3. All controller-bound traffic is encrypted but not authenticated. Controller↔controller HA replication and every libxce client (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller) run over TLS, but SSL_VERIFY_NONE is used throughout — no client certificates, no hostname/identity verification. This buys confidentiality and tamper-resistance in transit, not authentication of the peer; anyone who can reach the port and complete a TLS handshake is treated as trusted the moment the transport-level join succeeds.
4. apiserver's own TLS uses a checked-in self-signed cert with no CA trust story; xcecli hardcodes InsecureSkipVerify: true against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. (This is apiserver's REST-facing HTTPS, separate from its TLS-encrypted libxce connection to the controller in item 3.)
5. xceadmin config push against a real cluster — the decode-crash bug is fixed (2026-07-30, decode_reply() unwraps the Request envelope), but --dry-run still has no real controller-side handler (see Known Limitations item 2).
6. nix build .#xcecore (the packaged derivation) is all-or-nothing — there's no per-daemon Nix package target. For a single component, use /xce-build <component>, which itself runs inside the Nix dev shell (nix develop ./src --command ...) rather than through nix build.

Quick Reference

Daemon	Binary	Config	Controller-pointing flag	TLS
Controller	xce_controller	none (CLI only)	--peers (omit for single-node)	Yes, all inbound (peers + clients), no client cert
Worker	xcecd	worker.yaml (or CLI)	-m/-p (config file wins) or --peers	Yes, no client cert
Scheduler	xce_scheduler	none (CLI only)	--peers (required even for a single non-default controller)	Yes, no client cert
apiserver	xce_apiserver	config/{app,auth,rbac,logger}.yaml	app.yaml's backend: block	To controller: yes, no client cert. Own REST API: optional, self-terminated
xceadmin	xceadmin (Python)	none — takes YAML source dir	--controller host:port	Yes, no client cert