myxce/src/docs/deployment.md

XCE Deployment Guide

Covers building and running all four daemons (controller, worker, scheduler, apiserver) and using xceadmin to author and push cluster configuration. Local dev/test deployment is fully automated; production process supervision (systemd/Docker/etc.) is not shipped in this repo — see Production Gaps before treating anything here as a production runbook.

Component Overview

xceadmin (Python CLI)  --config push (TLS)-->  controller (C, HA cluster, 1..N nodes)
                                                     ^              ^
                                    TLS join         |              | TLS join
                                                     |              |
                                                 worker (C)     scheduler (C)
                                                     ^
                                    CGO-linked libxce client (TCP, TLS, HA-aware)
                                                     |
                                                apiserver (Go, REST) <-- xcecli (Go CLI, REST)

• controller (xce_controller) — the only stateful daemon. Runs as 1 or 3+ nodes for HA (Raft-style election + replication). All other components are its clients.
• worker (xcecd) — one per compute host, runs jobs, joins a controller.
• scheduler (xce_scheduler) — job-to-node placement engine, joins a controller.
• apiserver (xce_apiserver) — Go REST API, joins the controller cluster as a client role (HostRole_apiserver) via a CGO-linked libxce client, not a plain HTTP backend call.
• xcecli — Go CLI, talks to apiserver over REST, never touches the controller directly.
• xceadmin — Python CLI, the only tool that compiles and pushes cluster configuration (cluster.bin) to the controller. Separate concern from the four daemons above.

Every connection into the controller is TLS-encrypted — controller↔controller HA replication, and every libxce-client connection (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller). The controller runs a single iomgr bound to one server SSL_CTX (myapp.c); endpoint_accept()/endpoint_connect() in src/lib/netio.c have no plaintext code path at all — every accepted or outgoing socket is wrapped in SSL_new() before use. What's actually missing is client authentication and identity verification, not encryption: TLS verification is SSL_VERIFY_NONE everywhere in this codebase (server and client contexts alike, no client certificates), so any of these connections buys you encryption-in-transit, not authentication of who's on the other end.

Building

Two different Nix entry points, for two different jobs — don't conflate them:

• nix develop --command — single-component dev builds. /xce-build is not a separate build system; it's nix develop ./src --command bash src/tools/xce-build.sh <component> <profile> under the hood, i.e. the hermetic dev shell driving meson/ninja for exactly one component. This is the reproducible way to build just controller, just worker, etc.
• nix build — whole-stack production packaging. A packaged Nix derivation, all-or-nothing (see Production Gaps item 6) — it always produces every daemon in one result/, never a single one.

# Dev loop, per component (debug + ASAN, runs tests) — wraps `nix develop ./src --command ...`
/xce-build lib           # libxce only
/xce-build controller
/xce-build worker
/xce-build scheduler
/xce-build apiserver      # also builds xcecli in the same pass (same meson step)
/xce-build xcecli         # xcecli alone, if you don't need apiserver
/xce-build xceadmin
/xce-build              # all components

# Profiles: /xce-build [component] [profile]
#   asan (default)  -> xce.build/       debug+ASAN, runs tests
#   debug            -> build.dbg/       -O0 -g3, no sanitizer, gdb-friendly
#   release          -> build.release/   -O3, no sanitizer, build-only

# Production packaging (Nix, whole stack only)
nix build .#xcecore          # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation
nix build .#xcecore_debug    # same, debug + ASAN (this flake's default package)
nix build .#xcecli           # xcecli, as its own separate Nix *package* derivation (Go-only, no C toolchain) — `/xce-build xcecli` above gets you the same binary without going through `nix build` at all

nix build .#xcecore is all-or-nothing — there is no per-daemon Nix target. It produces one result/bin/ containing xce_controller, xcecd (+ xcecd_worker, xcecd_virtual), xce_scheduler, xce_apiserver, the xceadmin venv wrapper, and xce (the AppImage dispatcher script, see below). nix build .#xcecli is the one genuinely separate target.

AppImage (xce-x86_64.AppImage, from nix build .#xcecore) wraps src/tools/appimage/entry.sh, a thin per-daemon dispatcher — it runs exactly one daemon per invocation, never the whole stack:

./xce-x86_64.AppImage controller --myname ctrl-1 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990,ctrl-3=h3:9990" ...
./xce-x86_64.AppImage worker -c /etc/xce/worker.yaml
./xce-x86_64.AppImage scheduler --peers node1=h1:9990
./xce-x86_64.AppImage apiserver -d /opt/xce/apiserver
./xce-x86_64.AppImage shell     # interactive bash with all binaries + haproxy/glow/less on PATH

Run it once per daemon (likely on different hosts) to stand up a cluster.

xceadmin — Cluster Configuration

xceadmin is the only supported way to turn operator-authored YAML into the cluster.bin binary the controller loads, and to push it to a running cluster. It speaks the same raw TLS + Cap'n Proto wire protocol controllers use with each other (no client certificate — verify_mode=ssl.CERT_NONE, matching the controller's own outbound posture).

Install

# Dev, editable, fast iteration (must run inside `nix develop ./src`)
cd src/xceadmin && pip install -e .

# Production / CI (what /xce-build and `nix build .#xcecore` actually use)
/xce-build xceadmin

/xce-build xceadmin always does a clean, non-editable pip install into an out-of-tree venv (--system-site-packages, inheriting nix-provided pycapnp/pyyaml/jsonschema), runs the full pytest suite, syncs the bundled schemas fresh from src/schema/, and produces a standalone xceadmin wrapper script — this is what ships in the AppImage and nix build output.

This venv only works from inside nix develop ./src (or the AppImage's bundled env) — it is not a portable, standalone venv. --system-site-packages means pip install sees click/pycapnp/pyyaml/jsonschema already satisfied by nix's site-packages and skips vendoring its own copies — deliberately, since pycapnp is ABI-sensitive to the exact capnproto C library version, and letting pip resolve/build its own would risk a version mismatch that crashes at runtime. If you source .venv/bin/activate (or run the xceadmin binary directly) outside nix develop, none of those packages are on sys.path and you'll hit ModuleNotFoundError: No module named 'click' (or similar) immediately. Either stay inside nix develop ./src when invoking the built xceadmin, or use the AppImage, which bundles a working environment on PATH already.

Usage

# Compile YAML config into a binary, print its SHA-256 to stderr
xceadmin config build /path/to/configs -o cluster.bin
xceadmin config build /path/to/configs -o -            # to stdout instead of a file

# Push a compiled config to a running controller (any node — primary or standby; redirects to primary automatically, up to 3 hops)
xceadmin config push cluster.bin --controller 127.0.0.1:9991     # host:port; defaults to localhost:9991

# Check whether the controller would accept it, without committing
xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991

config build reads a fixed set of source files from the given directory — xce.yaml, hosts.yaml, queues.yaml, resources.yaml, users.yaml, limits.yaml, scaler.yaml (a missing file → empty section, not an error) — validates each against a bundled JSON schema, runs hard-fail business checks (reserved names, duplicates) and warn-only checks (undefined cross-references), then maps everything into a ClusterConfig Cap'n Proto message. Building twice from identical input is byte-for-byte deterministic (no wall-clock/hostname ever read). Fields that are never settable from YAML: port, hash-table-size tuning, xstrings, candidates (peer membership), configGeneration (controller-assigned).

config push exits non-zero unless the controller's reply reports ackedCount > 0 and no error string; --dry-run exits non-zero on any rejection reason the controller returns.

Known limitations (as of this writing)

• config push against a real controller fails with a client-side decode crash — Fixed 2026-07-30. decode_reply() (src/xceadmin/src/xceadmin/discovery.py:46-77) now unwraps the outer Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}} envelope before decoding the inner RequestReply. Both _redirect_target() and push_config() go through decode_reply(); the old bare-RequestReply decode path is gone. decode_reply() validates body.which() == "replymsg" and raises ReplyDecodeError on anything else. Live-verified against a real deployed controller.
• config push --dry-run's "OK" result is not meaningful evidence of anything — Fixed 2026-08-09 (bug-1506/1507). The controller gained a real OPCode_probeConfigTag handler (config_repl.c:handle_probe_config_tag) that returns the current config_version_tag through the standard Request/RequestReply envelope. probe_config() now wraps the probe in a Request envelope and goes through follow_redirects → decode_reply (same as push_config), so accepted=True means the controller actually replied with a valid tag, and --dry-run reports the real generation number from the cluster.
• No config diff, no rollback, no version history command. xceadmin today has exactly two subcommands: config build and config push [--dry-run].
• No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification.

Deploying the Controller

The controller is the only daemon with real config-file/CLI complexity for cluster topology.

xce_controller \
  --myname ctrl-1 \
  --peers "ctrl-1=host1:9990,ctrl-2=host2:9990,ctrl-3=host3:9990" \
  --workdir /var/lib/xce/controller \
  --certdir /var/lib/xce/certs \
  --logfile /var/log/xce/ctrl-1.log \
  --vcpus 4

• Omit --peers for a single-node, non-HA controller.
• --workdir holds per-node conf/ (cluster config) and data/ (WAL + checkpoint) subdirectories.
• --certdir must contain xce.crt/xce.key — this is the only TLS material in the whole stack, and the controller loads it into the single server SSL_CTX used for every inbound connection it accepts: controller↔controller HA traffic and every worker/scheduler/apiserver/xceadmin client join. Generate a self-signed pair for local/dev use:

  openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
    -keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce"
  

  TLS verification is SSL_VERIFY_NONE everywhere in this codebase — no hostname/identity check is enforced regardless of CA, so a CA-signed cert buys you encryption, not authentication.
• Exactly one node (conventionally the first started) needs its conf/ seeded with an initial cluster.bin (via xceadmin config build) before first boot; other nodes receive config over the wire via config-push once joined.
• WAL and snapshot retention in data/ is fully automatic — no manual purging needed. Every checkpoint trims WAL segments below a replication-safe floor (a segment is never deleted while any standby still needs it to catch up) and rotates old snapshots down to snap_keep_n most recent (default 2 — current + one previous, kept as a last-known-good fallback if a checkpoint is later found corrupt). snap_keep_n is tunable via the cluster config's PersistConfig.snapKeepN field, not a CLI flag.

Local dev cluster (fully automated)

/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993

This is the xce-deploy-ha skill wrapping src/tools/deploy-ha.sh, which does everything above automatically: generates certs, builds an initial config from src/configs/*.yaml via xceadmin, and writes scripts/start.sh ctrl-N [extra flags] / scripts/cleanup.sh under the deploy directory. Start nodes with a short stagger for clean election ordering:

scripts/start.sh ctrl-1 -vvv &
sleep 0.5
scripts/start.sh ctrl-2 -vvv &
sleep 0.5
scripts/start.sh ctrl-3 -vvv &

(-vvv is a positional arg to start.sh, not VERBOSE=-vvv — the skill's own docs have this wrong.)

scripts/cleanup.sh wipes logs and every node's data/, plus conf/ for nodes 2..N — ctrl-1's config and all certs survive a cleanup, giving a clean restart without regenerating TLS material or re-authoring config.

Deploying the Worker

xcecd -c /opt/xce/conf/worker.yaml
# or, without a config file:
xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var/log/xce

• Must run as root — check_user() exits immediately otherwise.
• Default config path is /opt/xce/conf/worker.yaml if -c is omitted (and it must exist, or the daemon exits).
• -c, if given, wins over -m/-p (those flags are ignored with a warning).
• Minimal worker.yaml:

  workerconf:
    management_list: ["ctrl-1-host"]
    controller_port: 9990
    log_level: "info"
    log_dir: "/var/log/xce"
  

• --peers "[name=]host:port,..." enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join.
• Worker↔controller traffic is TLS-encrypted (via the shared libxce client, xcecli_ctx_init() → xce_ctx_init()), same posture as everything else in the stack: no client certificate, SSL_VERIFY_NONE.

Deploying the Scheduler

xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.log -v

• No config file at all — pure CLI (argp).
• Gotcha: the default controller target is hard-coded to localhost:9990. There is no -m/-p equivalent — the only way to point the scheduler at any other host or port, even a single non-HA controller, is --peers. Forgetting --peers silently targets localhost:9990 regardless of where your controller actually is.
• -v is repeatable for more verbose logging; default log file is /tmp/xce_scheduler.log.
• TLS-encrypted to the controller, same as the worker (same shared libxce client path).
• Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary through the picker plugin interface (builtin_fcfs is the registered default), not dynamically loaded.

Session recording (live cluster)

Every normal-mode scheduler can record its scheduling sessions to a .replay file for later offline replay/debugging — off by default, no overhead when disabled:

xce_scheduler --peers "ctrl-1=controller-host:9990" --record-dir /var/log/xce/scheduler-recordings -v

• --record-dir DIR sets where recordings are written (default /var/log/xce/). This is the only way record_dir is provisioned — there is no per-request override.
• Recording itself is controlled at runtime, not via a startup flag, through the controller:

  xceadmin scheduler record start [FILENAME] --controller 127.0.0.1:9991   # filename optional, auto-generated if omitted
  xceadmin scheduler record status --controller 127.0.0.1:9991
  xceadmin scheduler record stop --controller 127.0.0.1:9991
  

  These go through a new OPCode_schedAdmin request the controller pass-through-forwards to its attached scheduler (xceadmin → controller → scheduler → reply, same primary-redirect behavior as config push). The controller holds no recording state itself.
• FILENAME (if given) must be a bare filename — no / and no .. segment. The scheduler resolves it under --record-dir and rejects anything that would escape that directory ("invalid record file name").
• A .replay file is a single self-contained Cap'n Proto binary stream (header + snapshot + interleaved events/decisions + periodic checkpoints) — inspect one offline, no running cluster needed:

  xceadmin scheduler dump /var/log/xce/scheduler-recordings/some-session.replay
  

Simulation mode (offline debugging, no controller connection)

--sim runs the scheduler as a standalone replay tool against a previously recorded (or hand-built) .replay file — zero connection to any controller, safe to run against a production recording on a laptop:

xce_scheduler --sim /path/to/session.replay                      # replay the recording
xce_scheduler --sim /path/to/session.replay --sim-verify         # replay + diff against the recorded decisions
xce_scheduler --sim /path/to/session.replay --sim-output /tmp/resim.replay --sim-alt-events /path/to/edited-events.replay   # "what-if": substitute events, record the new decisions
xce_scheduler --sim /path/to/session.replay --interactive        # step through it interactively (REPL)

• The simulator's clock moves only at recorded timestamps, never with the wall clock: it is set to the start snapshot's timestamp, then to each recorded event's timestamp and each session's recorded beginRound, exactly, even when that value is earlier than the current one. Between those points it does not move, so a session's recorded duration in simulation output (--sim-output) is 0, and a replay's decisions do not depend on host speed or load.
• --sim-speed is deprecated and has no effect. It is still accepted with any value so existing command lines keep working, logs one deprecation warning, and will be removed in a future release; drop it from scripts.
• --sim-verify re-derives decisions from the recorded events and asserts they match the decisions the file already recorded — the feature's core "deterministic replay" regression check, usable ad hoc against any recording.
• --sim-alt-events FILE swaps in a different event stream after the initial snapshot for what-if branching; --sim-output FILE records the resulting decisions as a new .replay file.
• --interactive opens a REPL (step, continue, breakpoints, state inspection, snapshot capture) driving the same simulation run a batch --sim would — REPL and batch produce byte-identical decisions for the same input.
• Simulation mode never opens a socket to any controller and never touches --peers/--record-dir — see .kiro/specs/scheduler-core/design.md's Boundary Commitments for the full isolation contract.

Deploying apiserver

apiserver -d /opt/xce/apiserver

-d <top_dir> is mandatory despite the --help text implying a default — omitting it exits immediately. <top_dir>/config/ must already exist and contain:

File	Purpose
app.yaml	HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, and the backend: block (address, port, from, key, optional peers for HA) pointing at the controller.
auth.yaml	Auth provider selection (config_file | pam | ldap | oauth2); for config_file, embeds usernames/passwords/roles directly.
rbac.yaml	Permission catalog + role→permission mappings (domains: self, self_ug, etc.). User→role binding comes from auth.yaml, not this file.
logger.yaml	zap logger level/encoding/rotation.

Both app.yaml and logger.yaml are JSON-Schema-validated at startup (fail-fast on malformed config).

Build gotcha — apiserver needs real CGO flags, not a bare go build

A bare go build in src/frontend/apiserver (no CGO_CFLAGS/CGO_LDFLAGS set) is not a supported way to produce a working apiserver binary. The real controller-connection code (backend/clib.go) is CGO and needs those flags to find libxce's headers. Depending on your environment this either fails to compile, or — if CGO_ENABLED=0 is ambient — silently links backend/stub.go instead, producing a binary where every backend call returns errNoCGO: backend requires CGO. There is no startup warning; the HTTP server comes up fine and only fails at request time.

Use /xce-build apiserver (or nix build .#xcecore) for anything that needs to actually talk to a controller — it computes the correct CGO_CFLAGS/CGO_LDFLAGS from pkg-config and produces xce_apiserver under src/xce.build/frontend/apiserver/ (or the equivalent path for other profiles). /xce-build apiserver also runs the pure-Go unit-test loop (CGO_ENABLED=0 is intentional there); a manual go build/go test invocation is not the canonical path.

Relationship to the controller

apiserver is a separate OS process, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as HostRole_apiserver via a CGO-linked libxce client (the same join mechanism workers/schedulers use), configured entirely by app.yaml's backend: block. If backend.address is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. This connection is TLS-encrypted, same as every other libxce client (no client certificate, SSL_VERIFY_NONE) — do not confuse it with apiserver's own separate HTTPS termination for its REST API, covered next.

TLS

apiserver can self-terminate HTTPS (app.yaml's security.https.enabled + cert_file/key_file, default a checked-in self-signed dev cert). A reverse-proxy pattern (sample config/nginx.https.conf, HAProxy mentioned in steering docs) is the intended production direction but is not wired into any startup automation — you'd have to set it up yourself.

Putting It Together: A Full Local Cluster

# 1. Build everything
/xce-build all

# 2. Bring up a 3-node HA controller cluster
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
# (start ctrl-1, then ctrl-2/ctrl-3 with a 0.5s stagger, per the deploy script)

# 3. Start a worker pointed at the cluster
xcecd -m localhost -p 9991 --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -l /tmp/xce-worker

# 4. Start a scheduler pointed at the cluster
xce_scheduler --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -v

# 5. Start apiserver (needs its own config/ dir prepared first — copy from src/frontend/apiserver/config/,
#    at minimum set backend.address/port to match the controller and use auth.yaml.example, not the checked-in auth.yaml)
apiserver -d /path/to/apiserver-topdir

# 6. Use xcecli against apiserver, or xceadmin against the controller directly
xcecli job submit myjob.json
xceadmin config push cluster.bin --controller localhost:9991

Production Gaps

Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features:

1. No process supervision is shipped. No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit sketch exists in src/docs/config-replication-design.md but has a wrong binary name (xcecontroller vs the real xce_controller) and specifies Type=notify, which the controller does not support (it never calls sd_notify) — using it as written will hang at startup and time out. If you adapt it, switch to Type=simple and fix the binary name.
2. src/frontend/apiserver/config/auth.yaml ships plaintext dev credentials checked into git: four accounts total — admin/admin and op7x2k9/CHANGE_ME_dev_only (both role cluster_admin), plus user1/user1 and user2/user2 (role basic_user). Treat all of them as exposed by default — don't ship the checked-in file as-is; copy auth.yaml.example and set real credentials/an external auth provider instead.
3. All controller-bound traffic is encrypted but not authenticated. Controller↔controller HA replication and every libxce client (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller) run over TLS, but SSL_VERIFY_NONE is used throughout — no client certificates, no hostname/identity verification. This buys confidentiality and tamper-resistance in transit, not authentication of the peer; anyone who can reach the port and complete a TLS handshake is treated as trusted the moment the transport-level join succeeds.
4. apiserver's own TLS uses a checked-in self-signed cert with no CA trust story; xcecli hardcodes InsecureSkipVerify: true against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. (This is apiserver's REST-facing HTTPS, separate from its TLS-encrypted libxce connection to the controller in item 3.)
5. xceadmin config push against a real cluster — the decode-crash bug is fixed (2026-07-30, decode_reply() unwraps the Request envelope), but --dry-run still has no real controller-side handler (see Known Limitations item 2).
6. nix build .#xcecore (the packaged derivation) is all-or-nothing — there's no per-daemon Nix package target. For a single component, use /xce-build <component>, which itself runs inside the Nix dev shell (nix develop ./src --command ...) rather than through nix build.

Quick Reference

Daemon	Binary	Config	Controller-pointing flag	TLS
Controller	xce_controller	none (CLI only)	--peers (omit for single-node)	Yes, all inbound (peers + clients), no client cert
Worker	xcecd	worker.yaml (or CLI)	-m/-p (config file wins) or --peers	Yes, no client cert
Scheduler	xce_scheduler	none (CLI only)	--peers (required even for a single non-default controller)	Yes, no client cert
apiserver	xce_apiserver	config/{app,auth,rbac,logger}.yaml	app.yaml's backend: block	To controller: yes, no client cert. Own REST API: optional, self-terminated
xceadmin	xceadmin (Python)	none — takes YAML source dir	--controller host:port	Yes, no client cert