diff --git a/.kiro/specs/admin-config-tool/spec.json b/.kiro/specs/admin-config-tool/spec.json index 46f6683..8391c70 100644 --- a/.kiro/specs/admin-config-tool/spec.json +++ b/.kiro/specs/admin-config-tool/spec.json @@ -1,9 +1,9 @@ { "feature_name": "admin-config-tool", "created_at": "2026-07-06T01:30:00Z", - "updated_at": "2026-07-22T14:50:00Z", + "updated_at": "2026-07-30T02:01:00Z", "language": "en", - "phase": "implementation-complete", + "phase": "tasks-generated", "approvals": { "requirements": { "generated": true, diff --git a/src/docs/deployment.md b/src/docs/deployment.md new file mode 100644 index 0000000..37bccba --- /dev/null +++ b/src/docs/deployment.md @@ -0,0 +1,252 @@ +# XCE Deployment Guide + +Covers building and running all four daemons (controller, worker, scheduler, apiserver) and using `xceadmin` to author and push cluster configuration. Local dev/test deployment is fully automated; production process supervision (systemd/Docker/etc.) is **not** shipped in this repo — see [Production Gaps](#production-gaps) before treating anything here as a production runbook. + +## Component Overview + +``` +xceadmin (Python CLI) --config push (TLS)--> controller (C, HA cluster, 1..N nodes) + ^ ^ + plaintext join | | plaintext join + | | + worker (C) scheduler (C) + ^ + CGO-linked libxce client (TCP, HA-aware) + | + apiserver (Go, REST) <-- xcecli (Go CLI, REST) +``` + +- **controller** (`xce_controller`) — the only stateful daemon. Runs as 1 or 3+ nodes for HA (Raft-style election + replication). All other components are its clients. +- **worker** (`xcecd`) — one per compute host, runs jobs, joins a controller. +- **scheduler** (`xce_scheduler`) — job-to-node placement engine, joins a controller. +- **apiserver** (`xce_apiserver`) — Go REST API, joins the controller cluster as a client role (`HostRole_apiserver`) via a CGO-linked `libxce` client, not a plain HTTP backend call. +- **xcecli** — Go CLI, talks to `apiserver` over REST, never touches the controller directly. +- **xceadmin** — Python CLI, the only tool that compiles and pushes *cluster configuration* (`cluster.bin`) to the controller. Separate concern from the four daemons above. + +Only controller↔controller HA replication traffic is TLS. Worker→controller, scheduler→controller, and apiserver→controller traffic is **plaintext**. + +## Building + +```bash +# Dev loop, per component (debug + ASAN, runs tests) +/xce-build controller +/xce-build worker +/xce-build scheduler +/xce-build apiserver +/xce-build xceadmin +/xce-build # all components + +# Profiles: /xce-build [component] [profile] +# asan (default) -> xce.build/ debug+ASAN, runs tests +# debug -> build.dbg/ -O0 -g3, no sanitizer, gdb-friendly +# release -> build.release/ -O3, no sanitizer, build-only + +# Production (Nix) +nix build .#xcecore # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation +nix build .#xcecore_debug # same, debug + ASAN (this flake's default package) +nix build .#xcecli # xcecli only (separate Go-only derivation) +``` + +`nix build .#xcecore` is all-or-nothing — there is no per-daemon Nix target. It produces one `result/bin/` containing `xce_controller`, `xcecd` (+ `xcecd_worker`, `xcecd_virtual`), `xce_scheduler`, `xce_apiserver`, the `xceadmin` venv wrapper, and `xce` (the AppImage dispatcher script, see below). `nix build .#xcecli` is the one genuinely separate target. + +**AppImage** (`xce-x86_64.AppImage`, from `nix build .#xcecore`) wraps `src/tools/appimage/entry.sh`, a thin per-daemon dispatcher — it runs exactly **one** daemon per invocation, never the whole stack: +```bash +./xce-x86_64.AppImage controller --myname ctrl-1 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990,ctrl-3=h3:9990" ... +./xce-x86_64.AppImage worker -c /etc/xce/worker.yaml +./xce-x86_64.AppImage scheduler --peers node1=h1:9990 +./xce-x86_64.AppImage apiserver -d /opt/xce/apiserver +./xce-x86_64.AppImage shell # interactive bash with all binaries + haproxy/glow/less on PATH +``` +Run it once per daemon (likely on different hosts) to stand up a cluster. + +## xceadmin — Cluster Configuration + +`xceadmin` is the only supported way to turn operator-authored YAML into the `cluster.bin` binary the controller loads, and to push it to a running cluster. It speaks the same raw TLS + Cap'n Proto wire protocol controllers use with each other (no client certificate — `verify_mode=ssl.CERT_NONE`, matching the controller's own outbound posture). + +### Install + +```bash +# Dev, editable, fast iteration (must run inside `nix develop ./src`) +cd src/xceadmin && pip install -e . + +# Production / CI (what /xce-build and `nix build .#xcecore` actually use) +/xce-build xceadmin +``` +`/xce-build xceadmin` always does a clean, non-editable `pip install` into an out-of-tree venv (`--system-site-packages`, inheriting nix-provided `pycapnp`/`pyyaml`/`jsonschema`), runs the full pytest suite, syncs the bundled schemas fresh from `src/schema/`, and produces a standalone `xceadmin` wrapper script — this is what ships in the AppImage and `nix build` output. + +### Usage + +```bash +# Compile YAML config into a binary, print its SHA-256 to stderr +xceadmin config build /path/to/configs -o cluster.bin +xceadmin config build /path/to/configs -o - # to stdout instead of a file + +# Push a compiled config to a running controller (any node — primary or standby; redirects to primary automatically, up to 3 hops) +xceadmin config push cluster.bin --controller 127.0.0.1:9991 # host:port; defaults to localhost:9991 + +# Check whether the controller would accept it, without committing +xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991 +``` + +`config build` reads a fixed set of source files from the given directory — `xce.yaml`, `hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`, `limits.yaml`, `scaler.yaml` (a missing file → empty section, not an error) — validates each against a bundled JSON schema, runs hard-fail business checks (reserved names, duplicates) and warn-only checks (undefined cross-references), then maps everything into a `ClusterConfig` Cap'n Proto message. Building twice from identical input is byte-for-byte deterministic (no wall-clock/hostname ever read). Fields that are **never** settable from YAML: `port`, hash-table-size tuning, `xstrings`, `candidates` (peer membership), `configGeneration` (controller-assigned). + +`config push` exits non-zero unless the controller's reply reports `ackedCount > 0` and no error string; `--dry-run` exits non-zero on any rejection reason the controller returns. + +### Known limitations (as of this writing) + +- **`config push` against a real controller fails with a client-side decode crash**, live-verified against a freshly deployed single-node controller: the controller's `OPCode_upgradeConfig` dispatch is fine — `CONF_ENABLE_UPGRADE_CONFIG` defaults on, the earlier dispatch-gate polarity bug (bug-1468) was already fixed 2026-07-25, and the controller log shows the push genuinely committed, activated, and hash-verified server-side. The bug is in `xceadmin` itself: `xce_reply()` (`src/lib/request.c`) wraps every reply — including the upgradeConfig reply — in an outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope, but `discovery.py`'s `_redirect_target()` decodes the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope. The resulting pointer-offset misalignment throws `capnp.lib.capnp.KjException: ... non-list pointer where text was expected` from `push_config()`. This affects every opcode reply routed through `follow_redirects`, not just upgradeConfig. Fix belongs in `xceadmin`, not the controller. +- **`config push --dry-run`'s "OK" result is not meaningful evidence of anything**: `probe_config()`'s docstring itself notes `OPCode_probeConfigTag` has no dispatch case in `cluster.c`, so the "accepted" response is a client-side inference, not a real wire signal from the controller. Live-verified: dry-run returns exit 0 with no corresponding entry in the controller log at all. +- No `config diff`, no rollback, no version history command. `xceadmin` today has exactly two subcommands: `config build` and `config push [--dry-run]`. +- No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification. + +## Deploying the Controller + +The controller is the only daemon with real config-file/CLI complexity for cluster topology. + +```bash +xce_controller \ + --myname ctrl-1 \ + --peers "ctrl-1=host1:9990,ctrl-2=host2:9990,ctrl-3=host3:9990" \ + --workdir /var/lib/xce/controller \ + --certdir /var/lib/xce/certs \ + --logfile /var/log/xce/ctrl-1.log \ + --vcpus 4 +``` + +- Omit `--peers` for a single-node, non-HA controller. +- `--workdir` holds per-node `conf/` (cluster config) and `data/` (WAL + checkpoint) subdirectories. +- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, used exclusively for controller↔controller HA traffic. Generate a self-signed pair for local/dev use: + ```bash + openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \ + -keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce" + ``` + TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase — no hostname/identity check is enforced regardless of CA, so a CA-signed cert buys you encryption, not authentication. +- Exactly one node (conventionally the first started) needs its `conf/` seeded with an initial `cluster.bin` (via `xceadmin config build`) before first boot; other nodes receive config over the wire via config-push once joined. + +### Local dev cluster (fully automated) + +```bash +/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993 +``` +This is the `xce-deploy-ha` skill wrapping `src/tools/deploy-ha.sh`, which does everything above automatically: generates certs, builds an initial config from `src/configs/*.yaml` via `xceadmin`, and writes `scripts/start.sh ctrl-N [extra flags]` / `scripts/cleanup.sh` under the deploy directory. Start nodes with a short stagger for clean election ordering: +```bash +scripts/start.sh ctrl-1 -vvv & +sleep 0.5 +scripts/start.sh ctrl-2 -vvv & +sleep 0.5 +scripts/start.sh ctrl-3 -vvv & +``` +(`-vvv` is a **positional** arg to `start.sh`, not `VERBOSE=-vvv` — the skill's own docs have this wrong.) + +`scripts/cleanup.sh` wipes logs and every node's `data/`, plus `conf/` for nodes 2..N — ctrl-1's config and all certs survive a cleanup, giving a clean restart without regenerating TLS material or re-authoring config. + +## Deploying the Worker + +```bash +xcecd -c /opt/xce/conf/worker.yaml +# or, without a config file: +xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var/log/xce +``` + +- **Must run as root** — `check_user()` exits immediately otherwise. +- Default config path is `/opt/xce/conf/worker.yaml` if `-c` is omitted (and it must exist, or the daemon exits). +- `-c`, if given, **wins over** `-m`/`-p` (those flags are ignored with a warning). +- Minimal `worker.yaml`: + ```yaml + workerconf: + management_list: ["ctrl-1-host"] + controller_port: 9990 + log_level: "info" + log_dir: "/var/log/xce" + ``` +- `--peers "[name=]host:port,..."` enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join. +- No TLS on this connection — worker↔controller traffic is plaintext. + +## Deploying the Scheduler + +```bash +xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.log -v +``` + +- No config file at all — pure CLI (`argp`). +- **Gotcha**: the default controller target is hard-coded to `localhost:9990`. There is no `-m`/`-p` equivalent — the *only* way to point the scheduler at any other host or port, even a single non-HA controller, is `--peers`. Forgetting `--peers` silently targets `localhost:9990` regardless of where your controller actually is. +- `-v` is repeatable for more verbose logging; default log file is `/tmp/xce_scheduler.log`. +- No TLS — plaintext to the controller, same as the worker. +- Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary, not dynamically loaded plugins. + +## Deploying apiserver + +```bash +apiserver -d /opt/xce/apiserver +``` + +`-d ` is **mandatory** despite the `--help` text implying a default — omitting it exits immediately. `/config/` must already exist and contain: + +| File | Purpose | +|---|---| +| `app.yaml` | HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, and the `backend:` block (`address`, `port`, `from`, `key`, optional `peers` for HA) pointing at the controller. | +| `auth.yaml` | Auth provider selection (`config_file` \| `pam` \| `ldap` \| `oauth2`); for `config_file`, embeds usernames/passwords/roles directly. | +| `rbac.yaml` | Permission catalog + role→permission mappings (domains: `self`, `self_ug`, etc.). User→role binding comes from `auth.yaml`, not this file. | +| `logger.yaml` | zap logger level/encoding/rotation. | + +Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fast on malformed config). + +### Build gotcha — use the Meson/Nix build, not `make build-server` + +`make build-server` (`src/frontend/Makefile`) runs a bare `go build` with **no `CGO_CFLAGS`/`CGO_LDFLAGS` set**. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time. + +**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). Reserve `make build-server`/`make test` for the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there). + +### Relationship to the controller + +apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. No TLS on this connection. + +### TLS + +apiserver can self-terminate HTTPS (`app.yaml`'s `security.https.enabled` + `cert_file`/`key_file`, default a checked-in self-signed dev cert). A reverse-proxy pattern (sample `config/nginx.https.conf`, HAProxy mentioned in steering docs) is the intended production direction but is **not wired into any startup automation** — you'd have to set it up yourself. + +## Putting It Together: A Full Local Cluster + +```bash +# 1. Build everything +/xce-build all + +# 2. Bring up a 3-node HA controller cluster +/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993 +# (start ctrl-1, then ctrl-2/ctrl-3 with a 0.5s stagger, per the deploy script) + +# 3. Start a worker pointed at the cluster +xcecd -m localhost -p 9991 --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -l /tmp/xce-worker + +# 4. Start a scheduler pointed at the cluster +xce_scheduler --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -v + +# 5. Start apiserver (needs its own config/ dir prepared first — copy from src/frontend/apiserver/config/, +# at minimum set backend.address/port to match the controller and use auth.yaml.example, not the checked-in auth.yaml) +apiserver -d /path/to/apiserver-topdir + +# 6. Use xcecli against apiserver, or xceadmin against the controller directly +xcecli job submit myjob.json +xceadmin config push cluster.bin --controller localhost:9991 +``` + +## Production Gaps + +Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features: + +1. **No process supervision is shipped.** No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit *sketch* exists in `src/docs/config-replication-design.md` but has a wrong binary name (`xcecontroller` vs the real `xce_controller`) and specifies `Type=notify`, which the controller does not support (it never calls `sd_notify`) — using it as written will hang at startup and time out. If you adapt it, switch to `Type=simple` and fix the binary name. +2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**, including an `admin`/`admin` account and a named-user `cluster_admin` account. Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead. +3. **Only controller↔controller traffic is encrypted.** Worker→controller, scheduler→controller, and apiserver→controller are all plaintext, with no path to enable TLS on them today. +4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. +5. **`xceadmin config push` doesn't work against a real cluster yet** (see xceadmin's Known Limitations above) — the controller side is fine; `xceadmin`'s `discovery.py` fails to unwrap the `Request` envelope around every reply, crashing on decode. +6. **`nix build .#xcecore` is all-or-nothing** — there's no way to build just one daemon via Nix; use `/xce-build ` for per-component builds. + +## Quick Reference + +| Daemon | Binary | Config | Controller-pointing flag | TLS | +|---|---|---|---|---| +| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, controller↔controller only | +| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | No | +| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | No | +| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | Optional, self-terminated | +| xceadmin | `xceadmin` (Python) | none — takes YAML source dir | `--controller host:port` | Yes, no client cert | diff --git a/src/frontend/apiserver/config/auth.yaml b/src/frontend/apiserver/config/auth.yaml index ca603c7..e8c5a42 100644 --- a/src/frontend/apiserver/config/auth.yaml +++ b/src/frontend/apiserver/config/auth.yaml @@ -11,8 +11,8 @@ config_file: - username: admin password: admin roles: [cluster_admin] - - username: rshen - password: Letmein + - username: op7x2k9 + password: CHANGE_ME_dev_only roles: [cluster_admin] - username: user1 password: user1