# Requirements Document

## Introduction

XCE operators need a tool to build and deploy cluster configuration. Currently, YAML config files (`configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml`) exist with JSON schemas for validation, but no tool compiles them into the `ClusterConfig` Cap'n Proto binary the controller expects. The controller-side `OP_upgradeConfig` handler exists (`config_repl.c` — receives binary, verifies SHA-256, pushes to standbys, awaits majority ACK), but no CLI sends it.

`xceadmin` is a Python CLI that fills this gap: `config build` compiles YAML into `cluster.bin`, `config push` sends it to a controller over TLS+capnp.

Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact — deterministic for the same inputs, never committed to git.

## Boundary Context

- **In scope**:
  - YAML → `ClusterConfig` capnp binary compilation (`config build`)
  - TLS+capnp config deployment to controller (`config push`)
  - Auto primary discovery via controller redirect reply
  - JSON Schema validation of YAML inputs
  - Reserved name defense-in-depth (`"lost_and_found"`)
  - `xcecommon` shared Python package (capnp schema, wire protocol, config models)
  - Integration with xce-build and xce-deploy-ha skills
  - Nix dev shell: Python package dependencies
  - `config push --dry-run` — validate without committing (nice-to-have)

- **Out of scope**:
  - Controller-side config replication, snapshot, restart logic (owned by `config-upgrade-snapshot`)
  - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`)
  - Performance tuning / per-instance parameter extraction (separate work item) — specifically `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, and `xstrings` (xstring hash size). These are not cluster-shared: each controller instance is expected to set them locally, so `xceadmin` never reads, accepts, or writes them (see Requirement 1.8).
  - `candidates` (peer/standby list) — changes independently and more frequently than the rest of cluster config (adding/removing a standby shouldn't require a `config build`/`config push` cycle); `xceadmin` never reads, accepts, or writes it (see Requirement 1.10). Peer membership is owned elsewhere (mechanism TBD — see brief.md Open Design Question #3).
  - `configGeneration` management — fully controller-owned (see Adjacent Expectations below); `xceadmin` never reads, increments, or overrides it.
  - xcecli Python rewrite (future, separate spec)
  - Controller config version history, rollback safety validation, config activation status (owned by `config-upgrade-snapshot`, see Future Roadmap in brief.md)
  - `config diff` — deferred to roadmap follow-up (2026-07-21): no operator has asked for it yet; revisit once `config push` has real-world usage (see brief.md R1)

- **Adjacent expectations**:
  - **Controller `OP_upgradeConfig` handler** (`config_repl.c`) exists and is operational. On receiving `UpgradeConfigRequest{sha256, bytes}`:
    1. SHA-256 transit verification
    2. Cap'n Proto decode + reserved name check (`"lost_and_found"`)
    3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary — the primary is the sole authority for `configGeneration`; whatever value `xceadmin` ships in `cluster.bin` is discarded and replaced
    4. Save to disk, replicate to standbys, await majority ACK
    5. Commit `WAL_CONFIG_MARKER`, signal controller restart
    6. Return `UpgradeConfigReply{versionTag, ackedCount, error}`
  - **Controller redirect**: Standby returns `redirectHost`/`redirectPort` in the `RequestReply` union — xceadmin follows the redirect to reach the primary.
  - **Per-instance field preservation** — RESOLVED structurally, not by controller behavior: `port`, `candidates`, the hash-table-size fields, and `xstrings` have been removed from the `ClusterConfig` capnp struct entirely (`clusterconfig-schema-split` spec, landed commit `519e11c`). `ClusterConfig` is now 13 fields (@0–@12, `controller.capnp:694-708`); these fields simply do not exist in `cluster.bin` — there is no zero-default-clobber risk to guard against. Req 1.8/1.9/1.10 remain as defense-in-depth documentation (if any of these fields is ever reintroduced, xceadmin must still not source it), not as active field-suppression logic.
  - **ProbeConfigTag RPC** (`xce_probe_config_tag()`, `config_repl.c:1829`): Separate direct-capnp RPC (not in `Request` union) for reading current config tag. Used by `config push --dry-run` to validate before committing.
  - `controller.capnp` schema is the single source of truth for `ClusterConfig`, `Request`/`Reply` unions, and all OP codes.
  - `config-upgrade-snapshot` spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status.
  - **Blocked controller-side gaps** (NOT owned by admin-config-tool):
    - Entity-bearing snapshot after config upgrade (blocked on `entity-bearing-snapshots` spec)
    - Config rollback safety (e.g., reject rollback while jobs reference deleted queues)
    - Config version history query (`xceadmin config history`)
  - **Dependency on `clusterconfig-schema-split`** — **RESOLVED, landed** (commit `519e11c`, phase=`implementation-complete`). Both subcommands are now unblocked:
    - `config build`: no dependency, unaffected either way.
    - `config push`: previously soft-blocked over the risk of `xceadmin` silently wiping `candidates`/`port`/perf-tuning on a zero-valued push. That risk no longer exists — those fields are physically absent from the `ClusterConfig` schema, so there is nothing for a push to clobber. `config push` may be exercised against a real cluster without this precondition.

## Requirements

### Requirement 1: Config Build

**Objective:** As an XCE operator, I want to compile YAML config files into a `ClusterConfig` capnp binary, so that I can produce a deployable cluster configuration from human-editable source files.

#### Acceptance Criteria

1. When the operator runs `xceadmin config build <configs_dir>`, the system shall scan the directory for YAML config files (`hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`, `limits.yaml`, `xce.yaml`, `scaler.yaml`), validate each against its corresponding JSON schema bundled with `xceadmin`, map the validated data to the `ClusterConfig` capnp struct, serialize it to binary, and write the result to `cluster.bin` in the current working directory.
2. When the operator specifies `-o <path>`, the system shall write the output to the given path instead of the default `cluster.bin`.
3. When the operator specifies `-o -`, the system shall write the binary output to stdout.
4. After writing output, the system shall print the SHA-256 hash of the resulting binary to stderr.
5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors).
6. **RESOLVED, superseded (2026-07-21)**: JSON Schema validation is performed against schemas bundled with the `xceadmin` package (see Requirement 5.5), not a `schema/` subdirectory the operator supplies. `configs_dir` therefore never needs its own `schema/` folder, and there is no "missing schema" error mode — the schemas travel with the tool, version-locked to the same release as the `ClusterConfig` field mapping they validate against (`xcecommon.mapping`), the same rationale already applied to `controller.capnp` (see Requirement 5.5, brief.md). An operator-supplied `schema/` subdirectory, if present, is ignored — not read, not merged, not treated as an override.
7. Where the YAML content passes JSON Schema validation but maps to fields not present in the `ClusterConfig` capnp struct, the system shall warn the operator about unrecognized fields.
8. The system shall not source `port`, the `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash-table-size fields, or `xstrings` (xstring hash size) from YAML or CLI input under any circumstance. These are per-controller-instance performance-tuning parameters, not cluster-shared configuration, and the built `cluster.bin` shall leave them unset (capnp default).
9. The system shall not source, read, or write `configGeneration`. The built `cluster.bin` shall leave it unset (capnp default) — the primary controller is the sole authority for assigning it (see Requirement 2 note).
10. The system shall not source `candidates` (controller peer/standby list) from YAML or CLI input. Standby membership changes independently of, and more frequently than, the rest of cluster configuration, so it shall not require a `config build`/`config push` cycle; the built `cluster.bin` shall leave `candidates` unset (capnp default). Peer membership sourcing mechanism is out of scope for this spec (open question — see brief.md Open Design Question #3).
11. `clusterName`, `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness — not the HA/Raft election heartbeat, which has no `ClusterConfig` field, see `.kiro/steering/config-design.md` Rule 0), `fetchJobInterval` (scheduler job-fetch pass interval), and `netioTimeout` (network I/O timeout) SHALL be sourced from a new `xce.yaml` file and included in `cluster.bin` — these must stay consistent across all controller instances (unlike Criteria 8–10, which are intentionally excluded). **RESOLVED**: source is `xce.yaml`, a flat (non-list) YAML file holding these 5 scalar `ClusterConfig` fields. `cluster_name` is required (no sensible default). `heartbeat_interval` and `fetch_job_interval` are optional — both have controller-side defaults (`fill_default_cluster_config()`, `myconf.c:72-76`: 5 and 10 respectively). `inactive_heartbeat_count` is required and has **no** controller-side default — `cluster.c:1622` computes the worker-dead timeout as `heartbeat_interval * inactive_heartbeat_count`, so an unset (0) value would mark workers dead immediately; xceadmin must not let this ship unset. `netio_timeout` is optional; note it currently has no reader anywhere in the codebase outside the struct declaration itself (`clusterconfig.h:12`) — the field is wired on the wire/schema but not yet consumed by any controller logic. That's a controller-side gap, out of scope for this spec, but xceadmin should still source it from YAML if present rather than silently drop it.
12. `hproviders` (`List(ScalerConfig)` — autoscaler provider definitions: `name`, `type`, `conf_path`, `script_path`, `script_option`) SHALL be sourced from a new `scaler.yaml` file and included in `cluster.bin`. Scaler definitions must be identical across controller instances (a queue configured against a scaler must resolve it the same way after failover) and change rarely — both Configuration Inclusion Rules fail, so this field is included per the Default rule.

Criteria 8–12 apply the **Configuration Inclusion Rules** (brief.md): a field is excluded from `cluster.bin` if it can change independently of the rest of cluster config, OR if it's expected to change frequently; otherwise it's included. Any future `ClusterConfig` field proposed for xceadmin's YAML mapping must be classified against these two rules first.

### Requirement 2: Config Push

**Objective:** As an XCE operator, I want to deploy a compiled `cluster.bin` to a controller cluster, so that the configuration takes effect on the running system.

#### Acceptance Criteria

1. When the operator runs `xceadmin config push <cluster.bin>`, the system shall connect to the controller via TLS, send the binary as a capnp `Request` with `OP_upgradeConfig` opcode, `upgradeconfigrequestmsg` body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result.
2. When the operator specifies `--controller <host:port>`, the system shall connect to the given controller address instead of the default `localhost:9991`.
3. ~~The system shall use the same TLS certificate bundle the controller cluster is deployed with (`xce.crt`/`xce.key`, default path `ha/data/`) to establish trust — no separate client CA flag or cert-issuance step.~~ **Superseded (2026-07-22)**: verified against the real client TLS setup (`netio.c`'s `create_ssl_context()`) that the actual controller-to-controller/CLI client presents **no** certificate at all when acting as a client (`cert_file`/`key_file` both `NULL` → bare `TLS_client_method()`) — only the server side loads a cert/key pair. `xceadmin`'s client previously diverged from this (loaded a client cert too); fixed to match exactly. No CA flag or cert-issuance step either way — `verify_mode=CERT_NONE` on both the real client and `xceadmin`.
4. ~~When the operator specifies `--cert-dir <path>`, the system shall load `xce.crt`/`xce.key` from the given directory instead of the default `ha/data/`.~~ **Superseded (2026-07-22)**: removed entirely — with no client cert to load, `--cert-dir` had no remaining purpose. `ha/data/` itself was also removed (deploy-ha.sh and the HA test harness now generate a fresh server-side cert/key per run via `openssl` instead of reading a checked-in fixture).
5. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message.
6. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0.
7. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero.

### Requirement 3: Auto Primary Discovery

**Objective:** As an XCE operator, I want xceadmin to automatically find the primary controller, so that I don't need to know which node is currently primary.

#### Acceptance Criteria

1. When the operator connects to any controller node (primary or standby), the system shall inspect the reply for `redirectHost` and `redirectPort` fields. If both are non-empty, the system shall reconnect to the redirect target and resend the request.
2. When the redirect target itself redirects, the system shall follow at most 3 redirect hops before reporting an error.
3. When a standby controller redirects the operator to the primary, the system shall display the redirect target to inform the operator where the config was actually pushed.
4. When connecting to a controller and no primary has been elected yet, the system shall time out after 30 seconds and report "no primary available" to the operator.

### Requirement 4: Validation

**Objective:** As an XCE operator, I want validation errors caught early, so that invalid configuration never reaches a running cluster.

#### Acceptance Criteria

1. When a YAML file fails JSON Schema validation, the system shall report the specific validation error with file name, line/field path, and the constraint violated, then exit non-zero.
2. When a YAML file uses the reserved name `"lost_and_found"` for a queue or host, the system shall reject it with a clear error message explaining the reservation, even if it passes schema validation.
3. When a YAML file contains duplicate entries (duplicate queue names, host names, resource names, group names, limit names — each checked within its own file/section; a limit name is not required to differ from a queue, host, or group name), the system shall report the duplicates and exit non-zero.
4. When a YAML file references an undefined entity (e.g., a queue references a host group that is not defined, or a limit references a user/queue/project/host group or a `resname` that is not defined in `resources.yaml`), the system shall warn the operator but allow the build to proceed (the controller performs final referential integrity checks at runtime).
5. `resources.yaml` is the exhaustive resource catalog: every `resname` referenced elsewhere (e.g., `limits.yaml`, `queues.yaml` `sched_stop`/`res_req`/`gpu_req`, `hosts.yaml` `sched_stop`) — including built-in/dynamic resources such as `slots`, `mem`, and the 8 per-host attributes the worker registers automatically (`type`, `model`, `cpuf`, `maxcpus`, `maxmem`, `maxswap`, `maxtmp`, `ndisks` — see `add_hostattrs_resvals()`, `controller/worker.c`) — shall be declared as an entry in `resources.yaml`. The system shall not treat any resource name as an implicit built-in exempt from this catalog; Criterion 4's undefined-entity warning applies equally to built-in resource names.

### Requirement 5: xcecommon Shared Package

**Objective:** As a future Python tool author, I want a shared `xcecommon` package providing capnp schema loading, wire protocol, and config model types, so that I don't duplicate protocol code across tools.

#### Acceptance Criteria

1. The `xcecommon` package shall provide a capnp schema loader that reads `controller.capnp` and exposes the compiled Python types (`ClusterConfig`, `Request`, `Reply`, `UpgradeConfigRequest`, `UpgradeConfigReply`, and all nested types).
2. The `xcecommon` package shall provide a wire protocol module that handles TLS connection, capnp message framing (8-byte uint64 size prefix in host byte order), send, and receive.
3. The `xcecommon` package shall provide YAML→capnp mapping functions that convert validated YAML data to `ClusterConfig` capnp struct fields.
4. Where the `xcecommon` package is installed for development, it shall be importable under the name `xcecommon` from any tool within the project.
5. **(2026-07-21)** The `xcecommon` package shall bundle its own copy of `controller.capnp`/`c.capnp` and the 7 JSON Schema files (one per `config build` input file), synced from `src/schema/` and `src/configs/schema/` respectively at build time (see Requirement 8.1/8.2) and packaged so they are available regardless of install mode (`pip install -e .` or `pip install .`). Neither schema is read from a path outside the installed package at runtime; an operator-supplied `configs_dir/schema/` directory, if present, is not consulted (superseded Requirement 1.6).

### Requirement 6: Dry-Run Push (Nice-to-Have)

**Objective:** As an XCE operator, I want to validate that the controller will accept a config before committing to the push.

#### Acceptance Criteria

1. When the operator runs `xceadmin config push --dry-run <cluster.bin>`, the system shall connect to the controller, probe the generation, validate SHA and version compatibility, and report whether the config would be accepted, without actually committing the config to the cluster.
2. When dry-run passes, the system shall report "OK: controller will accept this config (generation N→M)" and exit 0.
3. When dry-run fails, the system shall report the specific rejection reason and exit non-zero.

### Requirement 7: Error Handling & Resilience

**Objective:** As an XCE operator, I want clear, actionable error messages for all failure modes, so that I can diagnose and fix configuration issues without reading controller logs.

#### Acceptance Criteria

1. If the TLS connection to the controller fails, the system shall report "cannot connect to <host:port>: <reason>" and exit non-zero.
2. If the controller returns a `Reply` with non-zero `status`, the system shall report the status code and any error text from the reply, then exit non-zero.
3. If the capnp binary is corrupted or cannot be decoded, the system shall report "invalid cluster.bin: <reason>" and exit non-zero.
4. If the `controller.capnp` schema file is not found at the expected path, the system shall report the missing path and suggest checking the installation.
5. If the operator interrupts the push (SIGINT), the system shall clean up the TLS connection and exit gracefully with a "cancelled" message.
6. The system shall time out after 30 seconds on any network operation and report "timeout connecting to <host:port>".

### Requirement 8: Skill & Harness Integration

**Objective:** As a developer, I want xceadmin integrated into the existing development toolchain, so that building, deploying, and testing XCE clusters uses consistent tooling.

#### Acceptance Criteria

1. ~~When the developer runs `/xce-build xceadmin`, the system shall make the `xceadmin` CLI available on PATH and run the full test suite, reporting pass/fail.~~ **Superseded (2026-07-22)**: `xceadmin` is now meson-built (`src/xceadmin/meson.build`) — `xce-build.sh xceadmin` delegates to `meson compile -C "$BUILD_DIR" xceadmin`, and `nix build .#xcecore[_debug]` installs a standalone `result/bin/xceadmin` (no venv at runtime). See design.md's File Structure Plan Amendment.
2. ~~When the developer runs `/xce-build xceadmin release`, the system shall install `xceadmin` for production use (pinned, non-editable) and then run the full test suite against that non-editable install...~~ **Superseded (2026-07-22)**: the editable-vs-pinned distinction collapsed — every meson-driven build (`xce-build.sh xceadmin`, `meson compile`, `nix build`) now runs the pinned/non-editable install path unconditionally, a strictly stronger guarantee (exercises real `package_data` resolution every run, not just under a `release` profile).
3. ~~When the developer runs `/xce-deploy-ha` with the `--build-config` flag, the system shall invoke `xceadmin config build configs/ -o ha/data/cluster.bin` as a precondition before deploying the cluster.~~ **Superseded (2026-07-22)**: `--build-config` removed entirely — every `deploy-ha.sh` run now unconditionally builds `cluster.bin` fresh from `src/configs/*.yaml` directly into the deploy dir's own `controller/ctrl-1/conf/` (no `ha/data/` staging step, no flag needed since nothing is ever preserved across runs to opt out of rebuilding).
4. ~~When `cluster.bin` is missing from `ha/data/` during deploy, `/xce-deploy-ha` shall run `xceadmin config build` automatically (same as `--build-config`).~~ **Superseded (2026-07-22)**: `ha/data/` removed entirely (directory no longer exists). See 8.3 above — always-fresh generation directly into the deploy dir replaces the missing-file check.
5. ~~When `cluster.bin` already exists and `--build-config` is not passed, `/xce-deploy-ha` shall use the existing binary as-is (backward compatible).~~ **Superseded (2026-07-22)**: no longer applicable — nothing persists across deploys for this to preserve; every deploy is fully fresh (cert included, generated via `openssl`, self-signed, 10y validity — TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase, so a fresh pair per deploy loses no real security property versus a long-lived shared one).
6. The nix dev shell (`nix develop ./src`) shall include all Python dependencies required by xceadmin after the nix environment is updated (see Nix Environment section in brief.md).

### Requirement 9: Deterministic Builds

**Objective:** As an XCE operator, I want config builds to be deterministic, so that I can reproduce any historical config binary from its YAML source.

#### Acceptance Criteria

1. When `xceadmin config build` is run twice with the same YAML inputs and same xceadmin version, the system shall produce byte-identical `cluster.bin` output.
2. The system shall not embed build timestamps, hostnames, usernames, or any other non-deterministic data in the output binary.
3. When different operators on different machines build the same YAML with the same xceadmin version, the output shall be byte-identical.

## Requirements Summary

| # | Requirement | Priority | Phase |
|---|------------|----------|-------|
| 1 | Config Build | **Core** | Implementation |
| 2 | Config Push | **Core** | Implementation |
| 3 | Auto Primary Discovery | **Core** | Implementation |
| 4 | Validation | **Core** | Implementation |
| 5 | xcecommon Package | **Core** | Implementation |
| 6 | Dry-Run Push | Nice-to-Have | Follow-up |
| 7 | Error Handling | **Core** | Implementation |
| 8 | Skill & Harness | **Core** | Implementation |
| 9 | Deterministic Builds | **Core** | Implementation |