myxce/.kiro/specs/admin-config-tool/requirements.md

Requirements Document

Introduction

XCE operators need a tool to build and deploy cluster configuration. Currently, YAML config files (configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml) exist with JSON schemas for validation, but no tool compiles them into the ClusterConfig Cap'n Proto binary the controller expects. The controller-side OP_upgradeConfig handler exists (config_repl.c — receives binary, verifies SHA-256, pushes to standbys, awaits majority ACK), but no CLI sends it.

xceadmin is a Python CLI that fills this gap: config build compiles YAML into cluster.bin, config push sends it to a controller over TLS+capnp.

Config source of truth is git-managed YAML. cluster.bin is a derived artifact — deterministic for the same inputs, never committed to git.

Boundary Context

• In scope:

  • YAML → ClusterConfig capnp binary compilation (config build)
  • TLS+capnp config deployment to controller (config push)
  • Auto primary discovery via controller redirect reply
  • JSON Schema validation of YAML inputs
  • Reserved name defense-in-depth ("lost_and_found")
  • xcecommon shared Python package (capnp schema, wire protocol, config models)
  • Integration with xce-build and xce-deploy-ha skills
  • Nix dev shell: Python package dependencies
  • config push --dry-run — validate without committing (nice-to-have)

• Out of scope:

  • Controller-side config replication, snapshot, restart logic (owned by config-upgrade-snapshot)
  • Entity-bearing snapshot payload (owned by entity-bearing-snapshots)
  • Performance tuning / per-instance parameter extraction (separate work item) — specifically port, the jobmgr/workermgr/qmgr/resmgr hash-table-size fields, and xstrings (xstring hash size). These are not cluster-shared: each controller instance is expected to set them locally, so xceadmin never reads, accepts, or writes them (see Requirement 1.8).
  • candidates (peer/standby list) — changes independently and more frequently than the rest of cluster config (adding/removing a standby shouldn't require a config build/config push cycle); xceadmin never reads, accepts, or writes it (see Requirement 1.10). Peer membership is owned elsewhere (mechanism TBD — see brief.md Open Design Question #3).
  • configGeneration management — fully controller-owned (see Adjacent Expectations below); xceadmin never reads, increments, or overrides it.
  • xcecli Python rewrite (future, separate spec)
  • Controller config version history, rollback safety validation, config activation status (owned by config-upgrade-snapshot, see Future Roadmap in brief.md)
  • config diff — deferred to roadmap follow-up (2026-07-21): no operator has asked for it yet; revisit once config push has real-world usage (see brief.md R1)

• Adjacent expectations:

  • Controller OP_upgradeConfig handler (config_repl.c) exists and is operational. On receiving UpgradeConfigRequest{sha256, bytes}:
    1. SHA-256 transit verification
    2. Cap'n Proto decode + reserved name check ("lost_and_found")
    3. Auto-increment generation: new_gen = current_gen + 1, re-encode binary — the primary is the sole authority for configGeneration; whatever value xceadmin ships in cluster.bin is discarded and replaced
    4. Save to disk, replicate to standbys, await majority ACK
    5. Commit WAL_CONFIG_MARKER, signal controller restart
    6. Return UpgradeConfigReply{versionTag, ackedCount, error}
  • Controller redirect: Standby returns redirectHost/redirectPort in the RequestReply union — xceadmin follows the redirect to reach the primary.
  • Per-instance field preservation — RESOLVED structurally, not by controller behavior: port, candidates, the hash-table-size fields, and xstrings have been removed from the ClusterConfig capnp struct entirely (clusterconfig-schema-split spec, landed commit 519e11c). ClusterConfig is now 13 fields (@0–@12, controller.capnp:694-708); these fields simply do not exist in cluster.bin — there is no zero-default-clobber risk to guard against. Req 1.8/1.9/1.10 remain as defense-in-depth documentation (if any of these fields is ever reintroduced, xceadmin must still not source it), not as active field-suppression logic.
  • ProbeConfigTag RPC (xce_probe_config_tag(), config_repl.c:1829): Separate direct-capnp RPC (not in Request union) for reading current config tag. Used by config push --dry-run to validate before committing.
  • controller.capnp schema is the single source of truth for ClusterConfig, Request/Reply unions, and all OP codes.
  • config-upgrade-snapshot spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status.
  • Blocked controller-side gaps (NOT owned by admin-config-tool):
    • Entity-bearing snapshot after config upgrade (blocked on entity-bearing-snapshots spec)
    • Config rollback safety (e.g., reject rollback while jobs reference deleted queues)
    • Config version history query (xceadmin config history)
  • Dependency on clusterconfig-schema-split — RESOLVED, landed (commit 519e11c, phase=implementation-complete). Both subcommands are now unblocked:
    • config build: no dependency, unaffected either way.
    • config push: previously soft-blocked over the risk of xceadmin silently wiping candidates/port/perf-tuning on a zero-valued push. That risk no longer exists — those fields are physically absent from the ClusterConfig schema, so there is nothing for a push to clobber. config push may be exercised against a real cluster without this precondition.

Requirements

Requirement 1: Config Build

Objective: As an XCE operator, I want to compile YAML config files into a ClusterConfig capnp binary, so that I can produce a deployable cluster configuration from human-editable source files.

Acceptance Criteria

1. When the operator runs xceadmin config build <configs_dir>, the system shall scan the directory for YAML config files (hosts.yaml, queues.yaml, resources.yaml, users.yaml, limits.yaml, xce.yaml, scaler.yaml), validate each against its corresponding JSON schema bundled with xceadmin, map the validated data to the ClusterConfig capnp struct, serialize it to binary, and write the result to cluster.bin in the current working directory.
2. When the operator specifies -o <path>, the system shall write the output to the given path instead of the default cluster.bin.
3. When the operator specifies -o -, the system shall write the binary output to stdout.
4. After writing output, the system shall print the SHA-256 hash of the resulting binary to stderr.
5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors).
6. RESOLVED, superseded (2026-07-21): JSON Schema validation is performed against schemas bundled with the xceadmin package (see Requirement 5.5), not a schema/ subdirectory the operator supplies. configs_dir therefore never needs its own schema/ folder, and there is no "missing schema" error mode — the schemas travel with the tool, version-locked to the same release as the ClusterConfig field mapping they validate against (xcecommon.mapping), the same rationale already applied to controller.capnp (see Requirement 5.5, brief.md). An operator-supplied schema/ subdirectory, if present, is ignored — not read, not merged, not treated as an override.
7. Where the YAML content passes JSON Schema validation but maps to fields not present in the ClusterConfig capnp struct, the system shall warn the operator about unrecognized fields.
8. The system shall not source port, the jobmgr/workermgr/qmgr/resmgr hash-table-size fields, or xstrings (xstring hash size) from YAML or CLI input under any circumstance. These are per-controller-instance performance-tuning parameters, not cluster-shared configuration, and the built cluster.bin shall leave them unset (capnp default).
9. The system shall not source, read, or write configGeneration. The built cluster.bin shall leave it unset (capnp default) — the primary controller is the sole authority for assigning it (see Requirement 2 note).
10. The system shall not source candidates (controller peer/standby list) from YAML or CLI input. Standby membership changes independently of, and more frequently than, the rest of cluster configuration, so it shall not require a config build/config push cycle; the built cluster.bin shall leave candidates unset (capnp default). Peer membership sourcing mechanism is out of scope for this spec (open question — see brief.md Open Design Question #3).
11. clusterName, heartbeatInterval/inactiveHeartbeatCount (controller→worker liveness — not the HA/Raft election heartbeat, which has no ClusterConfig field, see .kiro/steering/config-design.md Rule 0), fetchJobInterval (scheduler job-fetch pass interval), and netioTimeout (network I/O timeout) SHALL be sourced from a new xce.yaml file and included in cluster.bin — these must stay consistent across all controller instances (unlike Criteria 8–10, which are intentionally excluded). RESOLVED: source is xce.yaml, a flat (non-list) YAML file holding these 5 scalar ClusterConfig fields. cluster_name is required (no sensible default). heartbeat_interval and fetch_job_interval are optional — both have controller-side defaults (fill_default_cluster_config(), myconf.c:72-76: 5 and 10 respectively). inactive_heartbeat_count is required and has no controller-side default — cluster.c:1622 computes the worker-dead timeout as heartbeat_interval * inactive_heartbeat_count, so an unset (0) value would mark workers dead immediately; xceadmin must not let this ship unset. netio_timeout is optional; note it currently has no reader anywhere in the codebase outside the struct declaration itself (clusterconfig.h:12) — the field is wired on the wire/schema but not yet consumed by any controller logic. That's a controller-side gap, out of scope for this spec, but xceadmin should still source it from YAML if present rather than silently drop it.
12. hproviders (List(ScalerConfig) — autoscaler provider definitions: name, type, conf_path, script_path, script_option) SHALL be sourced from a new scaler.yaml file and included in cluster.bin. Scaler definitions must be identical across controller instances (a queue configured against a scaler must resolve it the same way after failover) and change rarely — both Configuration Inclusion Rules fail, so this field is included per the Default rule.

Criteria 8–12 apply the Configuration Inclusion Rules (brief.md): a field is excluded from cluster.bin if it can change independently of the rest of cluster config, OR if it's expected to change frequently; otherwise it's included. Any future ClusterConfig field proposed for xceadmin's YAML mapping must be classified against these two rules first.

Requirement 2: Config Push

Objective: As an XCE operator, I want to deploy a compiled cluster.bin to a controller cluster, so that the configuration takes effect on the running system.

Acceptance Criteria

1. When the operator runs xceadmin config push <cluster.bin>, the system shall connect to the controller via TLS, send the binary as a capnp Request with OP_upgradeConfig opcode, upgradeconfigrequestmsg body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result.
2. When the operator specifies --controller <host:port>, the system shall connect to the given controller address instead of the default localhost:9991.
3. The system shall use the same TLS certificate bundle the controller cluster is deployed with (xce.crt/xce.key, default path ha/data/) to establish trust — no separate client CA flag or cert-issuance step. Superseded (2026-07-22): verified against the real client TLS setup (netio.c's create_ssl_context()) that the actual controller-to-controller/CLI client presents no certificate at all when acting as a client (cert_file/key_file both NULL → bare TLS_client_method()) — only the server side loads a cert/key pair. xceadmin's client previously diverged from this (loaded a client cert too); fixed to match exactly. No CA flag or cert-issuance step either way — verify_mode=CERT_NONE on both the real client and xceadmin.
4. When the operator specifies --cert-dir <path>, the system shall load xce.crt/xce.key from the given directory instead of the default ha/data/. Superseded (2026-07-22): removed entirely — with no client cert to load, --cert-dir had no remaining purpose. ha/data/ itself was also removed (deploy-ha.sh and the HA test harness now generate a fresh server-side cert/key per run via openssl instead of reading a checked-in fixture).
5. After receiving the controller's reply, the system shall display the result to the operator: versionTag (the new generation number assigned by the controller), ackedCount (number of standbys that acknowledged), and any error message.
6. When the controller indicates the config was committed (positive ackedCount), the system shall exit 0.
7. If the controller reports an error (non-empty error field or zero ackedCount), the system shall display the error and exit non-zero.

Requirement 3: Auto Primary Discovery

Objective: As an XCE operator, I want xceadmin to automatically find the primary controller, so that I don't need to know which node is currently primary.

Acceptance Criteria

1. When the operator connects to any controller node (primary or standby), the system shall inspect the reply for redirectHost and redirectPort fields. If both are non-empty, the system shall reconnect to the redirect target and resend the request.
2. When the redirect target itself redirects, the system shall follow at most 3 redirect hops before reporting an error.
3. When a standby controller redirects the operator to the primary, the system shall display the redirect target to inform the operator where the config was actually pushed.
4. When connecting to a controller and no primary has been elected yet, the system shall time out after 30 seconds and report "no primary available" to the operator.

Requirement 4: Validation

Objective: As an XCE operator, I want validation errors caught early, so that invalid configuration never reaches a running cluster.

Acceptance Criteria

1. When a YAML file fails JSON Schema validation, the system shall report the specific validation error with file name, line/field path, and the constraint violated, then exit non-zero.
2. When a YAML file uses the reserved name "lost_and_found" for a queue or host, the system shall reject it with a clear error message explaining the reservation, even if it passes schema validation.
3. When a YAML file contains duplicate entries (duplicate queue names, host names, resource names, group names, limit names — each checked within its own file/section; a limit name is not required to differ from a queue, host, or group name), the system shall report the duplicates and exit non-zero.
4. When a YAML file references an undefined entity (e.g., a queue references a host group that is not defined, or a limit references a user/queue/project/host group or a resname that is not defined in resources.yaml), the system shall warn the operator but allow the build to proceed (the controller performs final referential integrity checks at runtime).
5. resources.yaml is the exhaustive resource catalog: every resname referenced elsewhere (e.g., limits.yaml, queues.yaml sched_stop/res_req/gpu_req, hosts.yaml sched_stop) — including built-in/dynamic resources such as slots, mem, and the 8 per-host attributes the worker registers automatically (type, model, cpuf, maxcpus, maxmem, maxswap, maxtmp, ndisks — see add_hostattrs_resvals(), controller/worker.c) — shall be declared as an entry in resources.yaml. The system shall not treat any resource name as an implicit built-in exempt from this catalog; Criterion 4's undefined-entity warning applies equally to built-in resource names.

Requirement 5: xcecommon Shared Package

Objective: As a future Python tool author, I want a shared xcecommon package providing capnp schema loading, wire protocol, and config model types, so that I don't duplicate protocol code across tools.

Acceptance Criteria

1. The xcecommon package shall provide a capnp schema loader that reads controller.capnp and exposes the compiled Python types (ClusterConfig, Request, Reply, UpgradeConfigRequest, UpgradeConfigReply, and all nested types).
2. The xcecommon package shall provide a wire protocol module that handles TLS connection, capnp message framing (8-byte uint64 size prefix in host byte order), send, and receive.
3. The xcecommon package shall provide YAML→capnp mapping functions that convert validated YAML data to ClusterConfig capnp struct fields.
4. Where the xcecommon package is installed for development, it shall be importable under the name xcecommon from any tool within the project.
5. (2026-07-21) The xcecommon package shall bundle its own copy of controller.capnp/c.capnp and the 7 JSON Schema files (one per config build input file), synced from src/schema/ and src/configs/schema/ respectively at build time (see Requirement 8.1/8.2) and packaged so they are available regardless of install mode (pip install -e . or pip install .). Neither schema is read from a path outside the installed package at runtime; an operator-supplied configs_dir/schema/ directory, if present, is not consulted (superseded Requirement 1.6).

Requirement 6: Dry-Run Push (Nice-to-Have)

Objective: As an XCE operator, I want to validate that the controller will accept a config before committing to the push.

Acceptance Criteria

1. When the operator runs xceadmin config push --dry-run <cluster.bin>, the system shall connect to the controller, probe the generation, validate SHA and version compatibility, and report whether the config would be accepted, without actually committing the config to the cluster.
2. When dry-run passes, the system shall report "OK: controller will accept this config (generation N→M)" and exit 0.
3. When dry-run fails, the system shall report the specific rejection reason and exit non-zero.

Requirement 7: Error Handling & Resilience

Objective: As an XCE operator, I want clear, actionable error messages for all failure modes, so that I can diagnose and fix configuration issues without reading controller logs.

Acceptance Criteria

1. If the TLS connection to the controller fails, the system shall report "cannot connect to host:port: " and exit non-zero.
2. If the controller returns a Reply with non-zero status, the system shall report the status code and any error text from the reply, then exit non-zero.
3. If the capnp binary is corrupted or cannot be decoded, the system shall report "invalid cluster.bin: " and exit non-zero.
4. If the controller.capnp schema file is not found at the expected path, the system shall report the missing path and suggest checking the installation.
5. If the operator interrupts the push (SIGINT), the system shall clean up the TLS connection and exit gracefully with a "cancelled" message.
6. The system shall time out after 30 seconds on any network operation and report "timeout connecting to host:port".

Requirement 8: Skill & Harness Integration

Objective: As a developer, I want xceadmin integrated into the existing development toolchain, so that building, deploying, and testing XCE clusters uses consistent tooling.

Acceptance Criteria

1. When the developer runs /xce-build xceadmin, the system shall make the xceadmin CLI available on PATH and run the full test suite, reporting pass/fail. Superseded (2026-07-22): xceadmin is now meson-built (src/xceadmin/meson.build) — xce-build.sh xceadmin delegates to meson compile -C "$BUILD_DIR" xceadmin, and nix build .#xcecore[_debug] installs a standalone result/bin/xceadmin (no venv at runtime). See design.md's File Structure Plan Amendment.
2. When the developer runs /xce-build xceadmin release, the system shall install xceadmin for production use (pinned, non-editable) and then run the full test suite against that non-editable install... Superseded (2026-07-22): the editable-vs-pinned distinction collapsed — every meson-driven build (xce-build.sh xceadmin, meson compile, nix build) now runs the pinned/non-editable install path unconditionally, a strictly stronger guarantee (exercises real package_data resolution every run, not just under a release profile).
3. When the developer runs /xce-deploy-ha with the --build-config flag, the system shall invoke xceadmin config build configs/ -o ha/data/cluster.bin as a precondition before deploying the cluster. Superseded (2026-07-22): --build-config removed entirely — every deploy-ha.sh run now unconditionally builds cluster.bin fresh from src/configs/*.yaml directly into the deploy dir's own controller/ctrl-1/conf/ (no ha/data/ staging step, no flag needed since nothing is ever preserved across runs to opt out of rebuilding).
4. When cluster.bin is missing from ha/data/ during deploy, /xce-deploy-ha shall run xceadmin config build automatically (same as --build-config). Superseded (2026-07-22): ha/data/ removed entirely (directory no longer exists). See 8.3 above — always-fresh generation directly into the deploy dir replaces the missing-file check.
5. When cluster.bin already exists and --build-config is not passed, /xce-deploy-ha shall use the existing binary as-is (backward compatible). Superseded (2026-07-22): no longer applicable — nothing persists across deploys for this to preserve; every deploy is fully fresh (cert included, generated via openssl, self-signed, 10y validity — TLS verification is SSL_VERIFY_NONE everywhere in this codebase, so a fresh pair per deploy loses no real security property versus a long-lived shared one).
6. The nix dev shell (nix develop ./src) shall include all Python dependencies required by xceadmin after the nix environment is updated (see Nix Environment section in brief.md).

Requirement 9: Deterministic Builds

Objective: As an XCE operator, I want config builds to be deterministic, so that I can reproduce any historical config binary from its YAML source.

Acceptance Criteria

1. When xceadmin config build is run twice with the same YAML inputs and same xceadmin version, the system shall produce byte-identical cluster.bin output.
2. The system shall not embed build timestamps, hostnames, usernames, or any other non-deterministic data in the output binary.
3. When different operators on different machines build the same YAML with the same xceadmin version, the output shall be byte-identical.

Requirements Summary

#	Requirement	Priority	Phase
1	Config Build	Core	Implementation
2	Config Push	Core	Implementation
3	Auto Primary Discovery	Core	Implementation
4	Validation	Core	Implementation
5	xcecommon Package	Core	Implementation
6	Dry-Run Push	Nice-to-Have	Follow-up
7	Error Handling	Core	Implementation
8	Skill & Harness	Core	Implementation
9	Deterministic Builds	Core	Implementation