myxce/.kiro/specs/admin-config-tool/requirements.md

Requirements Document

Introduction

XCE operators need a tool to build and deploy cluster configuration. Currently, YAML config files (configs/{hosts,queues,resources,users}.yaml) exist with JSON schemas for validation, but no tool compiles them into the ClusterConfig Cap'n Proto binary the controller expects. The controller-side OP_upgradeConfig handler exists (config_repl.c — receives binary, verifies SHA-256, pushes to standbys, awaits majority ACK), but no CLI sends it.

xceadmin is a Python CLI that fills this gap: config build compiles YAML into cluster.bin, config push sends it to a controller over TLS+capnp.

Config source of truth is git-managed YAML. cluster.bin is a derived artifact — deterministic for the same inputs, never committed to git.

Boundary Context

• In scope:

  • YAML → ClusterConfig capnp binary compilation (config build)
  • TLS+capnp config deployment to controller (config push)
  • Auto primary discovery via controller redirect reply
  • JSON Schema validation of YAML inputs
  • Reserved name defense-in-depth ("lost_and_found")
  • xcecommon shared Python package (capnp schema, wire protocol, config models)
  • Integration with xce-build and xce-deploy-ha skills
  • Nix dev shell: Python package dependencies
  • config diff — human-readable diff of two built config binaries (nice-to-have)
  • config push --dry-run — validate without committing (nice-to-have)

• Out of scope:

  • Controller-side config replication, snapshot, restart logic (owned by config-upgrade-snapshot)
  • Entity-bearing snapshot payload (owned by entity-bearing-snapshots)
  • Performance tuning / per-instance parameter extraction (separate work item) — specifically port, the jobmgr/workermgr/qmgr/resmgr hash-table-size fields, and xstrings (xstring hash size). These are not cluster-shared: each controller instance is expected to set them locally, so xceadmin never reads, accepts, or writes them (see Requirement 1.8).
  • candidates (peer/standby list) — changes independently and more frequently than the rest of cluster config (adding/removing a standby shouldn't require a config build/config push cycle); xceadmin never reads, accepts, or writes it (see Requirement 1.10). Peer membership is owned elsewhere (mechanism TBD — see brief.md Open Design Question #3).
  • configGeneration management — fully controller-owned (see Adjacent Expectations below); xceadmin never reads, increments, or overrides it.
  • xcecli Python rewrite (future, separate spec)
  • Controller config version history, rollback safety validation, config activation status (owned by config-upgrade-snapshot, see Future Roadmap in brief.md)

• Adjacent expectations:

  • Controller OP_upgradeConfig handler (config_repl.c) exists and is operational. On receiving UpgradeConfigRequest{sha256, bytes}:
    1. SHA-256 transit verification
    2. Cap'n Proto decode + reserved name check ("lost_and_found")
    3. Auto-increment generation: new_gen = current_gen + 1, re-encode binary — the primary is the sole authority for configGeneration; whatever value xceadmin ships in cluster.bin is discarded and replaced
    4. Save to disk, replicate to standbys, await majority ACK
    5. Commit WAL_CONFIG_MARKER, signal controller restart
    6. Return UpgradeConfigReply{versionTag, ackedCount, error}
  • Controller redirect: Standby returns redirectHost/redirectPort in the RequestReply union — xceadmin follows the redirect to reach the primary.
  • Per-instance field preservation — RESOLVED structurally, not by controller behavior: port, candidates, the hash-table-size fields, and xstrings are being removed from the ClusterConfig capnp struct entirely, owned by the new clusterconfig-schema-split spec. Once that lands, there is no zero-default-clobber risk to preserve against — these fields simply won't exist in cluster.bin.
  • ProbeConfigTag RPC (config_repl.c:1774): Separate direct-capnp RPC (not in Request union) for reading current config tag. Used by config push --dry-run to validate before committing.
  • controller.capnp schema is the single source of truth for ClusterConfig, Request/Reply unions, and all OP codes.
  • config-upgrade-snapshot spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status.
  • Blocked controller-side gaps (NOT owned by admin-config-tool):
    • Entity-bearing snapshot after config upgrade (blocked on entity-bearing-snapshots spec)
    • Config rollback safety (e.g., reject rollback while jobs reference deleted queues)
    • Config version history query (xceadmin config history)
  • Dependency on clusterconfig-schema-split — split by subcommand:
    • config build: no dependency. Req 1.8/1.10 already exclude these fields from YAML/CLI regardless of schema state — safe to build in parallel, ship anytime.
    • config push: soft-blocked in practice. Until clusterconfig-schema-split lands (fields actually removed from the schema) or someone independently verifies config_repl.c's do_upgrade_config_with_bytes preserves per-instance fields on a zero-valued push, running xceadmin config push against a real cluster risks silently wiping candidates (peer list — HA/Raft membership), port, and perf-tuning on every node it touches. That's not a hypothetical: nobody has checked the controller does the right thing here today. Recommend clusterconfig-schema-split (or equivalent verification) land before config push is exercised against anything but a disposable test cluster; config build and the rest of the CLI can proceed in parallel.

Requirements

Requirement 1: Config Build

Objective: As an XCE operator, I want to compile YAML config files into a ClusterConfig capnp binary, so that I can produce a deployable cluster configuration from human-editable source files.

Acceptance Criteria

1. When the operator runs xceadmin config build <configs_dir>, the system shall scan the directory for YAML config files (hosts.yaml, queues.yaml, resources.yaml, users.yaml), validate each against its corresponding JSON schema from the schema/ subdirectory, map the validated data to the ClusterConfig capnp struct, serialize it to binary, and write the result to cluster.bin in the current working directory.
2. When the operator specifies -o <path>, the system shall write the output to the given path instead of the default cluster.bin.
3. When the operator specifies -o -, the system shall write the binary output to stdout.
4. After writing output, the system shall print the SHA-256 hash of the resulting binary to stderr.
5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors).
6. Where the schema/ subdirectory is missing from the configs directory, the system shall report an error and exit non-zero.
7. Where the YAML content passes JSON Schema validation but maps to fields not present in the ClusterConfig capnp struct, the system shall warn the operator about unrecognized fields.
8. The system shall not source port, the jobmgr/workermgr/qmgr/resmgr hash-table-size fields, or xstrings (xstring hash size) from YAML or CLI input under any circumstance. These are per-controller-instance performance-tuning parameters, not cluster-shared configuration, and the built cluster.bin shall leave them unset (capnp default).
9. The system shall not source, read, or write configGeneration. The built cluster.bin shall leave it unset (capnp default) — the primary controller is the sole authority for assigning it (see Requirement 2 note).
10. The system shall not source candidates (controller peer/standby list) from YAML or CLI input. Standby membership changes independently of, and more frequently than, the rest of cluster configuration, so it shall not require a config build/config push cycle; the built cluster.bin shall leave candidates unset (capnp default). Peer membership sourcing mechanism is out of scope for this spec (open question — see brief.md Open Design Question #3).
11. clusterName and the heartbeatInterval/inactiveHeartbeatCount fields (controller→worker liveness — not the HA/Raft election heartbeat, which has no ClusterConfig field, see .kiro/steering/config-design.md Rule 0) SHALL be sourced from YAML and included in cluster.bin — these must stay consistent across all controller instances (unlike Criteria 8–10, which are intentionally excluded). Source location (existing YAML file vs. new cluster.yaml) is a design-phase decision.

Criteria 8–11 apply the Configuration Inclusion Rules (brief.md): a field is excluded from cluster.bin if it can change independently of the rest of cluster config, OR if it's expected to change frequently; otherwise it's included. Any future ClusterConfig field proposed for xceadmin's YAML mapping must be classified against these two rules first.

Requirement 2: Config Push

Objective: As an XCE operator, I want to deploy a compiled cluster.bin to a controller cluster, so that the configuration takes effect on the running system.

Acceptance Criteria

1. When the operator runs xceadmin config push <cluster.bin>, the system shall connect to the controller via TLS, send the binary as a capnp Request with OP_upgradeConfig opcode, upgradeconfigrequestmsg body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result.
2. When the operator specifies --controller <host:port>, the system shall connect to the given controller address instead of the default localhost:9991.
3. The system shall use the same TLS certificate bundle the controller cluster is deployed with (xce.crt/xce.key, default path ha/data/) to establish trust — no separate client CA flag or cert-issuance step. This matches the trust model xce_test and deploy-ha.sh already use for controller-to-controller TLS.
4. When the operator specifies --cert-dir <path>, the system shall load xce.crt/xce.key from the given directory instead of the default ha/data/.
5. After receiving the controller's reply, the system shall display the result to the operator: versionTag (the new generation number assigned by the controller), ackedCount (number of standbys that acknowledged), and any error message.
6. When the controller indicates the config was committed (positive ackedCount), the system shall exit 0.
7. If the controller reports an error (non-empty error field or zero ackedCount), the system shall display the error and exit non-zero.

Requirement 3: Auto Primary Discovery

Objective: As an XCE operator, I want xceadmin to automatically find the primary controller, so that I don't need to know which node is currently primary.

Acceptance Criteria

1. When the operator connects to any controller node (primary or standby), the system shall inspect the reply for redirectHost and redirectPort fields. If both are non-empty, the system shall reconnect to the redirect target and resend the request.
2. When the redirect target itself redirects, the system shall follow at most 3 redirect hops before reporting an error.
3. When a standby controller redirects the operator to the primary, the system shall display the redirect target to inform the operator where the config was actually pushed.
4. When connecting to a controller and no primary has been elected yet, the system shall time out after 30 seconds and report "no primary available" to the operator.

Requirement 4: Validation

Objective: As an XCE operator, I want validation errors caught early, so that invalid configuration never reaches a running cluster.

Acceptance Criteria

1. When a YAML file fails JSON Schema validation, the system shall report the specific validation error with file name, line/field path, and the constraint violated, then exit non-zero.
2. When a YAML file uses the reserved name "lost_and_found" for a queue or host, the system shall reject it with a clear error message explaining the reservation, even if it passes schema validation.
3. When a YAML file contains duplicate entries (duplicate queue names, host names, resource names, group names), the system shall report the duplicates and exit non-zero.
4. When a YAML file references an undefined entity (e.g., a queue references a host group that is not defined), the system shall warn the operator but allow the build to proceed (the controller performs final referential integrity checks at runtime).

Requirement 5: xcecommon Shared Package

Objective: As a future Python tool author, I want a shared xcecommon package providing capnp schema loading, wire protocol, and config model types, so that I don't duplicate protocol code across tools.

Acceptance Criteria

1. The xcecommon package shall provide a capnp schema loader that reads controller.capnp and exposes the compiled Python types (ClusterConfig, Request, Reply, UpgradeConfigRequest, UpgradeConfigReply, and all nested types).
2. The xcecommon package shall provide a wire protocol module that handles TLS connection, capnp message framing (8-byte uint64 size prefix in host byte order), send, and receive.
3. The xcecommon package shall provide YAML→capnp mapping functions that convert validated YAML data to ClusterConfig capnp struct fields.
4. Where the xcecommon package is installed for development, it shall be importable under the name xcecommon from any tool within the project.

Requirement 6: Config Diff (Nice-to-Have)

Objective: As an XCE operator, I want to see what changed between two config binaries, so that I can review changes before pushing.

Acceptance Criteria

1. When the operator runs xceadmin config diff <cluster-v1.bin> <cluster-v2.bin>, the system shall decode both binaries, compare them field-by-field, and display a human-readable summary of additions, removals, and modifications.
2. The diff output shall group changes by section (queues, resources, hosts, users, limits, scalers) and show the specific fields that differ.
3. When the two binaries are identical, the system shall report "no differences" and exit 0.

Requirement 7: Dry-Run Push (Nice-to-Have)

Objective: As an XCE operator, I want to validate that the controller will accept a config before committing to the push.

Acceptance Criteria

1. When the operator runs xceadmin config push --dry-run <cluster.bin>, the system shall connect to the controller, probe the generation, validate SHA and version compatibility, and report whether the config would be accepted, without actually committing the config to the cluster.
2. When dry-run passes, the system shall report "OK: controller will accept this config (generation N→M)" and exit 0.
3. When dry-run fails, the system shall report the specific rejection reason and exit non-zero.

Requirement 8: Error Handling & Resilience

Objective: As an XCE operator, I want clear, actionable error messages for all failure modes, so that I can diagnose and fix configuration issues without reading controller logs.

Acceptance Criteria

1. If the TLS connection to the controller fails, the system shall report "cannot connect to host:port: " and exit non-zero.
2. If the controller returns a Reply with non-zero status, the system shall report the status code and any error text from the reply, then exit non-zero.
3. If the capnp binary is corrupted or cannot be decoded, the system shall report "invalid cluster.bin: " and exit non-zero.
4. If the controller.capnp schema file is not found at the expected path, the system shall report the missing path and suggest checking the installation.
5. If the operator interrupts the push (SIGINT), the system shall clean up the TLS connection and exit gracefully with a "cancelled" message.
6. The system shall time out after 30 seconds on any network operation and report "timeout connecting to host:port".

Requirement 9: Skill & Harness Integration

Objective: As a developer, I want xceadmin integrated into the existing development toolchain, so that building, deploying, and testing XCE clusters uses consistent tooling.

Acceptance Criteria

1. When the developer runs /xce-build xceadmin, the system shall make the xceadmin CLI available on PATH and run the full test suite, reporting pass/fail.
2. When the developer runs /xce-build xceadmin release, the system shall install xceadmin for production use (pinned, non-editable).
3. When the developer runs /xce-deploy-ha with the --build-config flag, the system shall invoke xceadmin config build configs/ -o ha/data/cluster.bin as a precondition before deploying the cluster.
4. When cluster.bin is missing from ha/data/ during deploy, /xce-deploy-ha shall run xceadmin config build automatically (same as --build-config).
5. When cluster.bin already exists and --build-config is not passed, /xce-deploy-ha shall use the existing binary as-is (backward compatible).
6. The nix dev shell (nix develop ./src) shall include all Python dependencies required by xceadmin after the nix environment is updated (see Nix Environment section in brief.md).

Requirement 10: Deterministic Builds

Objective: As an XCE operator, I want config builds to be deterministic, so that I can reproduce any historical config binary from its YAML source.

Acceptance Criteria

1. When xceadmin config build is run twice with the same YAML inputs and same xceadmin version, the system shall produce byte-identical cluster.bin output.
2. The system shall not embed build timestamps, hostnames, usernames, or any other non-deterministic data in the output binary.
3. When different operators on different machines build the same YAML with the same xceadmin version, the output shall be byte-identical.

Requirements Summary

#	Requirement	Priority	Phase
1	Config Build	Core	Implementation
2	Config Push	Core	Implementation
3	Auto Primary Discovery	Core	Implementation
4	Validation	Core	Implementation
5	xcecommon Package	Core	Implementation
6	Config Diff	Nice-to-Have	Follow-up
7	Dry-Run Push	Nice-to-Have	Follow-up
8	Error Handling	Core	Implementation
9	Skill & Harness	Core	Implementation
10	Deterministic Builds	Core	Implementation