179 lines
14 KiB
Markdown
179 lines
14 KiB
Markdown
# Requirements Document
|
|
|
|
## Introduction
|
|
|
|
XCE operators need a tool to build and deploy cluster configuration. Currently, YAML config files (`configs/{hosts,queues,resources,users}.yaml`) exist with JSON schemas for validation, but no tool compiles them into the `ClusterConfig` Cap'n Proto binary the controller expects. The controller-side `OP_upgradeConfig` handler exists (`config_repl.c` — receives binary, verifies SHA-256, pushes to standbys, awaits majority ACK), but no CLI sends it.
|
|
|
|
`xceadmin` is a Python CLI that fills this gap: `config build` compiles YAML into `cluster.bin`, `config push` sends it to a controller over TLS+capnp.
|
|
|
|
Config source of truth is git-managed YAML. `cluster.bin` is a derived artifact — deterministic for the same inputs, never committed to git.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**:
|
|
- YAML → `ClusterConfig` capnp binary compilation (`config build`)
|
|
- TLS+capnp config deployment to controller (`config push`)
|
|
- Auto primary discovery via controller redirect reply
|
|
- JSON Schema validation of YAML inputs
|
|
- Reserved name defense-in-depth (`"lost_and_found"`)
|
|
- `xcecommon` shared Python package (capnp schema, wire protocol, config models)
|
|
- Integration with xce-build and xce-deploy-ha skills
|
|
- Nix dev shell: Python package dependencies
|
|
- `config diff` — human-readable diff of two built config binaries (nice-to-have)
|
|
- `config push --dry-run` — validate without committing (nice-to-have)
|
|
|
|
- **Out of scope**:
|
|
- Controller-side config replication, snapshot, restart logic (owned by `config-upgrade-snapshot`)
|
|
- Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`)
|
|
- Performance tuning parameter extraction (separate work item)
|
|
- xcecli Python rewrite (future, separate spec)
|
|
- Controller config version history, rollback safety validation, config activation status (owned by `config-upgrade-snapshot`, see Future Roadmap in brief.md)
|
|
|
|
- **Adjacent expectations**:
|
|
- **Controller `OP_upgradeConfig` handler** (`config_repl.c`) exists and is operational. On receiving `UpgradeConfigRequest{sha256, bytes}`:
|
|
1. SHA-256 transit verification
|
|
2. Cap'n Proto decode + reserved name check (`"lost_and_found"`)
|
|
3. Auto-increment generation: `new_gen = current_gen + 1`, re-encode binary
|
|
4. Save to disk, replicate to standbys, await majority ACK
|
|
5. Commit `WAL_CONFIG_MARKER`, signal controller restart
|
|
6. Return `UpgradeConfigReply{versionTag, ackedCount, error}`
|
|
- **Controller redirect**: Standby returns `redirectHost`/`redirectPort` in the `RequestReply` union — xceadmin follows the redirect to reach the primary.
|
|
- **ProbeConfigTag RPC** (`config_repl.c:1774`): Separate direct-capnp RPC (not in `Request` union) for reading current config tag. Used by `config push --dry-run` to validate before committing.
|
|
- `controller.capnp` schema is the single source of truth for `ClusterConfig`, `Request`/`Reply` unions, and all OP codes.
|
|
- `config-upgrade-snapshot` spec will extend the controller-side pipeline with: entity-bearing snapshots after config change, rollback safety validation, version history tracking, config activation status.
|
|
- **Blocked controller-side gaps** (NOT owned by admin-config-tool):
|
|
- Entity-bearing snapshot after config upgrade (blocked on `entity-bearing-snapshots` spec)
|
|
- Config rollback safety (e.g., reject rollback while jobs reference deleted queues)
|
|
- Config version history query (`xceadmin config history`)
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Config Build
|
|
|
|
**Objective:** As an XCE operator, I want to compile YAML config files into a `ClusterConfig` capnp binary, so that I can produce a deployable cluster configuration from human-editable source files.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the operator runs `xceadmin config build <configs_dir>`, the system shall scan the directory for YAML config files (`hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`), validate each against its corresponding JSON schema from the `schema/` subdirectory, map the validated data to the `ClusterConfig` capnp struct, serialize it to binary, and write the result to `cluster.bin` in the current working directory.
|
|
2. When the operator specifies `-o <path>`, the system shall write the output to the given path instead of the default `cluster.bin`.
|
|
3. When the operator specifies `-o -`, the system shall write the binary output to stdout.
|
|
4. After writing output, the system shall print the SHA-256 hash of the resulting binary to stderr.
|
|
5. Where a YAML file is missing from the configs directory, the system shall treat the corresponding section as empty and continue building (missing files are not errors).
|
|
6. Where the `schema/` subdirectory is missing from the configs directory, the system shall report an error and exit non-zero.
|
|
7. Where the YAML content passes JSON Schema validation but maps to fields not present in the `ClusterConfig` capnp struct, the system shall warn the operator about unrecognized fields.
|
|
|
|
### Requirement 2: Config Push
|
|
|
|
**Objective:** As an XCE operator, I want to deploy a compiled `cluster.bin` to a controller cluster, so that the configuration takes effect on the running system.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the operator runs `xceadmin config push <cluster.bin>`, the system shall connect to the controller via TLS, send the binary as a capnp `Request` with `OP_upgradeConfig` opcode, `upgradeconfigrequestmsg` body (binary bytes + SHA-256 hash), framed with an 8-byte uint64 size prefix in host byte order, and display the result.
|
|
2. When the operator specifies `--controller <host:port>`, the system shall connect to the given controller address instead of the default `localhost:9991`.
|
|
3. After receiving the controller's reply, the system shall display the result to the operator: `versionTag` (the new generation number assigned by the controller), `ackedCount` (number of standbys that acknowledged), and any `error` message.
|
|
4. When the controller indicates the config was committed (positive `ackedCount`), the system shall exit 0.
|
|
5. If the controller reports an error (non-empty `error` field or zero `ackedCount`), the system shall display the error and exit non-zero.
|
|
|
|
### Requirement 3: Auto Primary Discovery
|
|
|
|
**Objective:** As an XCE operator, I want xceadmin to automatically find the primary controller, so that I don't need to know which node is currently primary.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the operator connects to any controller node (primary or standby), the system shall inspect the reply for `redirectHost` and `redirectPort` fields. If both are non-empty, the system shall reconnect to the redirect target and resend the request.
|
|
2. When the redirect target itself redirects, the system shall follow at most 3 redirect hops before reporting an error.
|
|
3. When a standby controller redirects the operator to the primary, the system shall display the redirect target to inform the operator where the config was actually pushed.
|
|
4. When connecting to a controller and no primary has been elected yet, the system shall time out after 30 seconds and report "no primary available" to the operator.
|
|
|
|
### Requirement 4: Validation
|
|
|
|
**Objective:** As an XCE operator, I want validation errors caught early, so that invalid configuration never reaches a running cluster.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When a YAML file fails JSON Schema validation, the system shall report the specific validation error with file name, line/field path, and the constraint violated, then exit non-zero.
|
|
2. When a YAML file uses the reserved name `"lost_and_found"` for a queue or host, the system shall reject it with a clear error message explaining the reservation, even if it passes schema validation.
|
|
3. When a YAML file contains duplicate entries (duplicate queue names, host names, resource names, group names), the system shall report the duplicates and exit non-zero.
|
|
4. When a YAML file references an undefined entity (e.g., a queue references a host group that is not defined), the system shall warn the operator but allow the build to proceed (the controller performs final referential integrity checks at runtime).
|
|
|
|
### Requirement 5: xcecommon Shared Package
|
|
|
|
**Objective:** As a future Python tool author, I want a shared `xcecommon` package providing capnp schema loading, wire protocol, and config model types, so that I don't duplicate protocol code across tools.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. The `xcecommon` package shall provide a capnp schema loader that reads `controller.capnp` and exposes the compiled Python types (`ClusterConfig`, `Request`, `Reply`, `UpgradeConfigRequest`, `UpgradeConfigReply`, and all nested types).
|
|
2. The `xcecommon` package shall provide a wire protocol module that handles TLS connection, capnp message framing (8-byte uint64 size prefix in host byte order), send, and receive.
|
|
3. The `xcecommon` package shall provide YAML→capnp mapping functions that convert validated YAML data to `ClusterConfig` capnp struct fields.
|
|
4. Where the `xcecommon` package is installed for development, it shall be importable under the name `xcecommon` from any tool within the project.
|
|
|
|
### Requirement 6: Config Diff (Nice-to-Have)
|
|
|
|
**Objective:** As an XCE operator, I want to see what changed between two config binaries, so that I can review changes before pushing.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the operator runs `xceadmin config diff <cluster-v1.bin> <cluster-v2.bin>`, the system shall decode both binaries, compare them field-by-field, and display a human-readable summary of additions, removals, and modifications.
|
|
2. The diff output shall group changes by section (queues, resources, hosts, users, limits, scalers) and show the specific fields that differ.
|
|
3. When the two binaries are identical, the system shall report "no differences" and exit 0.
|
|
|
|
### Requirement 7: Dry-Run Push (Nice-to-Have)
|
|
|
|
**Objective:** As an XCE operator, I want to validate that the controller will accept a config before committing to the push.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the operator runs `xceadmin config push --dry-run <cluster.bin>`, the system shall connect to the controller, probe the generation, validate SHA and version compatibility, and report whether the config would be accepted, without actually committing the config to the cluster.
|
|
2. When dry-run passes, the system shall report "OK: controller will accept this config (generation N→M)" and exit 0.
|
|
3. When dry-run fails, the system shall report the specific rejection reason and exit non-zero.
|
|
|
|
### Requirement 8: Error Handling & Resilience
|
|
|
|
**Objective:** As an XCE operator, I want clear, actionable error messages for all failure modes, so that I can diagnose and fix configuration issues without reading controller logs.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. If the TLS connection to the controller fails, the system shall report "cannot connect to <host:port>: <reason>" and exit non-zero.
|
|
2. If the controller returns a `Reply` with non-zero `status`, the system shall report the status code and any error text from the reply, then exit non-zero.
|
|
3. If the capnp binary is corrupted or cannot be decoded, the system shall report "invalid cluster.bin: <reason>" and exit non-zero.
|
|
4. If the `controller.capnp` schema file is not found at the expected path, the system shall report the missing path and suggest checking the installation.
|
|
5. If the operator interrupts the push (SIGINT), the system shall clean up the TLS connection and exit gracefully with a "cancelled" message.
|
|
6. The system shall time out after 30 seconds on any network operation and report "timeout connecting to <host:port>".
|
|
|
|
### Requirement 9: Skill & Harness Integration
|
|
|
|
**Objective:** As a developer, I want xceadmin integrated into the existing development toolchain, so that building, deploying, and testing XCE clusters uses consistent tooling.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When the developer runs `/xce-build xceadmin`, the system shall make the `xceadmin` CLI available on PATH and run the full test suite, reporting pass/fail.
|
|
2. When the developer runs `/xce-build xceadmin release`, the system shall install `xceadmin` for production use (pinned, non-editable).
|
|
3. When the developer runs `/xce-deploy-ha` with the `--build-config` flag, the system shall invoke `xceadmin config build configs/ -o ha/data/cluster.bin` as a precondition before deploying the cluster.
|
|
4. When `cluster.bin` is missing from `ha/data/` during deploy, `/xce-deploy-ha` shall run `xceadmin config build` automatically (same as `--build-config`).
|
|
5. When `cluster.bin` already exists and `--build-config` is not passed, `/xce-deploy-ha` shall use the existing binary as-is (backward compatible).
|
|
6. The nix dev shell (`nix develop ./src`) shall include all Python dependencies required by xceadmin after the nix environment is updated (see Nix Environment section in brief.md).
|
|
|
|
### Requirement 10: Deterministic Builds
|
|
|
|
**Objective:** As an XCE operator, I want config builds to be deterministic, so that I can reproduce any historical config binary from its YAML source.
|
|
|
|
#### Acceptance Criteria
|
|
|
|
1. When `xceadmin config build` is run twice with the same YAML inputs and same xceadmin version, the system shall produce byte-identical `cluster.bin` output.
|
|
2. The system shall not embed build timestamps, hostnames, usernames, or any other non-deterministic data in the output binary.
|
|
3. When different operators on different machines build the same YAML with the same xceadmin version, the output shall be byte-identical.
|
|
|
|
## Requirements Summary
|
|
|
|
| # | Requirement | Priority | Phase |
|
|
|---|------------|----------|-------|
|
|
| 1 | Config Build | **Core** | Implementation |
|
|
| 2 | Config Push | **Core** | Implementation |
|
|
| 3 | Auto Primary Discovery | **Core** | Implementation |
|
|
| 4 | Validation | **Core** | Implementation |
|
|
| 5 | xcecommon Package | **Core** | Implementation |
|
|
| 6 | Config Diff | Nice-to-Have | Follow-up |
|
|
| 7 | Dry-Run Push | Nice-to-Have | Follow-up |
|
|
| 8 | Error Handling | **Core** | Implementation |
|
|
| 9 | Skill & Harness | **Core** | Implementation |
|
|
| 10 | Deterministic Builds | **Core** | Implementation |
|