# Brief: admin-config-tool

## Problem

XCE operators have no tool to build and deploy cluster configuration. Currently:

- **Config authoring**: YAML config files (`configs/hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`, `limits.yaml`, `xce.yaml`, `scaler.yaml`) exist as documentation examples with JSON schemas for validation (`configs/schema/*.schema.json`), but **no tool compiles them** into the `ClusterConfig` Cap'n Proto binary that the controller expects.
- **Config deployment**: `OP_upgradeConfig` exists on the controller side (`config_repl.c:1591` `do_upgrade_config_with_bytes()` — receives binary blob, verifies SHA-256, saves, pushes to standbys, awaits majority ACK, signals restart), but **no CLI sends it**. Today: manual scp + restart.

End users use `xcecli` (Go/Cobra → REST → apiserver → CGO → controller) for non-admin operations. There is no admin-facing tool.

**Who**: XCE cluster operators. **Pain**: manual config management, error-prone YAML→binary translation, no primary auto-discovery.

## Current State

| What | Where | Status |
|------|-------|--------|
| `ClusterConfig` capnp struct | `src/schema/controller.capnp:694-708` | 13 fields (@0-@12), wire-ready — down from 21 after `clusterconfig-schema-split` removed `port`/`candidates`/hash-size/`xstrings` |
| YAML config examples | `configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml` | Human-authored, NOT loaded by any code |
| JSON schemas | `configs/schema/*.schema.json` | JSON Schema 2020-12 |
| `OP_upgradeConfig` handler | `config_repl.c:1591` `do_upgrade_config_with_bytes()` | Receives binary, verifies, pushes, awaits majority |
| `UpgradeConfigRequest/Reply` | `controller.capnp:792-797` | Defined in schema |
| libxce upgrade API | `request.c:1046` has extractors only | No send function |
| Controller wire protocol | TCP+TLS, capnp-encoded `Request`/`Reply`, same CA as controller-to-controller | Exists, used by all C daemons |

**Key gap**: YAML → ClusterConfig binary tool does not exist. Upload CLI does not exist.

## Desired Outcome

Single `xceadmin` Python CLI that operators use for cluster configuration management:

```
# Build: auto-discover YAML files + schemas from standard directory layout
xceadmin config build configs/                    # default output: cluster.bin
xceadmin config build configs/ -o cluster.bin     # explicit output
xceadmin config build configs/ -o -               # stdout (for piping)

# Push: zero-config dev (same as xce_test), single --controller for production
xceadmin config push cluster.bin                       # default: localhost:9991
xceadmin config push --controller ctrl-1:9991 cluster.bin  # production: any one controller
```

**Standard config directory layout** (auto-discovered):
```
configs/
├── hosts.yaml          # optional — missing = empty hosts section
├── queues.yaml         # optional
├── resources.yaml      # optional
├── users.yaml          # optional
├── limits.yaml         # optional
├── xce.yaml            # optional — missing = clusterName/heartbeat/etc. left unset
└── scaler.yaml         # optional
```
No `schema/` subdirectory — **RESOLVED (2026-07-21)**: the 7 JSON schemas are bundled with the `xceadmin` package itself (`xcecommon` `package_data`, synced from `src/configs/schema/` at build time), not supplied per `configs_dir`. This mirrors the `controller.capnp` bundling decision and for the same reason: the schemas are the formal spec of what `xcecommon.mapping`'s field-extraction code actually expects, so they need to stay version-locked to the `xceadmin` release doing the mapping, not left for an operator to drift out of sync. `src/configs/schema/*.schema.json` remains in the repo as the source of truth that gets synced — an operator's `configs_dir` never needs its own `schema/` folder, and one present there is not read.

- **config build**: Scan directory → validate present YAML files against `xceadmin`'s bundled JSON schemas → map to `ClusterConfig` capnp struct → serialize to `cluster.bin` → output SHA-256. Missing files = empty section (not error). No schema directory to require — bundled, not operator-supplied.
- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. `configGeneration` is fully controller-owned: the primary assigns `new_gen = current_gen + 1` on receipt and returns it as `versionTag`; xceadmin never reads, sets, or overrides it — no `--generation` flag.
- **Shared package**: `xcecommon` Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools

## Approach

**Chosen: Python CLI with single dependency on `controller.capnp`.**

```
src/schema/controller.capnp    ← single source of truth
        │
        ├──▶ c-capnproto → libxce → controller (C daemon, existing)
        │
        └──▶ pycapnp → xceadmin (Python, new)
                │
                ├── config build: YAML → ClusterConfig → bytes → cluster.bin
                └── config push:  Request{OP_upgradeConfig, ...} → TLS → Reply
```

- **Language**: Python. YAML parsing trivial (`yaml`), JSON Schema validation (`jsonschema`), capnp encoding (`pycapnp`), CLI (`click` or `argparse`).
- **Protocol**: TCP+TLS directly to controller (NOT through apiserver). Zero-config dev experience — default `localhost:9991`, same as xce_test. Single `--controller` flag for production override.
- **Dependencies**: `controller.capnp` schema file only. Zero CGO, zero libxce. Pure Python, open-source ready.
- **Auto primary detection**: Connect to given controller → controller reply carries peer list + primary address (standby redirect, already in reply schema) → follow redirect to primary → send request. One controller is enough to bootstrap full cluster discovery. Same mechanism xce_test uses
- **No CGO**: Wire protocol is TLS connect → send one capnp `Request` → receive one capnp `Reply` → exit. ~150 lines of Python. No iomgr event loop, no vtask thread pool, no persistent connections — those are for daemons, not one-shot CLI tools.

### Architecture Decisions (from 2026-07-06 discovery session)

| Decision | Rationale |
|----------|-----------|
| **Python** over Go | Future open-source release; shared capnp infra with planned Python xcecli rewrite; `pip install` no compilation |
| **Pure Python wire** over CGO+libxce | Open-source friendly; no C compilation; wire protocol is one-shot send/receive (~150 lines) |
| **Direct controller** over apiserver | Admin path needs no RBAC middleware; apiserver may be restarting during config upgrade; controller SSL is same CA |
| **Single `controller.capnp` dependency** | Schema IS the protocol contract; versioned; no extra API surface |
| **Separate `xceadmin` binary** over extending xcecli | `xcecli` = user tool (HTTP → apiserver); `xceadmin` = operator tool (TLS → controller); different audiences, different protocols |
| **Auto primary detection** | Operator shouldn't need to know which controller is primary; connect to any one, discover peers + primary from reply |
| **Config build offline** | YAML→binary is pure data transform; no controller connection needed; cluster.bin can be pre-built, reviewed, version-controlled |

### xcecli vs xceadmin

| | xcecli | xceadmin |
|---|--------|----------|
| **Audience** | End users | Operators |
| **Protocol** | HTTP/HTTPS (REST, JSON) | TCP+TLS (capnp binary) |
| **Target** | apiserver | controller (primary) |
| **Auth** | JWT token (login → apiserver) | None — same zero-config TLS as xce_test |
| **Auth middleware** | Full: WhitelistURL→WhitelistIP→RateLimit→JWT→CORS→RBAC | None — direct TCP+TLS, same trust model as controller peers |
| **Use cases** | Submit jobs, list queues, check status | Push config, cluster admin, force snapshot |
| **Language** | Go/Cobra (current), Python (future) | Python |

## Project Decomposition (from 2026-07-06 discovery)

This discovery session (Path E — mixed decomposition) identified 3 workstreams:

| # | Workstream | Type | Language | Dependencies |
|---|-----------|------|----------|--------------|
| 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only |
| 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) |
| 3 | **perf-tuning-extraction** | DIRECT work | doc | None |

**Dependency flow:**
```
admin-config-tool (standalone, ships first)
        │
        │ config push triggers controller-side path
        ▼
config-upgrade-snapshot (controller: replicate → snapshot → restart → activate)
        │
        │ blocked on
        ▼
entity-bearing-snapshots (upstream blocker, tracked in roadmap)
```

## Configuration Inclusion Rules

Decided 2026-07-20. Governs `ClusterConfig`/`cluster.bin` field classification now and for any new field added in the future — apply both rules to every candidate field; excluding on either is sufficient.

> Promoted to project-wide steering: `.kiro/steering/config-design.md` holds the general rule (reusable for worker/scheduler/apiserver shared-config decisions, not just `ClusterConfig`). This section keeps the `ClusterConfig`-specific field classification; see the steering doc for the rule statement and rationale.

1. **Independence rule**: if a parameter can change without needing to stay consistent with the rest of the cluster's config (i.e., changing it on one node doesn't require every other node to match), keep it OUT of `cluster.bin`. Example: `port` — each controller's own listen port is local, other nodes don't need to know or match it via config push.
2. **Frequency rule**: if a parameter is expected to change often relative to the rest of cluster config, keep it OUT of `cluster.bin`. Example: `candidates` — adding/removing a standby is a routine operational action; requiring a config build+push for it would make membership changes as heavyweight as a schema change.
3. **Default (both rules fail → include)**: a parameter that must be identical across every controller instance AND changes rarely belongs IN `cluster.bin`. Example: `clusterName` — divergence between nodes is a correctness bug, and it's set once at bootstrap.

**Correction (2026-07-20, after investigating `clusterconfig-schema-split`)**: the `heartbeatInterval`/`inactiveHeartbeatCount` fields actually govern controller→**worker** liveness (marking a worker unavailable), not the HA/Raft controller-to-controller election heartbeat as originally assumed here — that election heartbeat has no `ClusterConfig` field at all (100% hardcoded, and per a bootstrap-availability constraint it never can have one — see `.kiro/steering/config-design.md` Rule 0). The worker-liveness fields actually *pass* the independence rule (one controller's worker-timeout tolerance doesn't need to match another's) — they stay IN `cluster.bin` by deliberate operational-uniformity override, not because they fail the rules. Verdict unchanged; reasoning corrected.

**Applying the rules** (reference table for future field additions):

| Field | Independence rule | Frequency rule | Verdict |
|---|---|---|---|
| `clusterName` | fails (must match cluster-wide) | fails (rarely changes) | IN |
| heartbeat/inactive-count (worker liveness) | **passes** (per-controller worker-timeout tolerance, no cross-node dependency) | — | IN (operational-uniformity override — see correction above) |
| `port` | passes (per-node, no cross-node dependency) | — | OUT |
| `candidates` | passes (adding a standby doesn't require other nodes' config to change) | passes (routine op) | OUT |
| `jobmgr`/`workermgr`/`qmgr`/`resmgr` hash size | passes (perf tuning, local effect only) | — | OUT |
| `xstrings` hash size | passes (perf tuning, local effect only) | — | OUT |
| `limits` | fails (quotas must be enforced identically on every controller) | fails (rarely changes) | IN |
| `fetchJobInterval` | passes (per-controller scheduler-loop tuning, no cross-node dependency) | — | IN (operational-uniformity override, bundled into `xce.yaml` with the other cluster-identity/liveness fields rather than split into a separate file) |
| `netioTimeout` | passes (per-controller network tuning, no cross-node dependency) | — | IN (same override, same file — see `fetchJobInterval` row) |
| `hproviders` (scaler providers) | fails (a queue's scaler reference must resolve identically after failover) | fails (rarely changes) | IN |
| `configGeneration` | N/A — not operator input; controller-assigned on every push | N/A | OUT (different reason: not a config value at all, it's protocol state) |

When a new field is proposed for `ClusterConfig` in the future, run it through rules 1–2 before adding it to xceadmin's YAML→capnp mapping; document the verdict in this table.

## Scope

- **In**:
  - `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration`, `port`, and the per-instance perf-tuning fields (manager hash-table sizes, `xstrings`) are left unset — not xceadmin's to set.
  - `xceadmin config push`: single controller address (default `localhost:9991`), auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `configGeneration` assignment is entirely controller-side (primary assigns `current + 1`, returns it as `versionTag`) — no client probe, increment, or override flag.
  - `xceadmin config push --dry-run`: (nice-to-have) validate controller will accept the config without committing it
  - `xcecommon` shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types
  - Build system integration: pip-based install (editable for dev, pinned for release), nix dev shell packages, xce-build.sh component target
  - Skills integration (see Skill Integration section below):
    - `xce-build`: add `xceadmin` component target (Python package build + test)
    - `xce-deploy-ha`: replace manual `ha/data/cluster.bin` prerequisite with `xceadmin config build` step; add `--build-config` flag for on-the-fly YAML→binary generation
    - `xce-test-ha`: (future) config push validation round in soak loop — deferred until `config-upgrade-snapshot` lands on controller side
  - Validation: reject configs using reserved name `"lost_and_found"` for queue/host (defense-in-depth — controller enforces at runtime via FATAL, tool prevents at authoring time)
- **Out**:
  - Controller-side config replication/snapshot/restart logic (owned by `config-upgrade-snapshot`)
  - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`, upstream blocker)
  - Performance tuning / per-instance parameter extraction (separate direct work item) — `port`, manager hash-table-size fields, `xstrings`; these stay local to each controller instance, never in `cluster.bin`
  - xcecli Python rewrite (future, separate spec)
  - xce-test-ha config push validation round (deferred until `config-upgrade-snapshot` lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective)

## Skill Integration

Three existing skills touch `cluster.bin` — the binary config blob that `xceadmin config build` produces. Currently `cluster.bin` is a hand-crafted opaque artifact with no visible provenance. `xceadmin` replaces that with a reproducible build step.

### xce-build — add `xceadmin` component

New component target for the Python package:

```
/xce-build xceadmin          # pip install -e . (editable, for dev)
/xce-build xceadmin release  # pip install . (pinned, for production)
```

Build path differs from C components: Python, not C. Runs inside nix dev shell (pycapnp needs capnproto C++ runtime from nix). Build output: `xceadmin` CLI entry point + `xcecommon` package importable.

This is the **blocking** integration — other skills can't use xceadmin until this lands.

### xce-deploy-ha — generate cluster.bin from YAML

**Current flow:**
```
ha/data/cluster.bin  (pre-built, manual, opaque)  →  deploy-ha.sh  →  controller/ctrl-1/conf/
```

**New flow:**
```
configs/*.yaml  →  xceadmin config build  →  cluster.bin  →  deploy-ha.sh  →  ...
```

Changes to the skill:

| What | Before | After |
|------|--------|-------|
| Prerequisites | `cluster.bin` must exist under `ha/data/` | `cluster.bin` built from `configs/*.yaml` via `xceadmin config build` |
| Config build step | None (manual) | `xceadmin config build configs/ -o ha/data/cluster.bin` (auto, or skip if `--use-existing-config`) |
| `--build-config` flag | N/A | New flag: force rebuild cluster.bin from YAML before deploy |

`deploy-ha.sh` gains an optional precondition: if `cluster.bin` missing or `--build-config` passed, run `xceadmin config build`. Otherwise use existing `cluster.bin` as-is (backward compatible — operators can still hand-craft for custom configs).

### xce-test-ha — config push validation (future)

Deferred until `config-upgrade-snapshot` lands on controller side. When ready:

- **New soak round type**: kill primary, push new config via `xceadmin config push`, verify controller accepted + standbys received
- **Coverage**: exercises `OP_upgradeConfig` → replication → majority ACK path under failover
- **Verdict**: config push failure = FAIL (config consistency is a data-safety property)

Not scoped into admin-config-tool tasks — belongs to `config-upgrade-snapshot` spec. Listed here for cross-spec visibility.

### Integration dependency order

```
xce-build (+xceadmin)     ← THIS SPEC, blocking
        │
        ▼
xce-deploy-ha             ← THIS SPEC, immediate after xce-build
        │
        ▼
xce-test-ha               ← config-upgrade-snapshot spec (future)
  (config push round)         blocked on entity-bearing-snapshots
```

## Harness Enhancement

The `xceadmin` tool is Python — first Python component in a C/Go project. This requires harness changes across the development toolchain.

### xce-build.sh — new `xceadmin` component

Add a new case to `src/tools/xce-build.sh`:

```bash
xceadmin)
    echo "=== Building xceadmin Python package ==="
    pip install -e . --quiet
    echo "=== Running xceadmin tests ==="
    python -m pytest tests/ -v
    ;;
```

Python path — no meson setup. `pip install -e .` for dev (editable), `pip install .` for release. Tests via `pytest`, not meson test. Runs inside nix dev shell (capnproto C++ runtime needed for pycapnp compile). The `release` profile uses `pip install .` without `-e`.

### xce-build SKILL.md — add component + usage note

- **Usage line**: `Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin`
- **Profile notes**: `xceadmin` ignores meson profiles — it always uses pip. The profile arg affects C components only; xceadmin builds regardless of profile.
- **Examples**: `/xce-build xceadmin` (editable), `/xce-build xceadmin release` (pinned)

### xce-deploy-ha SKILL.md — config build step

Changes to the skill document:

- **Prerequisites**: replace "TLS certs and a `cluster.bin` must exist under `ha/data/`" with "TLS certs must exist under `ha/data/`. `cluster.bin` is built from `configs/*.yaml` via `xceadmin config build` (or use existing if `--use-existing-config`)."
- **New `--build-config` flag**: forces rebuild of `cluster.bin` from YAML before deploy. Without it, uses existing `cluster.bin` as-is (backward compatible).
- **New step** (between resolve paths and stop running): optional `xceadmin config build configs/ -o ha/data/cluster.bin` if `--build-config` passed or `cluster.bin` missing.
- **Troubleshooting**: add "`pycapnp` import error" → run `/xce-build xceadmin` first.

### settings.local.json — Python permissions

Add Bash permission allowlist entries for xceadmin development:

```json
"Bash(pip install *)",
"Bash(python -m pytest *)",
"Bash(xceadmin config build *)",
"Bash(xceadmin config push *)",
"Bash(ruff format *)",
"Bash(ruff check *)"
```

### PostToolUse hooks — Python formatting + lint

Add a ruff format hook for Python file edits (`settings.json`):

```json
{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff format --quiet \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}
```

Add a ruff check hook for lint feedback:

```json
{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff check --quiet \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}
```

### PreToolUse hook — YAML schema guard (nice-to-have)

Validate config YAML files against JSON schemas before accepting edits. Prevents commits of invalid config:

```json
{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == configs/*.yaml ]] && xceadmin config validate \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}
```

This requires `xceadmin config validate` subcommand (nice-to-have: validates a single YAML file against its schema, no binary output). Deferred to a follow-up task after core `config build` lands.

### `/xce-admin` skill (deferred)

Thin skill wrapper around `xceadmin config build/push` for agent-driven workflows. Not needed for spec implementation — CLI is sufficient. Deferred until CLI is stable and operator workflows are validated.

### Harness change summary

| # | What | Where | Priority | Phase |
|---|------|-------|----------|-------|
| 1 | `xceadmin` case in build script | `src/tools/xce-build.sh` | **Blocking** | Implementation |
| 2 | Component list + pip note | `.claude/skills/xce-build/SKILL.md` | **Blocking** | Implementation |
| 3 | `--build-config` + config build step | `.claude/skills/xce-deploy-ha/SKILL.md` | **Blocking** | Implementation |
| 4 | pip/pytest/xceadmin permissions | `.claude/settings.local.json` | **Blocking** | Implementation |
| 5 | ruff format hook on `.py` edits | `.claude/settings.json` | Nice-to-have | Implementation |
| 6 | YAML schema guard on config edits | `.claude/settings.json` | Nice-to-have | Follow-up |
| 7 | `/xce-admin` skill wrapper | `.claude/skills/xce-admin/SKILL.md` | Deferred | Post-spec |

## Boundary Candidates

- **`xceadmin` CLI** — operator-facing commands, argument parsing, output formatting
- **`xcecommon` Python package** — shared capnp schema loading, wire protocol (TLS+capnp), config model types. Reusable by future Python xcecli.
- **`config-build` module** — YAML→ClusterConfig mapping, JSON Schema validation, serialization
- **`config-push` module** — primary auto-detection, TLS+capnp send/receive, result parsing

## Upstream / Downstream

- **Upstream**: `controller.capnp` schema (source of truth for ClusterConfig, Request/Reply, OP codes)
- **Downstream**: Controller `OP_upgradeConfig` handler (`config_repl.c` — already exists)
- **Consumed by**:
  - `config-upgrade-snapshot` spec — xceadmin is the trigger; controller-side handles replication/snapshot/restart
  - `xce-build` skill — adds `xceadmin` component target (blocking integration)
  - `xce-deploy-ha` skill — `xceadmin config build` replaces manual `cluster.bin` prerequisite
  - `xce-test-ha` skill — (future) config push validation round, deferred until config-upgrade-snapshot lands
- **Future consumer**: Python xcecli rewrite (imports `xcecommon` for capnp schema types)

## Constraints

- **Python ≥ 3.10** (match system Python, avoid bleeding-edge features)
- **pycapnp** requires capnproto C++ runtime (`libcapnp-dev`) — separate from project's c-capnproto (C bindings). Both capnp runtimes can coexist; binary format is standardized.
- **Zero-config TLS** — same dev experience as xce_test: default `localhost:9991`, no cert flags needed. Controller peer trust model (X.509 CN validation via `validate_host()` — tracked in roadmap hardening).
- **No dependency on a running apiserver** — admin tool connects directly to controller. Config deployment works even when apiserver is down or restarting.
- **Open-source readiness** — pure Python, `pip install` friendly, no compilation step, clear single-schema dependency.

## Nix Environment

`xceadmin` is the first Python component in the XCE project. The nix dev shell currently provides C/C++ tooling only — zero Python packages. All needed Python packages are available in nixpkgs.

### Packages to add

Current `src/nixSupport/xce.nix` `nativeBuildInputs` already includes `capnproto` (line 21) — the C++ runtime that pycapnp wraps. Need to add:

| Package | Nix attr | Purpose | Size |
|---------|----------|---------|------|
| Python 3 | `python3` | Interpreter (≥ 3.10) | ~50MB |
| pycapnp | `python3Packages.pycapnp` | capnp schema → Python types | small |
| PyYAML | `python3Packages.pyyaml` | YAML config parsing | small |
| jsonschema | `python3Packages.jsonschema` | JSON Schema validation | small |
| click | `python3Packages.click` | CLI argument parsing | small |
| pytest | `python3Packages.pytest` | Test framework | small |
| ruff | `python3Packages.ruff` | Python formatter + linter | ~10MB |

All confirmed available in nixpkgs (unstable): pycapnp 2.2.3, pyyaml 6.0.3, jsonschema 4.26.0, click 8.3.1, pytest 9.0.3, ruff 0.15.20.

### Where to add

Two options:

**Option A: Add to `xce.nix`** (recommended). Every dev gets Python toolchain. Simpler — one flake, one shell. Python deps are ~60MB total, negligible vs current C toolchain.

**Option B: Separate `xceadmin.nix`**. Lighter shell for C-only devs. More maintenance — two nix files, two shells, cross-component workflows need manual env switching.

Recommend **Option A** — xceadmin is part of the standard toolchain (like xcecli), not an optional extra. All devs should be able to `xceadmin config build` without switching shells.

### New `nativeBuildInputs` block (addition to xce.nix)

```nix
# Python — for xceadmin (admin config tool)
python3
python3Packages.pycapnp
python3Packages.pyyaml
python3Packages.jsonschema
python3Packages.click
python3Packages.pytest
python3Packages.ruff
```

### pycapnp build note

`pycapnp` wraps the capnproto C++ library at compile time (`pip install` compiles a C extension). Nix's `python3Packages.pycapnp` handles this correctly — it links against nix-provided `capnproto`. No additional `libcapnp-dev` or system package needed. This is one of the advantages of using nix pycapnp over `pip install pycapnp`.

### CI implications

- CI already uses nix dev shell — adding packages to `xce.nix` makes them available in CI automatically
- `config build` golden-file tests need pycapnp for capnp encode (can't mock without schema)
- `config push` mock tests need pycapnp + TLS (Python `ssl` stdlib, no extra dep)

## Testing Strategy

### config build (offline — pure data transform)

**Golden-file tests** (primary). Given a `configs/` directory, verify output matches expected:

```
tests/
├── fixtures/
│   ├── minimal/           # hosts.yaml + queues.yaml only
│   │   ├── configs/
│   │   │   ├── hosts.yaml
│   │   │   └── queues.yaml   # no schema/ — validated against xceadmin's bundled schemas
│   │   └── expected.bin   # pre-built golden output
│   ├── full/              # all 7 YAML files
│   └── invalid/           # schema violations → expect error
├── test_config_build.py   # pytest: build → compare binary, verify SHA-256
└── test_schema_validation.py  # pytest: each schema violation case
```

- Framework: `pytest` (standard Python, no deps beyond dev)
- Zero external dependencies — no controller, no network
- Pre-build golden files once; commit to repo
- Schema validation: test each error case (missing required field, wrong type, invalid enum, reserved name `lost_and_found`)

### config push (online — needs controller)

**Option A: Mock controller** (unit/integration). Python TLS server that speaks enough capnp wire protocol to accept `OP_upgradeConfig` and return canned `UpgradeConfigReply`.

**Option B: Real controller** (E2E). Piggyback on existing `/xce-test-ha` cluster. After cluster is up, `xceadmin config push` a known config and verify controller accepts it.

- Option A for dev loop + CI (fast, no cluster setup)
- Option B for acceptance (real SSL, real replication, real controller)

Recommended: Option A for spec tasks. Option B deferred to xce-test-ha integration after config-upgrade-snapshot lands.

### CI Integration

- `config build` tests: run in any Python CI (no C deps needed — pycapnp only for encode, can mock)
- `config push` mock tests: need pycapnp + controller.capnp compiled for Python
- Target: 80%+ coverage (project standard)

## Open Design Questions

Issues identified during 2026-07-07 review — need resolution before or during requirements generation.

### Critical (resolve before requirements)

| # | Question | Why it matters | Candidates |
|---|----------|---------------|------------|
| 1 | **YAML→capnp mapping incomplete** | **RESOLVED — all 13 fields accounted for.** `ClusterConfig` is now 13 fields (@0-@12, post-`clusterconfig-schema-split`), covered by 7 YAML files: `resources`→`resources.yaml`, `queues`→`queues.yaml`, `ugroups`→`users.yaml`, `hgroups`+`hostConfigs`→`hosts.yaml`, `limits`→`limits.yaml` (6 fields, 5 files). `clusterName`/`heartbeatInterval`/`inactiveHeartbeatCount`/`fetchJobInterval`/`netioTimeout`→`xce.yaml` (5 fields, 1 file, Req 1.11). `hproviders`→`scaler.yaml` (1 field, 1 file, Req 1.12). `configGeneration` excluded, controller-assigned (1 field, row 2). 6+5+1+1 = 13. | Closed — no candidates remain |
| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: fully controller-owned. Controller auto-increments `new_gen = current_gen + 1` on every `OP_upgradeConfig` receipt (already implemented, `config_repl.c`) and returns it as `versionTag`. xceadmin never reads, sets, or overrides it — `config build` leaves the field unset, `config push` sends the binary as-is, no `--generation` flag. Same generation as current = no-op is a controller-internal detail, not something xceadmin needs to detect. | ✅ Decided: controller-only, no client involvement |
| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). **RESOLVED — excluded from configuration** (Req 1.10): standby add/remove changes independently and more frequently than the rest of cluster config, so it fails both Configuration Inclusion Rules below. xceadmin never sources it from YAML/CLI, `cluster.bin` leaves it unset. Open sub-question: how peer membership IS sourced/updated instead (today: CLI args to `deploy-ha.sh`) — not a `cluster.bin` field going forward | (a) keep as `deploy-ha.sh` CLI args (status quo, out of xceadmin scope), (b) separate `xceadmin peer add/remove` command (future), (c) controller-side dynamic membership (future, own spec) |
| 4 | **YAML key ≠ capnp field name** | YAML `groupname` → capnp `GroupConfig.name`, YAML `exclude_members` → capnp `exmembers`. Every field needs explicit mapping definition | Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible |
| 5 | **Nested struct defaults** | `QueueConf` has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults | Use controller's `get_default_conf()` values from `myconf.c`; document which fields are operator-facing vs system-defaulted |

### Important (resolve during design)

| # | Question | Why it matters | Candidates |
|---|----------|---------------|------------|
| 6 | **Wire framing format** | Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject | Document in design.md; mock controller test catches framing errors |
| 7 | **TLS cert discovery** | ✅ Decided: reuse server-side cert bundle. xceadmin loads `xce.crt`/`xce.key` from `ha/data/` by default (same files `deploy-ha.sh` copies for controller-to-controller TLS), overridable via `--cert-dir`. No separate CA flag, no client cert issuance — same trust model as `xce_test`. See Requirement 2.3/2.4. |
| 8 | **capnp schema compilation** | ✅ **RESOLVED (design phase)**: no `capnp compile -o python` codegen step exists — `pycapnp` loads `controller.capnp` via runtime dynamic reflection (`capnp.load()`, `DynamicStruct`), not static generated code. The only build-time step is a plain file sync (not compilation): `src/schema/{controller,c}.capnp` and `src/configs/schema/*.schema.json` are copied into `xcecommon`'s `package_data` by `xce-build.sh`'s `xceadmin)` case so both schema types survive `pip install .` (non-editable), resolved at runtime via `importlib.resources`. See `design.md` `xcecommon.schema`. |
| 9 | **Python package layout** | Where does code live? Affects pip install, import paths, xce-build.sh | (a) `src/xceadmin/` alongside C code, (b) `tools/xceadmin/` with deploy-ha.sh, (c) top-level `xceadmin/` |
| 10 | **existing cluster.bin provenance** | `ha/data/cluster.bin` is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it | Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs |

## Configuration Lifecycle & Rollback (Future Roadmap)

### Source of truth: Git-managed YAML

YAML config files (`configs/*.yaml`) are the authoritative source of truth. Operators manage them in git — version control, code review, audit trail are all free. `cluster.bin` is a **derived artifact** (like a compiled binary), never committed to git.

```
git tag v1.2.0  →  YAML at v1.2.0  →  xceadmin config build  →  cluster.bin (deterministic)
                                                                         │
                                                               xceadmin config push
                                                                         │
                                                                         ▼
                                                                  controller cluster
```

**Deterministic builds**: same YAML + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` is left unset by `xceadmin config build` and never touched by `xceadmin config push` — the primary controller alone assigns it on receipt (`current + 1`), so it plays no part in build determinism. Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries.

### What git gives us (free)

| Capability | How |
|-----------|-----|
| Version history | `git log configs/` |
| Diff between versions | `git diff v1.2.0 v1.3.0 -- configs/` |
| Rollback source | `git checkout v1.2.0 -- configs/ && xceadmin config build` |
| Audit trail | `git blame configs/queues.yaml` |
| Code review | PR against `configs/` branch |

### What still needs building

Git manages the **source**. These parts need implementation:

| # | Capability | What it does | Owned by |
|---|-----------|-------------|----------|
| R1 | **Binary diff** | `xceadmin config diff cluster-v1.bin cluster-v2.bin` — human-readable diff of two built config binaries (decode both → compare field-by-field → pretty-print changes). Useful for pre-push review: "what will this config change actually do?" | deferred (2026-07-21) — no spec owner yet |
| R2 | **Dry-run push** | `xceadmin config push --dry-run cluster.bin` — validate controller will accept the config (SHA check, version check) without committing. Safety net before real push | admin-config-tool |
| R3 | **Config change snapshot** | After config upgrade, take an entity-bearing snapshot of the new configuration state. This anchors the config change in the snapshot chain — without it, rollback has no durable record of what the cluster looked like before/after the change. Controller-side: insert snapshot step into the `OP_upgradeConfig` pipeline (between step 5 "majority ACK" and step 6 "commit WAL_CONFIG_MARKER" in the current handler). | config-upgrade-snapshot |
| R4 | **Controller version history** | Controller tracks last N config versions (tag + SHA + timestamp). `xceadmin config history` queries it. Needed for: "what config is the cluster running?" and "what was running before the last push?" Depends on R3 (snapshot chain provides durable version records). | config-upgrade-snapshot |
| R5 | **Controller-side rollback** | `xceadmin config rollback` (or `xceadmin config push --rollback-to v1.2.0`) — push a previous config version. Controller-side concerns: (a) validate rollback target is safe (no running jobs reference deleted queues/hosts), (b) snapshot before rollback for forward-recovery, (c) bump configGeneration even on rollback (monotonic), (d) restart/activate the rolled-back config | config-upgrade-snapshot |
| R6 | **Rollback safety validation** | Before rolling back, verify: (a) no running jobs use queues being deleted, (b) no active workers on hosts being removed, (c) resource definitions haven't changed in ways that break in-flight allocations. Controller-side check, client-side initiated | config-upgrade-snapshot |
| R7 | **Config activation status** | `xceadmin config status` — show: current config tag, when applied, which nodes have it, any pending restart, snapshot status. Operator visibility into config rollout state | config-upgrade-snapshot |
| R8 | **Config promotion pipeline** | Dev → staging → production config promotion. Same YAML source, different `--peers`/`--cluster-name` flags at build time. Git branches map to environments | admin-config-tool (future) |

### Controller-side pipeline: current vs target

```
CURRENT (config_repl.c — exists today):
  xceadmin push → SHA verify → decode → gen++ → save → replicate → WAL marker → restart signal
                                                                                      ↑
                                                                               step 8 MISSING
TARGET (after config-upgrade-snapshot):
  xceadmin push → SHA verify → decode → gen++ → save → replicate → [SNAPSHOT] → WAL marker → restart signal
                                                                   ↑
                                                            R3: entity-bearing
                                                            snapshot of new config
```

The snapshot insertion point matters: it goes AFTER replication (all standbys have the config) but BEFORE the WAL marker commit (so the snapshot is part of the committed state). This ensures a rollback target always has a consistent snapshot to restore from.

### Rollback workflow (end-to-end vision)

```
# 1. Discover what went wrong
xceadmin config diff cluster-current.bin cluster-suspect.bin
# → "queues.normal.priority changed 10→1, queues.express removed"

# 2. Find the last good version
xceadmin config history
# → v1.2.0 (active), v1.1.0 (previous), v1.0.0 (initial)

# 3. Dry-run the rollback
xceadmin config build --git-ref v1.1.0 -o cluster-v1.1.0.bin
xceadmin config push --dry-run cluster-v1.1.0.bin
# → "OK: controller will accept this config (generation 3→4)"

# 4. Execute rollback
xceadmin config push cluster-v1.1.0.bin
# → "Pushed (tag=v1.1.0, SHA=..., acked=3/3). Controller restarting."

# 5. Verify
xceadmin config status
# → "v1.1.0 active on 3/3 nodes"
```

### Rollback safety: what can go wrong

| Scenario | Risk | Mitigation |
|----------|------|------------|
| Rollback deletes a queue | Running jobs on that queue become unmanageable | R5: controller refuses rollback while jobs reference deleted queues |
| Rollback removes a host | Workers on that host lose config | R5: controller refuses rollback while active workers on removed hosts |
| Rollback changes resource defs | In-flight allocations invalid | Controller fences: drain before rollback, or mark resources "changing" |
| Rollback during network partition | Standbys miss the config push | Majority ACK required (existing OP_upgradeConfig behavior) |
| Rapid rollback→forward→rollback | configGeneration churn, snapshot chain fragmentation | Rate-limit config changes (1 per N seconds); snapshot GC |

### What's in-scope for admin-config-tool vs deferred

| Capability | This spec | Why |
|-----------|-----------|-----|
| YAML source in git | ✅ (documentation) | Free — already how configs/ works |
| Deterministic build | ✅ (core) | Same YAML → same binary (generation not set — controller-owned) |
| Binary diff (R1) | ❌ deferred (2026-07-21) | Descoped from admin-config-tool — pure Python/offline, no controller dependency, but no operator has asked for it yet; revisit as follow-up once `config push` sees real-world usage |
| Dry-run push (R2) | ✅ (nice-to-have) | Needs `ProbeConfigTag` RPC (already exists in controller) |
| Config change snapshot (R3) | ❌ deferred | Controller-side — insert entity-bearing snapshot into OP_upgradeConfig pipeline; blocked on entity-bearing-snapshots |
| Version history (R4) | ❌ deferred | Controller-side — depends on R3 snapshot chain |
| Rollback (R5) | ❌ deferred | Controller-side — needs snapshot chain + restart |
| Safety validation (R6) | ❌ deferred | Controller-side — needs job/worker state queries |
| Config status (R7) | ❌ deferred | Controller-side — needs cluster-wide state tracking |
| Promotion pipeline (R8) | ❌ deferred | Wraps git branches + xceadmin build — future enhancement |

## Existing Spec Touchpoints

- **Extends**: `config-upgrade-snapshot` (provides the client-side trigger that config-upgrade-snapshot's controller-side path responds to; R3 config-change-snapshot is the missing controller-side step — insert entity-bearing snapshot between replication and WAL marker commit in the `OP_upgradeConfig` pipeline)
- **Adjacent**: `ha-term-safety` (the WAL_CONFIG_MARKER this tool's upload commits anchors on the Raft log via ha-term-safety's term-index infrastructure)
- **Upstream blocker**: `entity-bearing-snapshots` — must land before R3 (config change snapshot) and R4 (version history) can be implemented
- **Blocks**: nothing — xceadmin ships independently; the current controller pipeline (SHA verify → decode → gen++ → save → replicate → WAL marker → restart) already works without the snapshot step
- **Future**: configuration lifecycle (R1–R8 above) — split across admin-config-tool (R2, R8) and config-upgrade-snapshot (R3–R7); R1 (binary diff) deferred, no spec owner yet