myxce/.kiro/specs/admin-config-tool/brief.md

# Brief: admin-config-tool

## Problem

XCE operators have no tool to build and deploy cluster configuration. Currently:

- **Config authoring**: YAML config files (`configs/hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`) exist as documentation examples with JSON schemas for validation (`configs/schema/*.schema.json`), but **no tool compiles them** into the `ClusterConfig` Cap'n Proto binary that the controller expects.
- **Config deployment**: `OP_upgradeConfig` exists on the controller side (`config_repl.c:1536` — receives binary blob, verifies SHA-256, saves, pushes to standbys, awaits majority ACK, signals restart), but **no CLI sends it**. Today: manual scp + restart.

End users use `xcecli` (Go/Cobra → REST → apiserver → CGO → controller) for non-admin operations. There is no admin-facing tool.

**Who**: XCE cluster operators. **Pain**: manual config management, error-prone YAML→binary translation, no primary auto-discovery.

## Current State

| What | Where | Status |
|------|-------|--------|
| `ClusterConfig` capnp struct | `src/schema/controller.capnp:710-732` | 21 fields, wire-ready |
| YAML config examples | `configs/{hosts,queues,resources,users}.yaml` | Human-authored, NOT loaded by any code |
| JSON schemas | `configs/schema/*.schema.json` | JSON Schema 2020-12 |
| `OP_upgradeConfig` handler | `config_repl.c:1536` `do_upgrade_config_with_bytes()` | Receives binary, verifies, pushes, awaits majority |
| `UpgradeConfigRequest/Reply` | `controller.capnp:792-797` | Defined in schema |
| libxce upgrade API | `request.c:1046` has extractors only | No send function |
| Controller wire protocol | TCP+TLS, capnp-encoded `Request`/`Reply`, same CA as controller-to-controller | Exists, used by all C daemons |

**Key gap**: YAML → ClusterConfig binary tool does not exist. Upload CLI does not exist.

## Desired Outcome

Single `xceadmin` Python CLI that operators use for cluster configuration management:

```
# Build: auto-discover YAML files + schemas from standard directory layout
xceadmin config build configs/                    # default output: cluster.bin
xceadmin config build configs/ -o cluster.bin     # explicit output
xceadmin config build configs/ -o -               # stdout (for piping)

# Push: zero-config dev (same as xce_test), single --controller for production
xceadmin config push cluster.bin                       # default: localhost:9991
xceadmin config push --controller ctrl-1:9991 cluster.bin  # production: any one controller
```

**Standard config directory layout** (auto-discovered):
```
configs/
├── hosts.yaml          # optional — missing = empty hosts section
├── queues.yaml         # optional
├── resources.yaml      # optional
├── users.yaml          # optional
└── schema/             # required for validation
    ├── hosts.schema.json
    ├── queues.schema.json
    ├── resources.schema.json
    └── users.schema.json
```

- **config build**: Scan directory → validate present YAML files against corresponding JSON schemas → map to `ClusterConfig` capnp struct → serialize to `cluster.bin` → output SHA-256. Missing files = empty section (not error). Schema directory must exist.
- **config push**: Read `cluster.bin` → connect to controller (default `localhost:9991`) → probe current `configGeneration` → if binary's generation ≤ current, increment to `current + 1` and re-encode (controller rejects non-increasing generations as no-op) → controller reply includes peer list + primary address → follow redirect if needed → send capnp `Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}}` → receive `Reply` → display `{versionTag, ackedCount, error}`. Admin can force a specific generation with `--generation N`.
- **Shared package**: `xcecommon` Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools

## Approach

**Chosen: Python CLI with single dependency on `controller.capnp`.**

```
src/schema/controller.capnp    ← single source of truth
        │
        ├──▶ c-capnproto → libxce → controller (C daemon, existing)
        │
        └──▶ pycapnp → xceadmin (Python, new)
                │
                ├── config build: YAML → ClusterConfig → bytes → cluster.bin
                └── config push:  Request{OP_upgradeConfig, ...} → TLS → Reply
```

- **Language**: Python. YAML parsing trivial (`yaml`), JSON Schema validation (`jsonschema`), capnp encoding (`pycapnp`), CLI (`click` or `argparse`).
- **Protocol**: TCP+TLS directly to controller (NOT through apiserver). Zero-config dev experience — default `localhost:9991`, same as xce_test. Single `--controller` flag for production override.
- **Dependencies**: `controller.capnp` schema file only. Zero CGO, zero libxce. Pure Python, open-source ready.
- **Auto primary detection**: Connect to given controller → controller reply carries peer list + primary address (standby redirect, already in reply schema) → follow redirect to primary → send request. One controller is enough to bootstrap full cluster discovery. Same mechanism xce_test uses
- **No CGO**: Wire protocol is TLS connect → send one capnp `Request` → receive one capnp `Reply` → exit. ~150 lines of Python. No iomgr event loop, no vtask thread pool, no persistent connections — those are for daemons, not one-shot CLI tools.

### Architecture Decisions (from 2026-07-06 discovery session)

| Decision | Rationale |
|----------|-----------|
| **Python** over Go | Future open-source release; shared capnp infra with planned Python xcecli rewrite; `pip install` no compilation |
| **Pure Python wire** over CGO+libxce | Open-source friendly; no C compilation; wire protocol is one-shot send/receive (~150 lines) |
| **Direct controller** over apiserver | Admin path needs no RBAC middleware; apiserver may be restarting during config upgrade; controller SSL is same CA |
| **Single `controller.capnp` dependency** | Schema IS the protocol contract; versioned; no extra API surface |
| **Separate `xceadmin` binary** over extending xcecli | `xcecli` = user tool (HTTP → apiserver); `xceadmin` = operator tool (TLS → controller); different audiences, different protocols |
| **Auto primary detection** | Operator shouldn't need to know which controller is primary; connect to any one, discover peers + primary from reply |
| **Config build offline** | YAML→binary is pure data transform; no controller connection needed; cluster.bin can be pre-built, reviewed, version-controlled |

### xcecli vs xceadmin

| | xcecli | xceadmin |
|---|--------|----------|
| **Audience** | End users | Operators |
| **Protocol** | HTTP/HTTPS (REST, JSON) | TCP+TLS (capnp binary) |
| **Target** | apiserver | controller (primary) |
| **Auth** | JWT token (login → apiserver) | None — same zero-config TLS as xce_test |
| **Auth middleware** | Full: WhitelistURL→WhitelistIP→RateLimit→JWT→CORS→RBAC | None — direct TCP+TLS, same trust model as controller peers |
| **Use cases** | Submit jobs, list queues, check status | Push config, cluster admin, force snapshot |
| **Language** | Go/Cobra (current), Python (future) | Python |

## Project Decomposition (from 2026-07-06 discovery)

This discovery session (Path E — mixed decomposition) identified 3 workstreams:

| # | Workstream | Type | Language | Dependencies |
|---|-----------|------|----------|--------------|
| 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only |
| 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) |
| 3 | **perf-tuning-extraction** | DIRECT work | doc | None |

**Dependency flow:**
```
admin-config-tool (standalone, ships first)
        │
        │ config push triggers controller-side path
        ▼
config-upgrade-snapshot (controller: replicate → snapshot → restart → activate)
        │
        │ blocked on
        ▼
entity-bearing-snapshots (upstream blocker, tracked in roadmap)
```

## Scope

- **In**:
  - `xceadmin config build`: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. `configGeneration` defaults to 0 (offline-safe).
  - `xceadmin config push`: single controller address (default `localhost:9991`), probe current configGeneration from controller, auto-increment if needed, auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. `--generation N` flag for explicit override.
  - `xceadmin config diff`: (nice-to-have) decode two cluster.bin files, human-readable field-by-field diff. Pre-push safety: "what will this config change actually do?"
  - `xceadmin config push --dry-run`: (nice-to-have) validate controller will accept the config without committing it
  - `xcecommon` shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types
  - Build system integration: pip-based install (editable for dev, pinned for release), nix dev shell packages, xce-build.sh component target
  - Skills integration (see Skill Integration section below):
    - `xce-build`: add `xceadmin` component target (Python package build + test)
    - `xce-deploy-ha`: replace manual `ha/data/cluster.bin` prerequisite with `xceadmin config build` step; add `--build-config` flag for on-the-fly YAML→binary generation
    - `xce-test-ha`: (future) config push validation round in soak loop — deferred until `config-upgrade-snapshot` lands on controller side
  - Validation: reject configs using reserved name `"lost_and_found"` for queue/host (defense-in-depth — controller enforces at runtime via FATAL, tool prevents at authoring time)
- **Out**:
  - Controller-side config replication/snapshot/restart logic (owned by `config-upgrade-snapshot`)
  - Entity-bearing snapshot payload (owned by `entity-bearing-snapshots`, upstream blocker)
  - Performance tuning parameter extraction (separate direct work item)
  - xcecli Python rewrite (future, separate spec)
  - xce-test-ha config push validation round (deferred until `config-upgrade-snapshot` lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective)

## Skill Integration

Three existing skills touch `cluster.bin` — the binary config blob that `xceadmin config build` produces. Currently `cluster.bin` is a hand-crafted opaque artifact with no visible provenance. `xceadmin` replaces that with a reproducible build step.

### xce-build — add `xceadmin` component

New component target for the Python package:

```
/xce-build xceadmin          # pip install -e . (editable, for dev)
/xce-build xceadmin release  # pip install . (pinned, for production)
```

Build path differs from C components: Python, not C. Runs inside nix dev shell (pycapnp needs capnproto C++ runtime from nix). Build output: `xceadmin` CLI entry point + `xcecommon` package importable.

This is the **blocking** integration — other skills can't use xceadmin until this lands.

### xce-deploy-ha — generate cluster.bin from YAML

**Current flow:**
```
ha/data/cluster.bin  (pre-built, manual, opaque)  →  deploy-ha.sh  →  controller/ctrl-1/conf/
```

**New flow:**
```
configs/*.yaml  →  xceadmin config build  →  cluster.bin  →  deploy-ha.sh  →  ...
```

Changes to the skill:

| What | Before | After |
|------|--------|-------|
| Prerequisites | `cluster.bin` must exist under `ha/data/` | `cluster.bin` built from `configs/*.yaml` via `xceadmin config build` |
| Config build step | None (manual) | `xceadmin config build configs/ -o ha/data/cluster.bin` (auto, or skip if `--use-existing-config`) |
| `--build-config` flag | N/A | New flag: force rebuild cluster.bin from YAML before deploy |

`deploy-ha.sh` gains an optional precondition: if `cluster.bin` missing or `--build-config` passed, run `xceadmin config build`. Otherwise use existing `cluster.bin` as-is (backward compatible — operators can still hand-craft for custom configs).

### xce-test-ha — config push validation (future)

Deferred until `config-upgrade-snapshot` lands on controller side. When ready:

- **New soak round type**: kill primary, push new config via `xceadmin config push`, verify controller accepted + standbys received
- **Coverage**: exercises `OP_upgradeConfig` → replication → majority ACK path under failover
- **Verdict**: config push failure = FAIL (config consistency is a data-safety property)

Not scoped into admin-config-tool tasks — belongs to `config-upgrade-snapshot` spec. Listed here for cross-spec visibility.

### Integration dependency order

```
xce-build (+xceadmin)     ← THIS SPEC, blocking
        │
        ▼
xce-deploy-ha             ← THIS SPEC, immediate after xce-build
        │
        ▼
xce-test-ha               ← config-upgrade-snapshot spec (future)
  (config push round)         blocked on entity-bearing-snapshots
```

## Harness Enhancement

The `xceadmin` tool is Python — first Python component in a C/Go project. This requires harness changes across the development toolchain.

### xce-build.sh — new `xceadmin` component

Add a new case to `src/tools/xce-build.sh`:

```bash
xceadmin)
    echo "=== Building xceadmin Python package ==="
    pip install -e . --quiet
    echo "=== Running xceadmin tests ==="
    python -m pytest tests/ -v
    ;;
```

Python path — no meson setup. `pip install -e .` for dev (editable), `pip install .` for release. Tests via `pytest`, not meson test. Runs inside nix dev shell (capnproto C++ runtime needed for pycapnp compile). The `release` profile uses `pip install .` without `-e`.

### xce-build SKILL.md — add component + usage note

- **Usage line**: `Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin`
- **Profile notes**: `xceadmin` ignores meson profiles — it always uses pip. The profile arg affects C components only; xceadmin builds regardless of profile.
- **Examples**: `/xce-build xceadmin` (editable), `/xce-build xceadmin release` (pinned)

### xce-deploy-ha SKILL.md — config build step

Changes to the skill document:

- **Prerequisites**: replace "TLS certs and a `cluster.bin` must exist under `ha/data/`" with "TLS certs must exist under `ha/data/`. `cluster.bin` is built from `configs/*.yaml` via `xceadmin config build` (or use existing if `--use-existing-config`)."
- **New `--build-config` flag**: forces rebuild of `cluster.bin` from YAML before deploy. Without it, uses existing `cluster.bin` as-is (backward compatible).
- **New step** (between resolve paths and stop running): optional `xceadmin config build configs/ -o ha/data/cluster.bin` if `--build-config` passed or `cluster.bin` missing.
- **Troubleshooting**: add "`pycapnp` import error" → run `/xce-build xceadmin` first.

### settings.local.json — Python permissions

Add Bash permission allowlist entries for xceadmin development:

```json
"Bash(pip install *)",
"Bash(python -m pytest *)",
"Bash(xceadmin config build *)",
"Bash(xceadmin config push *)",
"Bash(ruff format *)",
"Bash(ruff check *)"
```

### PostToolUse hooks — Python formatting + lint

Add a ruff format hook for Python file edits (`settings.json`):

```json
{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff format --quiet \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}
```

Add a ruff check hook for lint feedback:

```json
{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff check --quiet \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}
```

### PreToolUse hook — YAML schema guard (nice-to-have)

Validate config YAML files against JSON schemas before accepting edits. Prevents commits of invalid config:

```json
{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == configs/*.yaml ]] && xceadmin config validate \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}
```

This requires `xceadmin config validate` subcommand (nice-to-have: validates a single YAML file against its schema, no binary output). Deferred to a follow-up task after core `config build` lands.

### `/xce-admin` skill (deferred)

Thin skill wrapper around `xceadmin config build/push` for agent-driven workflows. Not needed for spec implementation — CLI is sufficient. Deferred until CLI is stable and operator workflows are validated.

### Harness change summary

| # | What | Where | Priority | Phase |
|---|------|-------|----------|-------|
| 1 | `xceadmin` case in build script | `src/tools/xce-build.sh` | **Blocking** | Implementation |
| 2 | Component list + pip note | `.claude/skills/xce-build/SKILL.md` | **Blocking** | Implementation |
| 3 | `--build-config` + config build step | `.claude/skills/xce-deploy-ha/SKILL.md` | **Blocking** | Implementation |
| 4 | pip/pytest/xceadmin permissions | `.claude/settings.local.json` | **Blocking** | Implementation |
| 5 | ruff format hook on `.py` edits | `.claude/settings.json` | Nice-to-have | Implementation |
| 6 | YAML schema guard on config edits | `.claude/settings.json` | Nice-to-have | Follow-up |
| 7 | `/xce-admin` skill wrapper | `.claude/skills/xce-admin/SKILL.md` | Deferred | Post-spec |

## Boundary Candidates

- **`xceadmin` CLI** — operator-facing commands, argument parsing, output formatting
- **`xcecommon` Python package** — shared capnp schema loading, wire protocol (TLS+capnp), config model types. Reusable by future Python xcecli.
- **`config-build` module** — YAML→ClusterConfig mapping, JSON Schema validation, serialization
- **`config-push` module** — primary auto-detection, TLS+capnp send/receive, result parsing

## Upstream / Downstream

- **Upstream**: `controller.capnp` schema (source of truth for ClusterConfig, Request/Reply, OP codes)
- **Downstream**: Controller `OP_upgradeConfig` handler (`config_repl.c` — already exists)
- **Consumed by**:
  - `config-upgrade-snapshot` spec — xceadmin is the trigger; controller-side handles replication/snapshot/restart
  - `xce-build` skill — adds `xceadmin` component target (blocking integration)
  - `xce-deploy-ha` skill — `xceadmin config build` replaces manual `cluster.bin` prerequisite
  - `xce-test-ha` skill — (future) config push validation round, deferred until config-upgrade-snapshot lands
- **Future consumer**: Python xcecli rewrite (imports `xcecommon` for capnp schema types)

## Constraints

- **Python ≥ 3.10** (match system Python, avoid bleeding-edge features)
- **pycapnp** requires capnproto C++ runtime (`libcapnp-dev`) — separate from project's c-capnproto (C bindings). Both capnp runtimes can coexist; binary format is standardized.
- **Zero-config TLS** — same dev experience as xce_test: default `localhost:9991`, no cert flags needed. Controller peer trust model (X.509 CN validation via `validate_host()` — tracked in roadmap hardening).
- **No dependency on a running apiserver** — admin tool connects directly to controller. Config deployment works even when apiserver is down or restarting.
- **Open-source readiness** — pure Python, `pip install` friendly, no compilation step, clear single-schema dependency.

## Nix Environment

`xceadmin` is the first Python component in the XCE project. The nix dev shell currently provides C/C++ tooling only — zero Python packages. All needed Python packages are available in nixpkgs.

### Packages to add

Current `src/nixSupport/xce.nix` `nativeBuildInputs` already includes `capnproto` (line 21) — the C++ runtime that pycapnp wraps. Need to add:

| Package | Nix attr | Purpose | Size |
|---------|----------|---------|------|
| Python 3 | `python3` | Interpreter (≥ 3.10) | ~50MB |
| pycapnp | `python3Packages.pycapnp` | capnp schema → Python types | small |
| PyYAML | `python3Packages.pyyaml` | YAML config parsing | small |
| jsonschema | `python3Packages.jsonschema` | JSON Schema validation | small |
| click | `python3Packages.click` | CLI argument parsing | small |
| pytest | `python3Packages.pytest` | Test framework | small |
| ruff | `python3Packages.ruff` | Python formatter + linter | ~10MB |

All confirmed available in nixpkgs (unstable): pycapnp 2.2.3, pyyaml 6.0.3, jsonschema 4.26.0, click 8.3.1, pytest 9.0.3, ruff 0.15.20.

### Where to add

Two options:

**Option A: Add to `xce.nix`** (recommended). Every dev gets Python toolchain. Simpler — one flake, one shell. Python deps are ~60MB total, negligible vs current C toolchain.

**Option B: Separate `xceadmin.nix`**. Lighter shell for C-only devs. More maintenance — two nix files, two shells, cross-component workflows need manual env switching.

Recommend **Option A** — xceadmin is part of the standard toolchain (like xcecli), not an optional extra. All devs should be able to `xceadmin config build` without switching shells.

### New `nativeBuildInputs` block (addition to xce.nix)

```nix
# Python — for xceadmin (admin config tool)
python3
python3Packages.pycapnp
python3Packages.pyyaml
python3Packages.jsonschema
python3Packages.click
python3Packages.pytest
python3Packages.ruff
```

### pycapnp build note

`pycapnp` wraps the capnproto C++ library at compile time (`pip install` compiles a C extension). Nix's `python3Packages.pycapnp` handles this correctly — it links against nix-provided `capnproto`. No additional `libcapnp-dev` or system package needed. This is one of the advantages of using nix pycapnp over `pip install pycapnp`.

### CI implications

- CI already uses nix dev shell — adding packages to `xce.nix` makes them available in CI automatically
- `config build` golden-file tests need pycapnp for capnp encode (can't mock without schema)
- `config push` mock tests need pycapnp + TLS (Python `ssl` stdlib, no extra dep)

## Testing Strategy

### config build (offline — pure data transform)

**Golden-file tests** (primary). Given a `configs/` directory, verify output matches expected:

```
tests/
├── fixtures/
│   ├── minimal/           # hosts.yaml + queues.yaml only
│   │   ├── configs/
│   │   │   ├── hosts.yaml
│   │   │   ├── queues.yaml
│   │   │   └── schema/
│   │   └── expected.bin   # pre-built golden output
│   ├── full/              # all 4 YAML files
│   └── invalid/           # schema violations → expect error
├── test_config_build.py   # pytest: build → compare binary, verify SHA-256
└── test_schema_validation.py  # pytest: each schema violation case
```

- Framework: `pytest` (standard Python, no deps beyond dev)
- Zero external dependencies — no controller, no network
- Pre-build golden files once; commit to repo
- Schema validation: test each error case (missing required field, wrong type, invalid enum, reserved name `lost_and_found`)

### config push (online — needs controller)

**Option A: Mock controller** (unit/integration). Python TLS server that speaks enough capnp wire protocol to accept `OP_upgradeConfig` and return canned `UpgradeConfigReply`.

**Option B: Real controller** (E2E). Piggyback on existing `/xce-test-ha` cluster. After cluster is up, `xceadmin config push` a known config and verify controller accepts it.

- Option A for dev loop + CI (fast, no cluster setup)
- Option B for acceptance (real SSL, real replication, real controller)

Recommended: Option A for spec tasks. Option B deferred to xce-test-ha integration after config-upgrade-snapshot lands.

### CI Integration

- `config build` tests: run in any Python CI (no C deps needed — pycapnp only for encode, can mock)
- `config push` mock tests: need pycapnp + controller.capnp compiled for Python
- Target: 80%+ coverage (project standard)

## Open Design Questions

Issues identified during 2026-07-07 review — need resolution before or during requirements generation.

### Critical (resolve before requirements)

| # | Question | Why it matters | Candidates |
|---|----------|---------------|------------|
| 1 | **YAML→capnp mapping incomplete** | `ClusterConfig` has 21 fields; 4 YAML files cover ~10. Remaining 11 (clusterName, port, candidates, heartbeat/interval timeouts, jobmgr/workermgr/qmgr/resmgr hash configs, xstrings, configGeneration) have no YAML source | (a) xceadmin fills defaults from controller's `get_default_conf()` values, (b) add a `cluster.yaml` for system-level fields, (c) accept CLI flags `--cluster-name`, `--port`, `--peers` |
| 2 | **configGeneration** | Monotonic counter. **RESOLVED**: Controller uses it to detect new configs — same generation as current = no-op (nothing changed). xceadmin MUST read the current generation from the controller, increment it, and embed the incremented value in the binary before pushing. `xceadmin config build` sets it to 0 (offline-safe default). `xceadmin config push` auto-reads current generation, increments, rebuilds with new generation, then pushes. Admin can override with `--generation N`. Controller strictly rejects generations ≤ current (monotonic enforcement). | ✅ Decided: xceadmin-managed increment, controller-enforced monotonic |
| 3 | **candidates field** | Controller peer list (`List(MasterCandidate)`). Currently passed as CLI args to `deploy-ha.sh`, not in YAML. xceadmin needs peer info for config build | (a) `--peers localhost:9991,localhost:9992,localhost:9993` CLI flag, (b) read from `hosts.yaml` host list, (c) separate `peers.yaml` |
| 4 | **YAML key ≠ capnp field name** | YAML `groupname` → capnp `GroupConfig.name`, YAML `exclude_members` → capnp `exmembers`. Every field needs explicit mapping definition | Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible |
| 5 | **Nested struct defaults** | `QueueConf` has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults | Use controller's `get_default_conf()` values from `myconf.c`; document which fields are operator-facing vs system-defaulted |

### Important (resolve during design)

| # | Question | Why it matters | Candidates |
|---|----------|---------------|------------|
| 6 | **Wire framing format** | Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject | Document in design.md; mock controller test catches framing errors |
| 7 | **TLS cert discovery** | xceadmin needs CA cert to verify controller TLS. Dev: `ha/data/xce.crt`. Production: configurable path | (a) `--ca-cert` flag, (b) `XCECERT` env var, (c) default `ha/data/xce.crt` for dev, require explicit for non-localhost |
| 8 | **capnp schema compilation** | `capnp compile -o python controller.capnp` → `controller_capnp.py`. When and where? | (a) build time in xce-build.sh (committed output), (b) runtime in xceadmin startup (needs capnp binary), (c) nix derivation that produces the .py file |
| 9 | **Python package layout** | Where does code live? Affects pip install, import paths, xce-build.sh | (a) `src/xceadmin/` alongside C code, (b) `tools/xceadmin/` with deploy-ha.sh, (c) top-level `xceadmin/` |
| 10 | **existing cluster.bin provenance** | `ha/data/cluster.bin` is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it | Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs |

## Configuration Lifecycle & Rollback (Future Roadmap)

### Source of truth: Git-managed YAML

YAML config files (`configs/*.yaml`) are the authoritative source of truth. Operators manage them in git — version control, code review, audit trail are all free. `cluster.bin` is a **derived artifact** (like a compiled binary), never committed to git.

```
git tag v1.2.0  →  YAML at v1.2.0  →  xceadmin config build  →  cluster.bin (deterministic)
                                                                         │
                                                               xceadmin config push
                                                                         │
                                                                         ▼
                                                                  controller cluster
```

**Deterministic builds**: same YAML + same generation value + same xceadmin version → byte-identical `cluster.bin`. `configGeneration` defaults to 0 in offline builds (`xceadmin config build`) and is auto-incremented by `xceadmin config push` (read current from controller, +1). Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries.

### What git gives us (free)

| Capability | How |
|-----------|-----|
| Version history | `git log configs/` |
| Diff between versions | `git diff v1.2.0 v1.3.0 -- configs/` |
| Rollback source | `git checkout v1.2.0 -- configs/ && xceadmin config build` |
| Audit trail | `git blame configs/queues.yaml` |
| Code review | PR against `configs/` branch |

### What still needs building

Git manages the **source**. These parts need implementation:

| # | Capability | What it does | Owned by |
|---|-----------|-------------|----------|
| R1 | **Binary diff** | `xceadmin config diff cluster-v1.bin cluster-v2.bin` — human-readable diff of two built config binaries (decode both → compare field-by-field → pretty-print changes). Useful for pre-push review: "what will this config change actually do?" | admin-config-tool |
| R2 | **Dry-run push** | `xceadmin config push --dry-run cluster.bin` — validate controller will accept the config (SHA check, version check) without committing. Safety net before real push | admin-config-tool |
| R3 | **Config change snapshot** | After config upgrade, take an entity-bearing snapshot of the new configuration state. This anchors the config change in the snapshot chain — without it, rollback has no durable record of what the cluster looked like before/after the change. Controller-side: insert snapshot step into the `OP_upgradeConfig` pipeline (between step 5 "majority ACK" and step 6 "commit WAL_CONFIG_MARKER" in the current handler). | config-upgrade-snapshot |
| R4 | **Controller version history** | Controller tracks last N config versions (tag + SHA + timestamp). `xceadmin config history` queries it. Needed for: "what config is the cluster running?" and "what was running before the last push?" Depends on R3 (snapshot chain provides durable version records). | config-upgrade-snapshot |
| R5 | **Controller-side rollback** | `xceadmin config rollback` (or `xceadmin config push --rollback-to v1.2.0`) — push a previous config version. Controller-side concerns: (a) validate rollback target is safe (no running jobs reference deleted queues/hosts), (b) snapshot before rollback for forward-recovery, (c) bump configGeneration even on rollback (monotonic), (d) restart/activate the rolled-back config | config-upgrade-snapshot |
| R6 | **Rollback safety validation** | Before rolling back, verify: (a) no running jobs use queues being deleted, (b) no active workers on hosts being removed, (c) resource definitions haven't changed in ways that break in-flight allocations. Controller-side check, client-side initiated | config-upgrade-snapshot |
| R7 | **Config activation status** | `xceadmin config status` — show: current config tag, when applied, which nodes have it, any pending restart, snapshot status. Operator visibility into config rollout state | config-upgrade-snapshot |
| R8 | **Config promotion pipeline** | Dev → staging → production config promotion. Same YAML source, different `--peers`/`--cluster-name` flags at build time. Git branches map to environments | admin-config-tool (future) |

### Controller-side pipeline: current vs target

```
CURRENT (config_repl.c — exists today):
  xceadmin push → SHA verify → decode → gen++ → save → replicate → WAL marker → restart signal
                                                                                      ↑
                                                                               step 8 MISSING
TARGET (after config-upgrade-snapshot):
  xceadmin push → SHA verify → decode → gen++ → save → replicate → [SNAPSHOT] → WAL marker → restart signal
                                                                   ↑
                                                            R3: entity-bearing
                                                            snapshot of new config
```

The snapshot insertion point matters: it goes AFTER replication (all standbys have the config) but BEFORE the WAL marker commit (so the snapshot is part of the committed state). This ensures a rollback target always has a consistent snapshot to restore from.

### Rollback workflow (end-to-end vision)

```
# 1. Discover what went wrong
xceadmin config diff cluster-current.bin cluster-suspect.bin
# → "queues.normal.priority changed 10→1, queues.express removed"

# 2. Find the last good version
xceadmin config history
# → v1.2.0 (active), v1.1.0 (previous), v1.0.0 (initial)

# 3. Dry-run the rollback
xceadmin config build --git-ref v1.1.0 -o cluster-v1.1.0.bin
xceadmin config push --dry-run cluster-v1.1.0.bin
# → "OK: controller will accept this config (generation 3→4)"

# 4. Execute rollback
xceadmin config push cluster-v1.1.0.bin
# → "Pushed (tag=v1.1.0, SHA=..., acked=3/3). Controller restarting."

# 5. Verify
xceadmin config status
# → "v1.1.0 active on 3/3 nodes"
```

### Rollback safety: what can go wrong

| Scenario | Risk | Mitigation |
|----------|------|------------|
| Rollback deletes a queue | Running jobs on that queue become unmanageable | R5: controller refuses rollback while jobs reference deleted queues |
| Rollback removes a host | Workers on that host lose config | R5: controller refuses rollback while active workers on removed hosts |
| Rollback changes resource defs | In-flight allocations invalid | Controller fences: drain before rollback, or mark resources "changing" |
| Rollback during network partition | Standbys miss the config push | Majority ACK required (existing OP_upgradeConfig behavior) |
| Rapid rollback→forward→rollback | configGeneration churn, snapshot chain fragmentation | Rate-limit config changes (1 per N seconds); snapshot GC |

### What's in-scope for admin-config-tool vs deferred

| Capability | This spec | Why |
|-----------|-----------|-----|
| YAML source in git | ✅ (documentation) | Free — already how configs/ works |
| Deterministic build | ✅ (core) | Same YAML → same binary (generation not set — controller-owned) |
| Binary diff (R1) | ✅ (nice-to-have) | Pure Python, offline, high operator value |
| Dry-run push (R2) | ✅ (nice-to-have) | Needs `ProbeConfigTag` RPC (already exists in controller) |
| Config change snapshot (R3) | ❌ deferred | Controller-side — insert entity-bearing snapshot into OP_upgradeConfig pipeline; blocked on entity-bearing-snapshots |
| Version history (R4) | ❌ deferred | Controller-side — depends on R3 snapshot chain |
| Rollback (R5) | ❌ deferred | Controller-side — needs snapshot chain + restart |
| Safety validation (R6) | ❌ deferred | Controller-side — needs job/worker state queries |
| Config status (R7) | ❌ deferred | Controller-side — needs cluster-wide state tracking |
| Promotion pipeline (R8) | ❌ deferred | Wraps git branches + xceadmin build — future enhancement |

## Existing Spec Touchpoints

- **Extends**: `config-upgrade-snapshot` (provides the client-side trigger that config-upgrade-snapshot's controller-side path responds to; R3 config-change-snapshot is the missing controller-side step — insert entity-bearing snapshot between replication and WAL marker commit in the `OP_upgradeConfig` pipeline)
- **Adjacent**: `ha-term-safety` (the WAL_CONFIG_MARKER this tool's upload commits anchors on the Raft log via ha-term-safety's term-index infrastructure)
- **Upstream blocker**: `entity-bearing-snapshots` — must land before R3 (config change snapshot) and R4 (version history) can be implemented
- **Blocks**: nothing — xceadmin ships independently; the current controller pipeline (SHA verify → decode → gen++ → save → replicate → WAL marker → restart) already works without the snapshot step
- **Future**: configuration lifecycle (R1–R8 above) — split across admin-config-tool (R1, R2, R8) and config-upgrade-snapshot (R3–R7)