myxce/.kiro/specs/admin-config-tool/brief.md

Brief: admin-config-tool

Problem

XCE operators have no tool to build and deploy cluster configuration. Currently:

• Config authoring: YAML config files (configs/hosts.yaml, queues.yaml, resources.yaml, users.yaml, limits.yaml, xce.yaml, scaler.yaml) exist as documentation examples with JSON schemas for validation (configs/schema/*.schema.json), but no tool compiles them into the ClusterConfig Cap'n Proto binary that the controller expects.
• Config deployment: OP_upgradeConfig exists on the controller side (config_repl.c:1591 do_upgrade_config_with_bytes() — receives binary blob, verifies SHA-256, saves, pushes to standbys, awaits majority ACK, signals restart), but no CLI sends it. Today: manual scp + restart.

End users use xcecli (Go/Cobra → REST → apiserver → CGO → controller) for non-admin operations. There is no admin-facing tool.

Who: XCE cluster operators. Pain: manual config management, error-prone YAML→binary translation, no primary auto-discovery.

Current State

What	Where	Status
ClusterConfig capnp struct	src/schema/controller.capnp:694-708	13 fields (@0-@12), wire-ready — down from 21 after clusterconfig-schema-split removed port/candidates/hash-size/xstrings
YAML config examples	configs/{hosts,queues,resources,users,limits,xce,scaler}.yaml	Human-authored, NOT loaded by any code
JSON schemas	configs/schema/*.schema.json	JSON Schema 2020-12
OP_upgradeConfig handler	config_repl.c:1591 do_upgrade_config_with_bytes()	Receives binary, verifies, pushes, awaits majority
UpgradeConfigRequest/Reply	controller.capnp:792-797	Defined in schema
libxce upgrade API	request.c:1046 has extractors only	No send function
Controller wire protocol	TCP+TLS, capnp-encoded Request/Reply, same CA as controller-to-controller	Exists, used by all C daemons

Key gap: YAML → ClusterConfig binary tool does not exist. Upload CLI does not exist.

Desired Outcome

Single xceadmin Python CLI that operators use for cluster configuration management:

# Build: auto-discover YAML files + schemas from standard directory layout
xceadmin config build configs/                    # default output: cluster.bin
xceadmin config build configs/ -o cluster.bin     # explicit output
xceadmin config build configs/ -o -               # stdout (for piping)

# Push: zero-config dev (same as xce_test), single --controller for production
xceadmin config push cluster.bin                       # default: localhost:9991
xceadmin config push --controller ctrl-1:9991 cluster.bin  # production: any one controller

Standard config directory layout (auto-discovered):

configs/
├── hosts.yaml          # optional — missing = empty hosts section
├── queues.yaml         # optional
├── resources.yaml      # optional
├── users.yaml          # optional
├── limits.yaml         # optional
├── xce.yaml            # optional — missing = clusterName/heartbeat/etc. left unset
└── scaler.yaml         # optional

No schema/ subdirectory — RESOLVED (2026-07-21): the 7 JSON schemas are bundled with the xceadmin package itself (xcecommon package_data, synced from src/configs/schema/ at build time), not supplied per configs_dir. This mirrors the controller.capnp bundling decision and for the same reason: the schemas are the formal spec of what xcecommon.mapping's field-extraction code actually expects, so they need to stay version-locked to the xceadmin release doing the mapping, not left for an operator to drift out of sync. src/configs/schema/*.schema.json remains in the repo as the source of truth that gets synced — an operator's configs_dir never needs its own schema/ folder, and one present there is not read.

• config build: Scan directory → validate present YAML files against xceadmin's bundled JSON schemas → map to ClusterConfig capnp struct → serialize to cluster.bin → output SHA-256. Missing files = empty section (not error). No schema directory to require — bundled, not operator-supplied.
• config push: Read cluster.bin → connect to controller (default localhost:9991) → controller reply includes peer list + primary address → follow redirect if needed → send capnp Request{OP_upgradeConfig, UpgradeConfigRequest{sha256, bytes}} → receive Reply → display {versionTag, ackedCount, error}. configGeneration is fully controller-owned: the primary assigns new_gen = current_gen + 1 on receipt and returns it as versionTag; xceadmin never reads, sets, or overrides it — no --generation flag.
• Shared package: xcecommon Python package with capnp schema handling, wire protocol, and config models — reusable by future Python-based tools

Approach

Chosen: Python CLI with single dependency on controller.capnp.

src/schema/controller.capnp    ← single source of truth
        │
        ├──▶ c-capnproto → libxce → controller (C daemon, existing)
        │
        └──▶ pycapnp → xceadmin (Python, new)
                │
                ├── config build: YAML → ClusterConfig → bytes → cluster.bin
                └── config push:  Request{OP_upgradeConfig, ...} → TLS → Reply

• Language: Python. YAML parsing trivial (yaml), JSON Schema validation (jsonschema), capnp encoding (pycapnp), CLI (click or argparse).
• Protocol: TCP+TLS directly to controller (NOT through apiserver). Zero-config dev experience — default localhost:9991, same as xce_test. Single --controller flag for production override.
• Dependencies: controller.capnp schema file only. Zero CGO, zero libxce. Pure Python, open-source ready.
• Auto primary detection: Connect to given controller → controller reply carries peer list + primary address (standby redirect, already in reply schema) → follow redirect to primary → send request. One controller is enough to bootstrap full cluster discovery. Same mechanism xce_test uses
• No CGO: Wire protocol is TLS connect → send one capnp Request → receive one capnp Reply → exit. ~150 lines of Python. No iomgr event loop, no vtask thread pool, no persistent connections — those are for daemons, not one-shot CLI tools.

Architecture Decisions (from 2026-07-06 discovery session)

Decision	Rationale
Python over Go	Future open-source release; shared capnp infra with planned Python xcecli rewrite; pip install no compilation
Pure Python wire over CGO+libxce	Open-source friendly; no C compilation; wire protocol is one-shot send/receive (~150 lines)
Direct controller over apiserver	Admin path needs no RBAC middleware; apiserver may be restarting during config upgrade; controller SSL is same CA
Single controller.capnp dependency	Schema IS the protocol contract; versioned; no extra API surface
Separate xceadmin binary over extending xcecli	xcecli = user tool (HTTP → apiserver); xceadmin = operator tool (TLS → controller); different audiences, different protocols
Auto primary detection	Operator shouldn't need to know which controller is primary; connect to any one, discover peers + primary from reply
Config build offline	YAML→binary is pure data transform; no controller connection needed; cluster.bin can be pre-built, reviewed, version-controlled

xcecli vs xceadmin

	xcecli	xceadmin
Audience	End users	Operators
Protocol	HTTP/HTTPS (REST, JSON)	TCP+TLS (capnp binary)
Target	apiserver	controller (primary)
Auth	JWT token (login → apiserver)	None — same zero-config TLS as xce_test
Auth middleware	Full: WhitelistURL→WhitelistIP→RateLimit→JWT→CORS→RBAC	None — direct TCP+TLS, same trust model as controller peers
Use cases	Submit jobs, list queues, check status	Push config, cluster admin, force snapshot
Language	Go/Cobra (current), Python (future)	Python

Project Decomposition (from 2026-07-06 discovery)

This discovery session (Path E — mixed decomposition) identified 3 workstreams:

#	Workstream	Type	Language	Dependencies
1	admin-config-tool	NEW spec	Python	controller.capnp only
2	config-upgrade-snapshot	EXISTING spec extension	C	entity-bearing-snapshots (blocked upstream)
3	perf-tuning-extraction	DIRECT work	doc	None

Dependency flow:

admin-config-tool (standalone, ships first)
        │
        │ config push triggers controller-side path
        ▼
config-upgrade-snapshot (controller: replicate → snapshot → restart → activate)
        │
        │ blocked on
        ▼
entity-bearing-snapshots (upstream blocker, tracked in roadmap)

Configuration Inclusion Rules

Decided 2026-07-20. Governs ClusterConfig/cluster.bin field classification now and for any new field added in the future — apply both rules to every candidate field; excluding on either is sufficient.

Promoted to project-wide steering: .kiro/steering/config-design.md holds the general rule (reusable for worker/scheduler/apiserver shared-config decisions, not just ClusterConfig). This section keeps the ClusterConfig-specific field classification; see the steering doc for the rule statement and rationale.

1. Independence rule: if a parameter can change without needing to stay consistent with the rest of the cluster's config (i.e., changing it on one node doesn't require every other node to match), keep it OUT of cluster.bin. Example: port — each controller's own listen port is local, other nodes don't need to know or match it via config push.
2. Frequency rule: if a parameter is expected to change often relative to the rest of cluster config, keep it OUT of cluster.bin. Example: candidates — adding/removing a standby is a routine operational action; requiring a config build+push for it would make membership changes as heavyweight as a schema change.
3. Default (both rules fail → include): a parameter that must be identical across every controller instance AND changes rarely belongs IN cluster.bin. Example: clusterName — divergence between nodes is a correctness bug, and it's set once at bootstrap.

Correction (2026-07-20, after investigating clusterconfig-schema-split): the heartbeatInterval/inactiveHeartbeatCount fields actually govern controller→worker liveness (marking a worker unavailable), not the HA/Raft controller-to-controller election heartbeat as originally assumed here — that election heartbeat has no ClusterConfig field at all (100% hardcoded, and per a bootstrap-availability constraint it never can have one — see .kiro/steering/config-design.md Rule 0). The worker-liveness fields actually pass the independence rule (one controller's worker-timeout tolerance doesn't need to match another's) — they stay IN cluster.bin by deliberate operational-uniformity override, not because they fail the rules. Verdict unchanged; reasoning corrected.

Applying the rules (reference table for future field additions):

Field	Independence rule	Frequency rule	Verdict
clusterName	fails (must match cluster-wide)	fails (rarely changes)	IN
heartbeat/inactive-count (worker liveness)	passes (per-controller worker-timeout tolerance, no cross-node dependency)	—	IN (operational-uniformity override — see correction above)
port	passes (per-node, no cross-node dependency)	—	OUT
candidates	passes (adding a standby doesn't require other nodes' config to change)	passes (routine op)	OUT
jobmgr/workermgr/qmgr/resmgr hash size	passes (perf tuning, local effect only)	—	OUT
xstrings hash size	passes (perf tuning, local effect only)	—	OUT
limits	fails (quotas must be enforced identically on every controller)	fails (rarely changes)	IN
fetchJobInterval	passes (per-controller scheduler-loop tuning, no cross-node dependency)	—	IN (operational-uniformity override, bundled into xce.yaml with the other cluster-identity/liveness fields rather than split into a separate file)
netioTimeout	passes (per-controller network tuning, no cross-node dependency)	—	IN (same override, same file — see fetchJobInterval row)
hproviders (scaler providers)	fails (a queue's scaler reference must resolve identically after failover)	fails (rarely changes)	IN
configGeneration	N/A — not operator input; controller-assigned on every push	N/A	OUT (different reason: not a config value at all, it's protocol state)

When a new field is proposed for ClusterConfig in the future, run it through rules 1–2 before adding it to xceadmin's YAML→capnp mapping; document the verdict in this table.

Scope

• In:
  • xceadmin config build: YAML parsing, JSON Schema validation, capnp ClusterConfig serialization, cluster.bin output with SHA-256. configGeneration, port, and the per-instance perf-tuning fields (manager hash-table sizes, xstrings) are left unset — not xceadmin's to set.
  • xceadmin config push: single controller address (default localhost:9991), auto primary detection via peer-list-from-reply, TLS+capnp wire protocol, capnp UpgradeConfigRequest/Reply, result display. configGeneration assignment is entirely controller-side (primary assigns current + 1, returns it as versionTag) — no client probe, increment, or override flag.
  • xceadmin config push --dry-run: (nice-to-have) validate controller will accept the config without committing it
  • xcecommon shared Python package: capnp schema loading, wire protocol (TLS+capnp framing), config model types
  • Build system integration: pip-based install (editable for dev, pinned for release), nix dev shell packages, xce-build.sh component target
  • Skills integration (see Skill Integration section below):
    • xce-build: add xceadmin component target (Python package build + test)
    • xce-deploy-ha: replace manual ha/data/cluster.bin prerequisite with xceadmin config build step; add --build-config flag for on-the-fly YAML→binary generation
    • xce-test-ha: (future) config push validation round in soak loop — deferred until config-upgrade-snapshot lands on controller side
  • Validation: reject configs using reserved name "lost_and_found" for queue/host (defense-in-depth — controller enforces at runtime via FATAL, tool prevents at authoring time)
• Out:
  • Controller-side config replication/snapshot/restart logic (owned by config-upgrade-snapshot)
  • Entity-bearing snapshot payload (owned by entity-bearing-snapshots, upstream blocker)
  • Performance tuning / per-instance parameter extraction (separate direct work item) — port, manager hash-table-size fields, xstrings; these stay local to each controller instance, never in cluster.bin
  • xcecli Python rewrite (future, separate spec)
  • xce-test-ha config push validation round (deferred until config-upgrade-snapshot lands — config push to a controller that can't replicate/snapshot is a no-op from the soak's perspective)

Skill Integration

Three existing skills touch cluster.bin — the binary config blob that xceadmin config build produces. Currently cluster.bin is a hand-crafted opaque artifact with no visible provenance. xceadmin replaces that with a reproducible build step.

xce-build — add xceadmin component

New component target for the Python package:

/xce-build xceadmin          # pip install -e . (editable, for dev)
/xce-build xceadmin release  # pip install . (pinned, for production)

Build path differs from C components: Python, not C. Runs inside nix dev shell (pycapnp needs capnproto C++ runtime from nix). Build output: xceadmin CLI entry point + xcecommon package importable.

This is the blocking integration — other skills can't use xceadmin until this lands.

xce-deploy-ha — generate cluster.bin from YAML

Current flow:

ha/data/cluster.bin  (pre-built, manual, opaque)  →  deploy-ha.sh  →  controller/ctrl-1/conf/

New flow:

configs/*.yaml  →  xceadmin config build  →  cluster.bin  →  deploy-ha.sh  →  ...

Changes to the skill:

What	Before	After
Prerequisites	cluster.bin must exist under ha/data/	cluster.bin built from configs/*.yaml via xceadmin config build
Config build step	None (manual)	xceadmin config build configs/ -o ha/data/cluster.bin (auto, or skip if --use-existing-config)
--build-config flag	N/A	New flag: force rebuild cluster.bin from YAML before deploy

deploy-ha.sh gains an optional precondition: if cluster.bin missing or --build-config passed, run xceadmin config build. Otherwise use existing cluster.bin as-is (backward compatible — operators can still hand-craft for custom configs).

xce-test-ha — config push validation (future)

Deferred until config-upgrade-snapshot lands on controller side. When ready:

• New soak round type: kill primary, push new config via xceadmin config push, verify controller accepted + standbys received
• Coverage: exercises OP_upgradeConfig → replication → majority ACK path under failover
• Verdict: config push failure = FAIL (config consistency is a data-safety property)

Not scoped into admin-config-tool tasks — belongs to config-upgrade-snapshot spec. Listed here for cross-spec visibility.

Integration dependency order

xce-build (+xceadmin)     ← THIS SPEC, blocking
        │
        ▼
xce-deploy-ha             ← THIS SPEC, immediate after xce-build
        │
        ▼
xce-test-ha               ← config-upgrade-snapshot spec (future)
  (config push round)         blocked on entity-bearing-snapshots

Harness Enhancement

The xceadmin tool is Python — first Python component in a C/Go project. This requires harness changes across the development toolchain.

xce-build.sh — new xceadmin component

Add a new case to src/tools/xce-build.sh:

xceadmin)
    echo "=== Building xceadmin Python package ==="
    pip install -e . --quiet
    echo "=== Running xceadmin tests ==="
    python -m pytest tests/ -v
    ;;

Python path — no meson setup. pip install -e . for dev (editable), pip install . for release. Tests via pytest, not meson test. Runs inside nix dev shell (capnproto C++ runtime needed for pycapnp compile). The release profile uses pip install . without -e.

xce-build SKILL.md — add component + usage note

• Usage line: Components: all (default), lib, controller, scheduler, worker, apiserver, xcecli, xceadmin
• Profile notes: xceadmin ignores meson profiles — it always uses pip. The profile arg affects C components only; xceadmin builds regardless of profile.
• Examples: /xce-build xceadmin (editable), /xce-build xceadmin release (pinned)

xce-deploy-ha SKILL.md — config build step

Changes to the skill document:

• Prerequisites: replace "TLS certs and a cluster.bin must exist under ha/data/" with "TLS certs must exist under ha/data/. cluster.bin is built from configs/*.yaml via xceadmin config build (or use existing if --use-existing-config)."
• New --build-config flag: forces rebuild of cluster.bin from YAML before deploy. Without it, uses existing cluster.bin as-is (backward compatible).
• New step (between resolve paths and stop running): optional xceadmin config build configs/ -o ha/data/cluster.bin if --build-config passed or cluster.bin missing.
• Troubleshooting: add "pycapnp import error" → run /xce-build xceadmin first.

settings.local.json — Python permissions

Add Bash permission allowlist entries for xceadmin development:

"Bash(pip install *)",
"Bash(python -m pytest *)",
"Bash(xceadmin config build *)",
"Bash(xceadmin config push *)",
"Bash(ruff format *)",
"Bash(ruff check *)"

PostToolUse hooks — Python formatting + lint

Add a ruff format hook for Python file edits (settings.json):

{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff format --quiet \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}

Add a ruff check hook for lint feedback:

{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == *.py ]] && ruff check --quiet \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}

PreToolUse hook — YAML schema guard (nice-to-have)

Validate config YAML files against JSON schemas before accepting edits. Prevents commits of invalid config:

{
  "matcher": "Write|Edit",
  "hooks": [
    {
      "type": "command",
      "command": "[[ \"$FILE_PATH\" == configs/*.yaml ]] && xceadmin config validate \"$FILE_PATH\" || true",
      "timeout": 5
    }
  ]
}

This requires xceadmin config validate subcommand (nice-to-have: validates a single YAML file against its schema, no binary output). Deferred to a follow-up task after core config build lands.

/xce-admin skill (deferred)

Thin skill wrapper around xceadmin config build/push for agent-driven workflows. Not needed for spec implementation — CLI is sufficient. Deferred until CLI is stable and operator workflows are validated.

Harness change summary

#	What	Where	Priority	Phase
1	xceadmin case in build script	src/tools/xce-build.sh	Blocking	Implementation
2	Component list + pip note	.claude/skills/xce-build/SKILL.md	Blocking	Implementation
3	--build-config + config build step	.claude/skills/xce-deploy-ha/SKILL.md	Blocking	Implementation
4	pip/pytest/xceadmin permissions	.claude/settings.local.json	Blocking	Implementation
5	ruff format hook on .py edits	.claude/settings.json	Nice-to-have	Implementation
6	YAML schema guard on config edits	.claude/settings.json	Nice-to-have	Follow-up
7	/xce-admin skill wrapper	.claude/skills/xce-admin/SKILL.md	Deferred	Post-spec

Boundary Candidates

• xceadmin CLI — operator-facing commands, argument parsing, output formatting
• xcecommon Python package — shared capnp schema loading, wire protocol (TLS+capnp), config model types. Reusable by future Python xcecli.
• config-build module — YAML→ClusterConfig mapping, JSON Schema validation, serialization
• config-push module — primary auto-detection, TLS+capnp send/receive, result parsing

Upstream / Downstream

• Upstream: controller.capnp schema (source of truth for ClusterConfig, Request/Reply, OP codes)
• Downstream: Controller OP_upgradeConfig handler (config_repl.c — already exists)
• Consumed by:
  • config-upgrade-snapshot spec — xceadmin is the trigger; controller-side handles replication/snapshot/restart
  • xce-build skill — adds xceadmin component target (blocking integration)
  • xce-deploy-ha skill — xceadmin config build replaces manual cluster.bin prerequisite
  • xce-test-ha skill — (future) config push validation round, deferred until config-upgrade-snapshot lands
• Future consumer: Python xcecli rewrite (imports xcecommon for capnp schema types)

Constraints

• Python ≥ 3.10 (match system Python, avoid bleeding-edge features)
• pycapnp requires capnproto C++ runtime (libcapnp-dev) — separate from project's c-capnproto (C bindings). Both capnp runtimes can coexist; binary format is standardized.
• Zero-config TLS — same dev experience as xce_test: default localhost:9991, no cert flags needed. Controller peer trust model (X.509 CN validation via validate_host() — tracked in roadmap hardening).
• No dependency on a running apiserver — admin tool connects directly to controller. Config deployment works even when apiserver is down or restarting.
• Open-source readiness — pure Python, pip install friendly, no compilation step, clear single-schema dependency.

Nix Environment

xceadmin is the first Python component in the XCE project. The nix dev shell currently provides C/C++ tooling only — zero Python packages. All needed Python packages are available in nixpkgs.

Packages to add

Current src/nixSupport/xce.nix nativeBuildInputs already includes capnproto (line 21) — the C++ runtime that pycapnp wraps. Need to add:

Package	Nix attr	Purpose	Size
Python 3	python3	Interpreter (≥ 3.10)	~50MB
pycapnp	python3Packages.pycapnp	capnp schema → Python types	small
PyYAML	python3Packages.pyyaml	YAML config parsing	small
jsonschema	python3Packages.jsonschema	JSON Schema validation	small
click	python3Packages.click	CLI argument parsing	small
pytest	python3Packages.pytest	Test framework	small
ruff	python3Packages.ruff	Python formatter + linter	~10MB

All confirmed available in nixpkgs (unstable): pycapnp 2.2.3, pyyaml 6.0.3, jsonschema 4.26.0, click 8.3.1, pytest 9.0.3, ruff 0.15.20.

Where to add

Two options:

Option A: Add to xce.nix (recommended). Every dev gets Python toolchain. Simpler — one flake, one shell. Python deps are ~60MB total, negligible vs current C toolchain.

Option B: Separate xceadmin.nix. Lighter shell for C-only devs. More maintenance — two nix files, two shells, cross-component workflows need manual env switching.

Recommend Option A — xceadmin is part of the standard toolchain (like xcecli), not an optional extra. All devs should be able to xceadmin config build without switching shells.

New nativeBuildInputs block (addition to xce.nix)

# Python — for xceadmin (admin config tool)
python3
python3Packages.pycapnp
python3Packages.pyyaml
python3Packages.jsonschema
python3Packages.click
python3Packages.pytest
python3Packages.ruff

pycapnp build note

pycapnp wraps the capnproto C++ library at compile time (pip install compiles a C extension). Nix's python3Packages.pycapnp handles this correctly — it links against nix-provided capnproto. No additional libcapnp-dev or system package needed. This is one of the advantages of using nix pycapnp over pip install pycapnp.

CI implications

• CI already uses nix dev shell — adding packages to xce.nix makes them available in CI automatically
• config build golden-file tests need pycapnp for capnp encode (can't mock without schema)
• config push mock tests need pycapnp + TLS (Python ssl stdlib, no extra dep)

Testing Strategy

config build (offline — pure data transform)

Golden-file tests (primary). Given a configs/ directory, verify output matches expected:

tests/
├── fixtures/
│   ├── minimal/           # hosts.yaml + queues.yaml only
│   │   ├── configs/
│   │   │   ├── hosts.yaml
│   │   │   └── queues.yaml   # no schema/ — validated against xceadmin's bundled schemas
│   │   └── expected.bin   # pre-built golden output
│   ├── full/              # all 7 YAML files
│   └── invalid/           # schema violations → expect error
├── test_config_build.py   # pytest: build → compare binary, verify SHA-256
└── test_schema_validation.py  # pytest: each schema violation case

• Framework: pytest (standard Python, no deps beyond dev)
• Zero external dependencies — no controller, no network
• Pre-build golden files once; commit to repo
• Schema validation: test each error case (missing required field, wrong type, invalid enum, reserved name lost_and_found)

config push (online — needs controller)

Option A: Mock controller (unit/integration). Python TLS server that speaks enough capnp wire protocol to accept OP_upgradeConfig and return canned UpgradeConfigReply.

Option B: Real controller (E2E). Piggyback on existing /xce-test-ha cluster. After cluster is up, xceadmin config push a known config and verify controller accepts it.

• Option A for dev loop + CI (fast, no cluster setup)
• Option B for acceptance (real SSL, real replication, real controller)

Recommended: Option A for spec tasks. Option B deferred to xce-test-ha integration after config-upgrade-snapshot lands.

CI Integration

• config build tests: run in any Python CI (no C deps needed — pycapnp only for encode, can mock)
• config push mock tests: need pycapnp + controller.capnp compiled for Python
• Target: 80%+ coverage (project standard)

Open Design Questions

Issues identified during 2026-07-07 review — need resolution before or during requirements generation.

Critical (resolve before requirements)

#	Question	Why it matters	Candidates
1	YAML→capnp mapping incomplete	RESOLVED — all 13 fields accounted for. ClusterConfig is now 13 fields (@0-@12, post-clusterconfig-schema-split), covered by 7 YAML files: resources→resources.yaml, queues→queues.yaml, ugroups→users.yaml, hgroups+hostConfigs→hosts.yaml, limits→limits.yaml (6 fields, 5 files). clusterName/heartbeatInterval/inactiveHeartbeatCount/fetchJobInterval/netioTimeout→xce.yaml (5 fields, 1 file, Req 1.11). hproviders→scaler.yaml (1 field, 1 file, Req 1.12). configGeneration excluded, controller-assigned (1 field, row 2). 6+5+1+1 = 13.	Closed — no candidates remain
2	configGeneration	Monotonic counter. RESOLVED: fully controller-owned. Controller auto-increments new_gen = current_gen + 1 on every OP_upgradeConfig receipt (already implemented, config_repl.c) and returns it as versionTag. xceadmin never reads, sets, or overrides it — config build leaves the field unset, config push sends the binary as-is, no --generation flag. Same generation as current = no-op is a controller-internal detail, not something xceadmin needs to detect.	✅ Decided: controller-only, no client involvement
3	candidates field	Controller peer list (List(MasterCandidate)). RESOLVED — excluded from configuration (Req 1.10): standby add/remove changes independently and more frequently than the rest of cluster config, so it fails both Configuration Inclusion Rules below. xceadmin never sources it from YAML/CLI, cluster.bin leaves it unset. Open sub-question: how peer membership IS sourced/updated instead (today: CLI args to deploy-ha.sh) — not a cluster.bin field going forward	(a) keep as deploy-ha.sh CLI args (status quo, out of xceadmin scope), (b) separate xceadmin peer add/remove command (future), (c) controller-side dynamic membership (future, own spec)
4	YAML key ≠ capnp field name	YAML groupname → capnp GroupConfig.name, YAML exclude_members → capnp exmembers. Every field needs explicit mapping definition	Write a field-mapping table in design.md; auto-generate from capnp schema annotations if possible
5	Nested struct defaults	QueueConf has 14 fields — YAML sets ~6. Remaining (attrs, status, schedDelay, resreq, preExec, postExec) need defaults	Use controller's get_default_conf() values from myconf.c; document which fields are operator-facing vs system-defaulted

Important (resolve during design)

#	Question	Why it matters	Candidates
6	Wire framing format	Controller expects 8-byte uint64 size prefix (host byte order) + capnp payload. ~5 lines of Python but wrong byte order = silent reject	Document in design.md; mock controller test catches framing errors
7	TLS cert discovery	✅ Decided: reuse server-side cert bundle. xceadmin loads xce.crt/xce.key from ha/data/ by default (same files deploy-ha.sh copies for controller-to-controller TLS), overridable via --cert-dir. No separate CA flag, no client cert issuance — same trust model as xce_test. See Requirement 2.3/2.4.
8	capnp schema compilation	✅ RESOLVED (design phase): no capnp compile -o python codegen step exists — pycapnp loads controller.capnp via runtime dynamic reflection (capnp.load(), DynamicStruct), not static generated code. The only build-time step is a plain file sync (not compilation): src/schema/{controller,c}.capnp and src/configs/schema/*.schema.json are copied into xcecommon's package_data by xce-build.sh's xceadmin) case so both schema types survive pip install . (non-editable), resolved at runtime via importlib.resources. See design.md xcecommon.schema.
9	Python package layout	Where does code live? Affects pip install, import paths, xce-build.sh	(a) src/xceadmin/ alongside C code, (b) tools/xceadmin/ with deploy-ha.sh, (c) top-level xceadmin/
10	existing cluster.bin provenance	ha/data/cluster.bin is 8704 bytes, SHA 4986624f..., built June 21 by unknown method. Can't validate xceadmin output against it	Regenerate from known YAML inputs; use as golden-file test once xceadmin produces identical output for same inputs

Configuration Lifecycle & Rollback (Future Roadmap)

Source of truth: Git-managed YAML

YAML config files (configs/*.yaml) are the authoritative source of truth. Operators manage them in git — version control, code review, audit trail are all free. cluster.bin is a derived artifact (like a compiled binary), never committed to git.

git tag v1.2.0  →  YAML at v1.2.0  →  xceadmin config build  →  cluster.bin (deterministic)
                                                                         │
                                                               xceadmin config push
                                                                         │
                                                                         ▼
                                                                  controller cluster

Deterministic builds: same YAML + same xceadmin version → byte-identical cluster.bin. configGeneration is left unset by xceadmin config build and never touched by xceadmin config push — the primary controller alone assigns it on receipt (current + 1), so it plays no part in build determinism. Any git revision can be rebuilt to the exact binary that was (or would have been) pushed. No need to store historical binaries.

What git gives us (free)

Capability	How
Version history	git log configs/
Diff between versions	git diff v1.2.0 v1.3.0 -- configs/
Rollback source	git checkout v1.2.0 -- configs/ && xceadmin config build
Audit trail	git blame configs/queues.yaml
Code review	PR against configs/ branch

What still needs building

Git manages the source. These parts need implementation:

#	Capability	What it does	Owned by
R1	Binary diff	xceadmin config diff cluster-v1.bin cluster-v2.bin — human-readable diff of two built config binaries (decode both → compare field-by-field → pretty-print changes). Useful for pre-push review: "what will this config change actually do?"	deferred (2026-07-21) — no spec owner yet
R2	Dry-run push	xceadmin config push --dry-run cluster.bin — validate controller will accept the config (SHA check, version check) without committing. Safety net before real push	admin-config-tool
R3	Config change snapshot	After config upgrade, take an entity-bearing snapshot of the new configuration state. This anchors the config change in the snapshot chain — without it, rollback has no durable record of what the cluster looked like before/after the change. Controller-side: insert snapshot step into the OP_upgradeConfig pipeline (between step 5 "majority ACK" and step 6 "commit WAL_CONFIG_MARKER" in the current handler).	config-upgrade-snapshot
R4	Controller version history	Controller tracks last N config versions (tag + SHA + timestamp). xceadmin config history queries it. Needed for: "what config is the cluster running?" and "what was running before the last push?" Depends on R3 (snapshot chain provides durable version records).	config-upgrade-snapshot
R5	Controller-side rollback	xceadmin config rollback (or xceadmin config push --rollback-to v1.2.0) — push a previous config version. Controller-side concerns: (a) validate rollback target is safe (no running jobs reference deleted queues/hosts), (b) snapshot before rollback for forward-recovery, (c) bump configGeneration even on rollback (monotonic), (d) restart/activate the rolled-back config	config-upgrade-snapshot
R6	Rollback safety validation	Before rolling back, verify: (a) no running jobs use queues being deleted, (b) no active workers on hosts being removed, (c) resource definitions haven't changed in ways that break in-flight allocations. Controller-side check, client-side initiated	config-upgrade-snapshot
R7	Config activation status	xceadmin config status — show: current config tag, when applied, which nodes have it, any pending restart, snapshot status. Operator visibility into config rollout state	config-upgrade-snapshot
R8	Config promotion pipeline	Dev → staging → production config promotion. Same YAML source, different --peers/--cluster-name flags at build time. Git branches map to environments	admin-config-tool (future)

Controller-side pipeline: current vs target

CURRENT (config_repl.c — exists today):
  xceadmin push → SHA verify → decode → gen++ → save → replicate → WAL marker → restart signal
                                                                                      ↑
                                                                               step 8 MISSING
TARGET (after config-upgrade-snapshot):
  xceadmin push → SHA verify → decode → gen++ → save → replicate → [SNAPSHOT] → WAL marker → restart signal
                                                                   ↑
                                                            R3: entity-bearing
                                                            snapshot of new config

The snapshot insertion point matters: it goes AFTER replication (all standbys have the config) but BEFORE the WAL marker commit (so the snapshot is part of the committed state). This ensures a rollback target always has a consistent snapshot to restore from.

Rollback workflow (end-to-end vision)

# 1. Discover what went wrong
xceadmin config diff cluster-current.bin cluster-suspect.bin
# → "queues.normal.priority changed 10→1, queues.express removed"

# 2. Find the last good version
xceadmin config history
# → v1.2.0 (active), v1.1.0 (previous), v1.0.0 (initial)

# 3. Dry-run the rollback
xceadmin config build --git-ref v1.1.0 -o cluster-v1.1.0.bin
xceadmin config push --dry-run cluster-v1.1.0.bin
# → "OK: controller will accept this config (generation 3→4)"

# 4. Execute rollback
xceadmin config push cluster-v1.1.0.bin
# → "Pushed (tag=v1.1.0, SHA=..., acked=3/3). Controller restarting."

# 5. Verify
xceadmin config status
# → "v1.1.0 active on 3/3 nodes"

Rollback safety: what can go wrong

Scenario	Risk	Mitigation
Rollback deletes a queue	Running jobs on that queue become unmanageable	R5: controller refuses rollback while jobs reference deleted queues
Rollback removes a host	Workers on that host lose config	R5: controller refuses rollback while active workers on removed hosts
Rollback changes resource defs	In-flight allocations invalid	Controller fences: drain before rollback, or mark resources "changing"
Rollback during network partition	Standbys miss the config push	Majority ACK required (existing OP_upgradeConfig behavior)
Rapid rollback→forward→rollback	configGeneration churn, snapshot chain fragmentation	Rate-limit config changes (1 per N seconds); snapshot GC

What's in-scope for admin-config-tool vs deferred

Capability	This spec	Why
YAML source in git	✅ (documentation)	Free — already how configs/ works
Deterministic build	✅ (core)	Same YAML → same binary (generation not set — controller-owned)
Binary diff (R1)	❌ deferred (2026-07-21)	Descoped from admin-config-tool — pure Python/offline, no controller dependency, but no operator has asked for it yet; revisit as follow-up once config push sees real-world usage
Dry-run push (R2)	✅ (nice-to-have)	Needs ProbeConfigTag RPC (already exists in controller)
Config change snapshot (R3)	❌ deferred	Controller-side — insert entity-bearing snapshot into OP_upgradeConfig pipeline; blocked on entity-bearing-snapshots
Version history (R4)	❌ deferred	Controller-side — depends on R3 snapshot chain
Rollback (R5)	❌ deferred	Controller-side — needs snapshot chain + restart
Safety validation (R6)	❌ deferred	Controller-side — needs job/worker state queries
Config status (R7)	❌ deferred	Controller-side — needs cluster-wide state tracking
Promotion pipeline (R8)	❌ deferred	Wraps git branches + xceadmin build — future enhancement

Existing Spec Touchpoints

• Extends: config-upgrade-snapshot (provides the client-side trigger that config-upgrade-snapshot's controller-side path responds to; R3 config-change-snapshot is the missing controller-side step — insert entity-bearing snapshot between replication and WAL marker commit in the OP_upgradeConfig pipeline)
• Adjacent: ha-term-safety (the WAL_CONFIG_MARKER this tool's upload commits anchors on the Raft log via ha-term-safety's term-index infrastructure)
• Upstream blocker: entity-bearing-snapshots — must land before R3 (config change snapshot) and R4 (version history) can be implemented
• Blocks: nothing — xceadmin ships independently; the current controller pipeline (SHA verify → decode → gen++ → save → replicate → WAL marker → restart) already works without the snapshot step
• Future: configuration lifecycle (R1–R8 above) — split across admin-config-tool (R2, R8) and config-upgrade-snapshot (R3–R7); R1 (binary diff) deferred, no spec owner yet