myxce/.kiro/specs/admin-config-tool/research.md

# Research & Design Decisions: admin-config-tool

## Summary
- **Feature**: `admin-config-tool`
- **Discovery Scope**: Complex Integration — first Python component in a C/Go codebase, first external client of the controller's raw TLS wire protocol, no in-repo precedent for either.
- **Key Findings**:
  - The controller's `OP_upgradeConfig` network dispatch path is currently broken for every role (standby rejected by `ha_dispatch_gate`, primary rejected by an inverted `ha_is_primary()` check in `cluster.c`) — a controller-side defect discovered during this design's discovery phase, out of this spec's boundary but blocking real E2E validation of `config push`.
  - Redirect (`redirectHost`/`redirectPort`) is wired only for `OPCode_join`, never `OPCode_upgradeConfig` — auto-primary-discovery (Req 3) has no live data to act on for config push until the controller is extended.
  - `pycapnp` handles `controller.capnp`'s `$C.mapname(...)` annotations without issue (declared, inert metadata under dynamic reflection) — no custom-annotation workaround needed.
  - `src/frontend/xcecli/meson.build`'s `go_build`-wrapper + `custom_target()` pattern is a directly reusable template for `src/xceadmin/meson.build`, resolving brief.md's open question on Python package location/build integration.

## Research Log

### TLS setup and certificate validation
- **Context**: `config push` needs to establish the same trust model the controller cluster already uses (brief.md Req 2.3-2.4 assumption).
- **Sources Consulted**: `src/lib/netio.c:1208-1243` (`create_ssl_context`), `src/controller/myapp.c:94-109`, `src/controller/main.c:374-375,90`, `src/controller/join.c:13-16`.
- **Findings**:
  - `create_ssl_context(cert_file, key_file)` builds a client context (`TLS_client_method()`) when no cert/key is given, a server context otherwise. `myconf->certfile`/`keyfile` resolve to `<certdir>/xce.crt`/`xce.key`.
  - The controller's own outbound client context explicitly sets `SSL_CTX_set_verify(client_ssl_ctx, SSL_VERIFY_NONE, NULL)` (`myapp.c:109`).
  - No `SSL_CTX_set_verify(..., SSL_VERIFY_PEER, ...)`, `X509_check_host`, or `SSL_get_peer_certificate` call exists anywhere in the codebase. `validate_host()` (`join.c:13-16`) is an unrelated stub (`return 1;`) used only in the cluster-join path, not TLS.
  - No minimum TLS version or cipher-suite pinning exists (`SSL_CTX_set_min_proto_version`/`set_ciphersuites` not found).
- **Implications**: `xceadmin` should connect with `ssl.CERT_NONE`/`check_hostname=False` to match current behavior exactly — this is encryption-only transport security throughout XCE today, not a weakening introduced by this spec. Captured in design.md Security Considerations.

### Wire framing byte order
- **Context**: brief.md asserted "8-byte uint64 size prefix in host byte order" — needed independent confirmation before committing a Python `struct.pack` format string.
- **Sources Consulted**: `src/include/netio.h:30`, `src/lib/netio.c:111-199`, `src/lib/request.c:863-938`, `src/controller/config_repl.c:1925-1968`.
- **Findings**: `io_packet_t.sz` (`uint64_t`) is written/read as raw memory (`&pkt->sz`, `sizeof(...)`) directly into `SSL_read`/`SSL_write` — no `htole64`/`htobe64`/`ntohll` anywhere in the path. `request.c:863-864` states this explicitly in a comment; `config_repl.c`'s `xce_probe_config_tag` send/recv code (`:1925-1968`) follows the identical pattern.
- **Implications**: Python must use `struct.pack("=Q", size)` (native byte order), not network byte order (`">Q"`). Documented in design.md `xcecommon.wire`.

### OP_upgradeConfig dispatch path
- **Context**: Verifying the request would actually reach `do_upgrade_config_with_bytes()` on a real cluster before designing around it as a working dependency.
- **Sources Consulted**: `src/controller/ha.c:3794-3960` (`ha_dispatch_gate`), `src/controller/cluster.c:238-682` (`myreq_handler`), `src/controller/config_repl.c:1591-1810`, `src/controller/config_repl.h:347-349`, `src/controller/testsuite/test_config_repl.c:2042`, `test_commit_barrier_rollout.c:1484,1531`.
- **Findings**:
  - `ha_dispatch_gate()`'s `CTRL_STANDBY` allow-list (`ha.c:3926-3951`) does not include `OPCode_upgradeConfig` (only `upgradeConfigAck` is listed) — standbys are rejected with `-XCE_ERR_STANDBY` before reaching the opcode switch.
  - For `CTRL_PRIMARY` with quorum, the gate falls through (`ha.c:3880`) — primaries pass the gate.
  - But `cluster.c:667-679`'s in-switch handler does `if (ha_is_primary(&c->ha)) { /* reject, -EAGAIN */ } else { handle_upgrade_config(...) }` — i.e., it only calls the real handler for a **non-primary**, the opposite of what passed the gate.
  - Net: standby → rejected at the gate; primary → passes the gate, rejected by the inverted in-switch check. No role reaches `handle_upgrade_config()`/`do_upgrade_config_with_bytes()` via the network today. That function is exercised only by unit tests calling it directly.
  - `config_repl.h:347-349`'s doc comment ("handle an OP_upgradeConfig request from an admin tool") and the handler's actual logic (WAL marker commit, `push_config_to_standbys`, restart signal) both imply it should run on the **primary** — the `cluster.c:671` guard looks inverted/stale relative to both the dispatch gate and the handler's own semantics.
- **Implications**: This is a controller-side defect, not an xceadmin gap — out of this spec's boundary (see design.md Boundary Commitments), but it makes real-cluster E2E testing of `config push` currently impossible. Testing Strategy in design.md treats the mock-controller test suite as load-bearing, not merely a dev-loop convenience, until this is fixed. Recommend logging this as a bug/blocking item outside this spec's implementation tasks.

### Redirect wiring scope
- **Context**: Requirement 3 (Auto Primary Discovery) assumes `redirectHost`/`redirectPort` get populated on a standby's reply to any request, including `upgradeConfig`.
- **Sources Consulted**: `src/schema/controller.capnp:573-595` (`RequestReply`), `src/controller/join.c:27-31,64,277`.
- **Findings**: `fill_redirect_reply()` is defined at `join.c:27-31` and its only production call site is `join.c:277`, inside the `OPCode_join` handling path. No other opcode handler calls it.
- **Implications**: `xceadmin.discovery`'s redirect-following is implemented generically against the `RequestReply` schema shape (correct for any opcode that does populate these fields, e.g. `join`), but will not actually fire for `config push` until the controller extends `fill_redirect_reply()`'s call sites to include `upgradeConfig`. Documented as an Out-of-Boundary item and Revalidation Trigger in design.md.

### ProbeConfigTag liveness
- **Context**: `config push --dry-run` (Requirement 6) was planned around reusing `ProbeConfigTag`.
- **Sources Consulted**: `src/controller/config_repl.c:1811-1998` (`xce_probe_config_tag` + doc comment), `src/schema/controller.capnp:821-827`, `ha.c:7409`, `ha.c:3828,3860,3889,3911,3943` (dispatch-gate allow-lists).
- **Findings**: `xce_probe_config_tag()` is a direct-struct RPC (no `Request`/`Reply` wrapper) with only test callers (`test_ha_bootstrap.c:259` and a mock in `test_ha_bootstrap_support.c`). No server-side dispatch case for `OPCode_probeConfigTag` exists anywhere in `cluster.c`, despite `ha_dispatch_gate` listing the opcode as allowed for every role — the allow-list entry is moot because nothing answers it. A stale doc comment at `ha.c:7409` claims it's "already called in myapp_run before ha_start_election_async"; no such call exists in `myapp.c` or anywhere in production code.
- **Implications**: `config push --dry-run`'s wire client can and should still be implemented against the documented contract (Requirement 6 remains valid future-facing design), but it has no live counterpart to call today. Reflected in design.md as an Out-of-Boundary item; Requirement 6 stays Nice-to-Have/Follow-up as already scoped in requirements.md.

### No existing external-client precedent
- **Sources Consulted**: `src/tools/` (`xce-build.sh`, `deploy-ha.sh`, `go_build`, `appimage/entry.sh`), `src/controller/testsuite/test.c` (built as `xce_test`), `src/frontend/xcecli/common/request/request.go`, repo-wide `find . -iname "*.py"`.
- **Findings**: `src/tools/` has no socket/capnp client. `xce_test` is a WAL/snapshot file-inspection CLI with zero network code. `xcecli` (Go) speaks HTTP/JSON to `apiserver`, never the raw wire protocol. No Python file exists anywhere in the repository.
- **Implications**: `xceadmin` is the first client of this wire protocol from outside a C daemon — there is no reference implementation to diff against; correctness rests entirely on independently re-deriving the framing/union contract from the C source (done above), not on copying an existing pattern.

### Build tooling precedent for a non-C, non-Meson-native component
- **Sources Consulted**: `src/tools/xce-build.sh:52,63-131`, `.claude/skills/xce-build/SKILL.md`, `.claude/skills/xce-deploy-ha/SKILL.md`, `src/nixSupport/xce.nix:18-70`, `src/frontend/xcecli/meson.build`, `src/meson.build:34-35,84-86`, `src/meson.options:3-6`.
- **Findings**: `xce-build.sh` dispatches on component name via a `case` statement; `apiserver`/`xcecli` branches additionally shell out to `make test`/`go test`. `xcecli`'s `meson.build` wraps `go build` via a `custom_target()` calling a generic `tools/go_build` script, gated by the `component` meson option (`combo`: `all|xcecore|xcecli`) and conditionally included via `if component != 'xcecore': subdir('frontend/xcecli')`. `xce.nix`'s `nativeBuildInputs` has zero Python packages today — confirmed genuinely absent, not just undocumented.
- **Implications**: `src/xceadmin/meson.build` should mirror `xcecli`'s pattern exactly (a new `tools/py_build` wrapper + `custom_target()` + a `component` choice or unconditional `subdir('xceadmin')`), resolving brief.md's open question #9 (Python package layout) in favor of `src/xceadmin/`, consistent with `structure.md`'s "each major subsystem lives under `src/<component>/`" convention.

### pycapnp schema loading and message encoding
- **Context**: Confirm `controller.capnp`'s C-specific annotations (`$C.mapname(...)`) don't break pycapnp's schema compilation, and confirm the right API surface for non-RPC encode/decode.
- **Sources Consulted**: [pycapnp GitHub/README](https://github.com/capnproto/pycapnp), [pycapnp CHANGELOG](https://github.com/capnproto/pycapnp/blob/master/CHANGELOG.md), [Quickstart (2.2.4 docs)](https://capnproto.github.io/pycapnp/quickstart.html), [capnp.pyx source](https://github.com/capnproto/pycapnp/blob/master/capnp/lib/capnp.pyx).
- **Findings**: `$C.mapname(...)` is a properly declared annotation (via `c.capnp`'s `annotation mapname @0x...`), not an unknown construct — pycapnp's `DynamicStruct` reflection parses it as inert metadata; the only real failure mode is an unresolved `/c.capnp` import path, handled via `capnp.load(path, imports=[schema_dir])`. `msg.write(f)` calls `f.fileno()` and writes to the raw fd, bypassing any Python-level wrapper — safe for `open(path, 'wb')`, unsafe for a socket. `msg.to_bytes()`/`Struct.from_bytes()` are pure in-memory and the correct choice for the socket path.
- **Implications**: No custom annotation-stripping preprocessing needed before schema load. `xcecommon.wire` must use `from_bytes()`/`to_bytes()` exclusively (never `.write()`/`.read()` on a live socket) — captured as an explicit constraint in design.md.

### pycapnp + Python ssl module composition
- **Findings**: pycapnp's async/KJ-loop machinery (`AsyncIoStream`, `capnp.run()`) is scoped to its RPC/vat-network layer (`TwoPartyClient/Server`) and does not intrude on plain synchronous `from_bytes()`/`to_bytes()` usage. `ssl.SSLContext().wrap_socket()` gives a normal `SSLSocket` whose `.recv()`/`.sendall()` perform the framing at the plaintext layer; pycapnp never touches the socket object directly in this design.
- **Implications**: Confirms the `xcecommon.wire` design (manual framing over `ssl.SSLSocket`, in-memory capnp encode/decode) has no hidden event-loop conflict.

### click and pytest conventions
- **Findings**: Current stable `click` (8.3.x/8.4.x) idiom for grouped subcommands: `@click.group()` → `@cli.group()` → `@config.command()`, giving `xceadmin config build`/`xceadmin config push` naturally. `pytest-regressions`' `file_regression` fixture (binary-capable, deliberate `--force-regen` updates) is the more actively maintained golden-file library versus `pytest-golden` (YAML-only, no binary support) — evaluated but not adopted (see Design Decisions) in favor of a plain `assert produced == expected_bytes` fixture comparison, since the byte-equality need here doesn't need the extra dependency's ergonomics.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|----------------------|-------|
| Layered CLI over shared library (selected) | Thin `click` CLI (`xceadmin`) over a protocol/schema/mapping library (`xcecommon`) with no CLI dependency | Matches project's inward-dependency convention; `xcecommon` directly reusable by future Python `xcecli` (brief.md Future Roadmap); each layer independently testable | None material for this scope | — |
| Single-file script | One `xceadmin.py` doing everything | Fastest to write initially | Not reusable by future `xcecli`; violates project's "one domain per file" convention (`structure.md`); harder to unit-test wire framing in isolation | Rejected |
| Two separate pip distributions (`xceadmin`, `xcecommon`) from day one | Split packaging immediately | Cleaner dependency graph if `xcecommon` gets a second consumer soon | No second consumer exists yet; premature packaging overhead (Simplification lens) | Deferred, not rejected — revisit when Python `xcecli` work starts |

## Design Decisions

### Decision: `src/xceadmin/` as the package location
- **Context**: brief.md left this as Open Design Question #9 (candidates: `src/xceadmin/`, `tools/xceadmin/`, top-level `xceadmin/`).
- **Alternatives Considered**: 1) `tools/xceadmin/` alongside `deploy-ha.sh`. 2) top-level `xceadmin/`. 3) `src/xceadmin/`.
- **Selected Approach**: `src/xceadmin/`, mirroring `src/frontend/`'s existing precedent for a non-C, non-Meson-native language living directly under `src/` with its own build wrapper.
- **Rationale**: `structure.md`'s "Organization Philosophy" states "each major subsystem lives under `src/<component>/`" — `xcecli` (Go) already establishes that this applies across languages, not just C. Discovery confirmed a directly reusable `custom_target()` + wrapper-script pattern from `xcecli/meson.build`.
- **Trade-offs**: None material — this is the path of least resistance given an existing, working precedent.
- **Follow-up**: Confirm during implementation whether `component` meson option needs a new `xceadmin` choice or can use unconditional `subdir()` inclusion (both patterns exist for `xcecli` depending on option value — see `src/meson.build:84-86`).

### Decision: plain byte-equality golden-file tests, not `pytest-regressions`
- **Context**: External research surfaced `pytest-regressions` as the more idiomatic 2025/2026-era binary golden-file library versus brief.md's unspecified "compare binary, verify SHA-256."
- **Alternatives Considered**: 1) `pytest-regressions` (`file_regression.check(..., binary=True)`). 2) `pytest-golden` (YAML-only, ruled out — no binary support). 3) Plain `assert produced_bytes == open(expected_path, 'rb').read()` in a parametrized fixture.
- **Selected Approach**: Option 3 — plain assertion.
- **Rationale**: `pytest-regressions` is not yet in this project's nix-pinned dependency list (`brief.md`'s Nix Environment section) and would need independent nixpkgs-availability verification before adoption; the actual requirement (9.1-9.3, byte-for-byte determinism against a committed fixture) doesn't need `file_regression`'s deliberate-update ergonomics for a small, stable fixture set (Simplification lens).
- **Trade-offs**: Loses `--force-regen`'s one-flag golden-file update convenience; gains zero new dependencies to verify/pin.
- **Follow-up**: Revisit if the fixture set grows large enough that manual golden-file regeneration becomes a recurring pain point.

### Decision: generic (opcode-agnostic) redirect-following and wire exchange
- **Context**: Discovery found `fill_redirect_reply()` only wired for `OPCode_join`, and two distinct message shapes (`Request`/`Reply`-wrapped vs. `ProbeConfigTagRequest`/`Reply` direct-struct) both need send/receive.
- **Alternatives Considered**: 1) An `upgradeConfig`-specific redirect function, built now and generalized later if `dry-run`/other opcodes need it. 2) One generic `follow_redirects()` against the `RequestReply` shape, and one generic `send_request()`/framing pair reused by both message shapes.
- **Selected Approach**: Option 2, generic from the start.
- **Rationale**: Generalization lens — both consumers (upgradeConfig today, join-shaped mocks in tests, potentially dry-run's direct-struct variant) are instances of the same "read `RequestReply`-shaped fields, frame-and-exchange bytes" problem; building the opcode-specific version first and generalizing later would mean deleting code, not adding it (the redirect logic and framing logic are trivial regardless — no meaningful cost to being generic from day one).
- **Trade-offs**: None material.
- **Follow-up**: None.

### Decision: `push_config()`/`probe_config()` split into two functions, not one boolean-flagged function (Design Review correction, 2026-07-21, second pass)
- **Context**: A second `/kiro-validate-design` pass found the first draft's `push_config(..., dry_run: bool) -> PushResult` implied a dry-run reply has `acked_count`, which `ProbeConfigTagReply` (only a `configTag` field) does not have — the shared return type would misrepresent one of the two call shapes.
- **Alternatives Considered**: 1) Keep one function, make `PushResult` fields `Optional` for the dry-run case. 2) Two functions, two result types.
- **Selected Approach**: Option 2 — `push_config() -> PushResult` and `probe_config() -> DryRunResult`.
- **Rationale**: The two wire messages and replies share no fields; a boolean flag silently switching which fields are populated is a Type Safety anti-pattern this project's `design-principles.md` implicitly warns against ("discriminated unions for error handling" — same spirit applies to result shapes).
- **Trade-offs**: None material — strictly a clarity improvement over the first draft.
- **Follow-up**: None.

### Decision: schema files are packaged as `package_data`, not referenced from `src/schema/` at runtime
- **Context**: User question after design approval — does `pip install .` (Req 8.2, non-editable) actually work? It doesn't as originally drafted: non-editable installs copy files into `site-packages`, severing any fixed relative path back to `src/schema/controller.capnp`. A `__file__`-relative walk (e.g. `Path(__file__).parents[3] / "schema"`) only survives editable installs, where `__file__` still resolves into the checked-out repo tree.
- **Alternatives Considered**: 1) Require `--schema-path` on every invocation (no default) — rejected, bad ergonomics for the common case. 2) `__file__`-relative resolution — rejected, breaks for `pip install .`. 3) Package a synced copy of the schema as `package_data`, resolved via `importlib.resources` (packaging-aware, correct for both install modes).
- **Selected Approach**: Option 3. `src/schema/{controller,c}.capnp` synced (`cp`) into `src/xceadmin/src/xcecommon/schema_files/` by `xce-build.sh`'s `xceadmin)` case on every build, packaged via `pyproject.toml` `package_data`, resolved at runtime via `importlib.resources.files("xcecommon") / "schema_files" / "controller.capnp"`.
- **Rationale**: `importlib.resources` is the standard-library-correct way to locate packaged non-code files regardless of install mode (editable, wheel-installed, zipped) — it doesn't care where `site-packages` physically is. `src/schema/` remains the single source of truth (this spec doesn't own it, doesn't hand-edit the copy); the sync step is mechanical and runs every build, so it can't drift stale.
- **Trade-offs**: One more build-step to keep in sync (mitigated: it's a single `cp`, not a generation/compilation step); the packaged copy is momentarily stale between a `controller.capnp` edit and the next `xce-build xceadmin` run — acceptable, matches how any other build artifact in this repo already works (nothing is watched/hot-reloaded).
- **Follow-up**: None.

## Risks & Mitigations
- **Controller `OP_upgradeConfig` dispatch defect (`cluster.c:667-679` vs `ha.c:3926-3951`)** — no role can process a push today. Mitigation: documented as an explicit Out-of-Boundary/Revalidation-Trigger item in design.md; `config push` implementation and testing proceed against the documented contract via a mock controller; real-cluster E2E is blocked until a separate fix (outside this spec) lands. Recommend raising this as its own tracked defect.
- **Redirect not wired for `upgradeConfig`** — auto-discovery (Requirement 3) is untestable end-to-end against a real cluster for this opcode specifically. Mitigation: generic implementation is correct and forward-compatible; mock-controller tests exercise the logic using synthetic `redirectHost`/`redirectPort` replies.
- **`ProbeConfigTag` has no server handler** — `config push --dry-run` cannot be validated live. Mitigation: already Nice-to-Have/Follow-up in requirements.md; client implemented against the schema contract, tested via mock only.
- **First Python component in the repo** — nix/build/CI integration has no precedent to copy exactly (only an analogous one, `xcecli`'s Go integration). Mitigation: `xcecli`'s `meson.build`/`custom_target()` pattern is close enough structurally to de-risk this; flagged for extra scrutiny during task review.
- **`pycapnp` C-extension build inside nix** — `pip install pycapnp` normally compiles a C extension against `capnproto`; getting this wrong would break `nix develop`. Mitigation: use `python3Packages.pycapnp` from nixpkgs (pre-built against nix-provided `capnproto`), not a bare `pip install pycapnp`, exactly as brief.md's Nix Environment section already specifies.

### Decision: `xceadmin` is not wired into meson (Design Review correction, 2026-07-21)
- **Context**: `/kiro-validate-design` review flagged that the initial design draft's `src/xceadmin/meson.build` (mirroring `xcecli`'s `custom_target()`) was architecturally unsound — Go's pattern gates on a single static-binary output; `pip install -e .` mutates a venv/site-packages tree with no single trackable output for meson to gate on.
- **Alternatives Considered**: 1) Force it through meson anyway via a stamp-file `custom_target()`. 2) Skip meson entirely, dispatch purely from `xce-build.sh`'s case statement.
- **Selected Approach**: Option 2.
- **Rationale**: `brief.md`'s own harness sketch (`xce-build.sh`'s `xceadmin)` case) already specified direct `pip install`/`pytest` dispatch with no `meson compile` step — the meson integration was an unjustified addition during design drafting, not something the original harness plan called for.
- **Trade-offs**: `xceadmin` doesn't participate in `meson compile -C build <targets>` the way C/Go components do; acceptable since it has no C-extension compile step of its own that meson would need to orchestrate (pycapnp's C extension is pip/nix's concern, not this project's meson graph).
- **Follow-up**: None — resolved at design time, `component` meson option is unchanged.

### Decision: bundle the 7 JSON Schema files too, not just `controller.capnp` (2026-07-21, post-approval user question)
- **Context**: After design approval, user asked whether the JSON schemas used to validate YAML input should also ship with `xceadmin`, the same way `controller.capnp` does. Initial answer was "no, they're operator-supplied input per Req 1.1/1.6" — but on reconsideration, that framing didn't hold up: the JSON schemas are the formal spec of what `xcecommon.mapping`'s hardcoded field-extraction logic (`groupname`, `exclude_members`, `resname`, …) actually expects, not independent operator content. An operator-managed copy that drifts from the shipped mapping code risks a *silent* failure mode — validation passes on YAML that then maps wrong (or rejects YAML that would map fine) — arguably worse than the capnp case (hard crash) because nothing raises. No requirement or brief text ever described org-custom validation schemas as an actual use case; the `configs_dir/schema/` layout in `brief.md`'s original sketch was inherited unreflectively from `src/configs/`'s example layout, not a deliberate feature.
- **Alternatives Considered**: 1) Keep operator-supplied, `configs_dir/schema/` required (original design). 2) Bundle as `package_data`, same mechanism as `controller.capnp`, `configs_dir/schema/` becomes irrelevant.
- **Selected Approach**: Option 2 — confirmed with the user directly (`AskUserQuestion`, "Yes, bundle" selected over "Keep operator-supplied").
- **Rationale**: Consistency with the capnp decision's own rationale (version-lock a tool-internal contract to the release that consumes it); removes an entire operator-facing requirement (maintaining a `schema/` folder they didn't author) for zero documented benefit forfeited.
- **Trade-offs**: No mechanism left for an org to enforce custom/stricter YAML validation rules — acceptable, since none was ever a stated requirement; can be revisited as a follow-up feature (e.g. a future `--extra-schema` overlay) if a real need surfaces.
- **Process note**: this reopened an *approved* acceptance criterion (Requirement 1.6) — amended requirements.md directly (added 5.5, rewrote 1.6 as "resolved, superseded") rather than looping back through `/kiro-spec-requirements`, since the user's direct confirmation in this conversation constitutes the approval for this specific amendment.
- **Follow-up**: None.

## References
- [pycapnp GitHub](https://github.com/capnproto/pycapnp) — README, quickstart usage patterns
- [pycapnp CHANGELOG](https://github.com/capnproto/pycapnp/blob/master/CHANGELOG.md) — confirms synchronous `from_bytes`/`to_bytes`/`write`/`read` remain available through 2.2.x after RPC-layer sync support removal in 2.0.0b1
- [pycapnp Quickstart (2.2.4 docs)](https://capnproto.github.io/pycapnp/quickstart.html)
- [capnp.pyx source](https://github.com/capnproto/pycapnp/blob/master/capnp/lib/capnp.pyx) — verified `.write()`'s `fileno()` behavior directly from source
- [Click — Commands and Groups](https://click.palletsprojects.com/en/stable/commands-and-groups/)
- [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/overview.html) / [GitHub](https://github.com/ESSS/pytest-regressions) — evaluated, not adopted (see Design Decisions)
- [pytest-golden PyPI](https://pypi.org/project/pytest-golden/) — ruled out, YAML-only
- `src/schema/controller.capnp`, `src/schema/c.capnp`, `src/lib/netio.c`, `src/lib/request.c`, `src/controller/{ha,cluster,config_repl,join,myapp,main}.c`, `src/frontend/xcecli/meson.build`, `src/nixSupport/xce.nix` — primary in-repo sources for all findings above

---

## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)

### 1. Current State Investigation

All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles:

| Spec | Phase | Scope |
|---|---|---|
| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` |
| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller |
| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing |

**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.**

**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."*

### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker

`admin-config-tool/spec.json`'s `blocked_on` array still lists:
> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"

This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity.

- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists.
- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).

### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477

`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this.

- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.

### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration

`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.

### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence

Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.

### Summary Table

| # | Finding | Classification | Effort | Risk | Action |
|---|---|---|---|---|---|
| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation |
| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable |
| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None |
| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None |

### Research Needed (carried forward)
- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.

**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.