myxce/.kiro/specs/admin-config-tool/research.md

Research & Design Decisions: admin-config-tool

Summary

• Feature: admin-config-tool
• Discovery Scope: Complex Integration — first Python component in a C/Go codebase, first external client of the controller's raw TLS wire protocol, no in-repo precedent for either.
• Key Findings:
  • The controller's OP_upgradeConfig network dispatch path is currently broken for every role (standby rejected by ha_dispatch_gate, primary rejected by an inverted ha_is_primary() check in cluster.c) — a controller-side defect discovered during this design's discovery phase, out of this spec's boundary but blocking real E2E validation of config push.
  • Redirect (redirectHost/redirectPort) is wired only for OPCode_join, never OPCode_upgradeConfig — auto-primary-discovery (Req 3) has no live data to act on for config push until the controller is extended.
  • pycapnp handles controller.capnp's $C.mapname(...) annotations without issue (declared, inert metadata under dynamic reflection) — no custom-annotation workaround needed.
  • src/frontend/xcecli/meson.build's go_build-wrapper + custom_target() pattern is a directly reusable template for src/xceadmin/meson.build, resolving brief.md's open question on Python package location/build integration.

Research Log

TLS setup and certificate validation

• Context: config push needs to establish the same trust model the controller cluster already uses (brief.md Req 2.3-2.4 assumption).
• Sources Consulted: src/lib/netio.c:1208-1243 (create_ssl_context), src/controller/myapp.c:94-109, src/controller/main.c:374-375,90, src/controller/join.c:13-16.
• Findings:
  • create_ssl_context(cert_file, key_file) builds a client context (TLS_client_method()) when no cert/key is given, a server context otherwise. myconf->certfile/keyfile resolve to <certdir>/xce.crt/xce.key.
  • The controller's own outbound client context explicitly sets SSL_CTX_set_verify(client_ssl_ctx, SSL_VERIFY_NONE, NULL) (myapp.c:109).
  • No SSL_CTX_set_verify(..., SSL_VERIFY_PEER, ...), X509_check_host, or SSL_get_peer_certificate call exists anywhere in the codebase. validate_host() (join.c:13-16) is an unrelated stub (return 1;) used only in the cluster-join path, not TLS.
  • No minimum TLS version or cipher-suite pinning exists (SSL_CTX_set_min_proto_version/set_ciphersuites not found).
• Implications: xceadmin should connect with ssl.CERT_NONE/check_hostname=False to match current behavior exactly — this is encryption-only transport security throughout XCE today, not a weakening introduced by this spec. Captured in design.md Security Considerations.

Wire framing byte order

• Context: brief.md asserted "8-byte uint64 size prefix in host byte order" — needed independent confirmation before committing a Python struct.pack format string.
• Sources Consulted: src/include/netio.h:30, src/lib/netio.c:111-199, src/lib/request.c:863-938, src/controller/config_repl.c:1925-1968.
• Findings: io_packet_t.sz (uint64_t) is written/read as raw memory (&pkt->sz, sizeof(...)) directly into SSL_read/SSL_write — no htole64/htobe64/ntohll anywhere in the path. request.c:863-864 states this explicitly in a comment; config_repl.c's xce_probe_config_tag send/recv code (:1925-1968) follows the identical pattern.
• Implications: Python must use struct.pack("=Q", size) (native byte order), not network byte order (">Q"). Documented in design.md xcecommon.wire.

OP_upgradeConfig dispatch path

• Context: Verifying the request would actually reach do_upgrade_config_with_bytes() on a real cluster before designing around it as a working dependency.
• Sources Consulted: src/controller/ha.c:3794-3960 (ha_dispatch_gate), src/controller/cluster.c:238-682 (myreq_handler), src/controller/config_repl.c:1591-1810, src/controller/config_repl.h:347-349, src/controller/testsuite/test_config_repl.c:2042, test_commit_barrier_rollout.c:1484,1531.
• Findings:
  • ha_dispatch_gate()'s CTRL_STANDBY allow-list (ha.c:3926-3951) does not include OPCode_upgradeConfig (only upgradeConfigAck is listed) — standbys are rejected with -XCE_ERR_STANDBY before reaching the opcode switch.
  • For CTRL_PRIMARY with quorum, the gate falls through (ha.c:3880) — primaries pass the gate.
  • But cluster.c:667-679's in-switch handler does if (ha_is_primary(&c->ha)) { /* reject, -EAGAIN */ } else { handle_upgrade_config(...) } — i.e., it only calls the real handler for a non-primary, the opposite of what passed the gate.
  • Net: standby → rejected at the gate; primary → passes the gate, rejected by the inverted in-switch check. No role reaches handle_upgrade_config()/do_upgrade_config_with_bytes() via the network today. That function is exercised only by unit tests calling it directly.
  • config_repl.h:347-349's doc comment ("handle an OP_upgradeConfig request from an admin tool") and the handler's actual logic (WAL marker commit, push_config_to_standbys, restart signal) both imply it should run on the primary — the cluster.c:671 guard looks inverted/stale relative to both the dispatch gate and the handler's own semantics.
• Implications: This is a controller-side defect, not an xceadmin gap — out of this spec's boundary (see design.md Boundary Commitments), but it makes real-cluster E2E testing of config push currently impossible. Testing Strategy in design.md treats the mock-controller test suite as load-bearing, not merely a dev-loop convenience, until this is fixed. Recommend logging this as a bug/blocking item outside this spec's implementation tasks.

Redirect wiring scope

• Context: Requirement 3 (Auto Primary Discovery) assumes redirectHost/redirectPort get populated on a standby's reply to any request, including upgradeConfig.
• Sources Consulted: src/schema/controller.capnp:573-595 (RequestReply), src/controller/join.c:27-31,64,277.
• Findings: fill_redirect_reply() is defined at join.c:27-31 and its only production call site is join.c:277, inside the OPCode_join handling path. No other opcode handler calls it.
• Implications: xceadmin.discovery's redirect-following is implemented generically against the RequestReply schema shape (correct for any opcode that does populate these fields, e.g. join), but will not actually fire for config push until the controller extends fill_redirect_reply()'s call sites to include upgradeConfig. Documented as an Out-of-Boundary item and Revalidation Trigger in design.md.

ProbeConfigTag liveness

• Context: config push --dry-run (Requirement 6) was planned around reusing ProbeConfigTag.
• Sources Consulted: src/controller/config_repl.c:1811-1998 (xce_probe_config_tag + doc comment), src/schema/controller.capnp:821-827, ha.c:7409, ha.c:3828,3860,3889,3911,3943 (dispatch-gate allow-lists).
• Findings: xce_probe_config_tag() is a direct-struct RPC (no Request/Reply wrapper) with only test callers (test_ha_bootstrap.c:259 and a mock in test_ha_bootstrap_support.c). No server-side dispatch case for OPCode_probeConfigTag exists anywhere in cluster.c, despite ha_dispatch_gate listing the opcode as allowed for every role — the allow-list entry is moot because nothing answers it. A stale doc comment at ha.c:7409 claims it's "already called in myapp_run before ha_start_election_async"; no such call exists in myapp.c or anywhere in production code.
• Implications: config push --dry-run's wire client can and should still be implemented against the documented contract (Requirement 6 remains valid future-facing design), but it has no live counterpart to call today. Reflected in design.md as an Out-of-Boundary item; Requirement 6 stays Nice-to-Have/Follow-up as already scoped in requirements.md.

No existing external-client precedent

• Sources Consulted: src/tools/ (xce-build.sh, deploy-ha.sh, go_build, appimage/entry.sh), src/controller/testsuite/test.c (built as xce_test), src/frontend/xcecli/common/request/request.go, repo-wide find . -iname "*.py".
• Findings: src/tools/ has no socket/capnp client. xce_test is a WAL/snapshot file-inspection CLI with zero network code. xcecli (Go) speaks HTTP/JSON to apiserver, never the raw wire protocol. No Python file exists anywhere in the repository.
• Implications: xceadmin is the first client of this wire protocol from outside a C daemon — there is no reference implementation to diff against; correctness rests entirely on independently re-deriving the framing/union contract from the C source (done above), not on copying an existing pattern.

Build tooling precedent for a non-C, non-Meson-native component

• Sources Consulted: src/tools/xce-build.sh:52,63-131, .claude/skills/xce-build/SKILL.md, .claude/skills/xce-deploy-ha/SKILL.md, src/nixSupport/xce.nix:18-70, src/frontend/xcecli/meson.build, src/meson.build:34-35,84-86, src/meson.options:3-6.
• Findings: xce-build.sh dispatches on component name via a case statement; apiserver/xcecli branches additionally shell out to make test/go test. xcecli's meson.build wraps go build via a custom_target() calling a generic tools/go_build script, gated by the component meson option (combo: all|xcecore|xcecli) and conditionally included via if component != 'xcecore': subdir('frontend/xcecli'). xce.nix's nativeBuildInputs has zero Python packages today — confirmed genuinely absent, not just undocumented.
• Implications: src/xceadmin/meson.build should mirror xcecli's pattern exactly (a new tools/py_build wrapper + custom_target() + a component choice or unconditional subdir('xceadmin')), resolving brief.md's open question #9 (Python package layout) in favor of src/xceadmin/, consistent with structure.md's "each major subsystem lives under src/<component>/" convention.

pycapnp schema loading and message encoding

• Context: Confirm controller.capnp's C-specific annotations ($C.mapname(...)) don't break pycapnp's schema compilation, and confirm the right API surface for non-RPC encode/decode.
• Sources Consulted: pycapnp GitHub/README, pycapnp CHANGELOG, Quickstart (2.2.4 docs), capnp.pyx source.
• Findings: $C.mapname(...) is a properly declared annotation (via c.capnp's annotation mapname @0x...), not an unknown construct — pycapnp's DynamicStruct reflection parses it as inert metadata; the only real failure mode is an unresolved /c.capnp import path, handled via capnp.load(path, imports=[schema_dir]). msg.write(f) calls f.fileno() and writes to the raw fd, bypassing any Python-level wrapper — safe for open(path, 'wb'), unsafe for a socket. msg.to_bytes()/Struct.from_bytes() are pure in-memory and the correct choice for the socket path.
• Implications: No custom annotation-stripping preprocessing needed before schema load. xcecommon.wire must use from_bytes()/to_bytes() exclusively (never .write()/.read() on a live socket) — captured as an explicit constraint in design.md.

pycapnp + Python ssl module composition

• Findings: pycapnp's async/KJ-loop machinery (AsyncIoStream, capnp.run()) is scoped to its RPC/vat-network layer (TwoPartyClient/Server) and does not intrude on plain synchronous from_bytes()/to_bytes() usage. ssl.SSLContext().wrap_socket() gives a normal SSLSocket whose .recv()/.sendall() perform the framing at the plaintext layer; pycapnp never touches the socket object directly in this design.
• Implications: Confirms the xcecommon.wire design (manual framing over ssl.SSLSocket, in-memory capnp encode/decode) has no hidden event-loop conflict.

click and pytest conventions

• Findings: Current stable click (8.3.x/8.4.x) idiom for grouped subcommands: @click.group() → @cli.group() → @config.command(), giving xceadmin config build/xceadmin config push naturally. pytest-regressions' file_regression fixture (binary-capable, deliberate --force-regen updates) is the more actively maintained golden-file library versus pytest-golden (YAML-only, no binary support) — evaluated but not adopted (see Design Decisions) in favor of a plain assert produced == expected_bytes fixture comparison, since the byte-equality need here doesn't need the extra dependency's ergonomics.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Layered CLI over shared library (selected)	Thin click CLI (xceadmin) over a protocol/schema/mapping library (xcecommon) with no CLI dependency	Matches project's inward-dependency convention; xcecommon directly reusable by future Python xcecli (brief.md Future Roadmap); each layer independently testable	None material for this scope	—
Single-file script	One xceadmin.py doing everything	Fastest to write initially	Not reusable by future xcecli; violates project's "one domain per file" convention (structure.md); harder to unit-test wire framing in isolation	Rejected
Two separate pip distributions (xceadmin, xcecommon) from day one	Split packaging immediately	Cleaner dependency graph if xcecommon gets a second consumer soon	No second consumer exists yet; premature packaging overhead (Simplification lens)	Deferred, not rejected — revisit when Python xcecli work starts

Design Decisions

Decision: src/xceadmin/ as the package location

• Context: brief.md left this as Open Design Question #9 (candidates: src/xceadmin/, tools/xceadmin/, top-level xceadmin/).
• Alternatives Considered: 1) tools/xceadmin/ alongside deploy-ha.sh. 2) top-level xceadmin/. 3) src/xceadmin/.
• Selected Approach: src/xceadmin/, mirroring src/frontend/'s existing precedent for a non-C, non-Meson-native language living directly under src/ with its own build wrapper.
• Rationale: structure.md's "Organization Philosophy" states "each major subsystem lives under src/<component>/" — xcecli (Go) already establishes that this applies across languages, not just C. Discovery confirmed a directly reusable custom_target() + wrapper-script pattern from xcecli/meson.build.
• Trade-offs: None material — this is the path of least resistance given an existing, working precedent.
• Follow-up: Confirm during implementation whether component meson option needs a new xceadmin choice or can use unconditional subdir() inclusion (both patterns exist for xcecli depending on option value — see src/meson.build:84-86).

Decision: plain byte-equality golden-file tests, not pytest-regressions

• Context: External research surfaced pytest-regressions as the more idiomatic 2025/2026-era binary golden-file library versus brief.md's unspecified "compare binary, verify SHA-256."
• Alternatives Considered: 1) pytest-regressions (file_regression.check(..., binary=True)). 2) pytest-golden (YAML-only, ruled out — no binary support). 3) Plain assert produced_bytes == open(expected_path, 'rb').read() in a parametrized fixture.
• Selected Approach: Option 3 — plain assertion.
• Rationale: pytest-regressions is not yet in this project's nix-pinned dependency list (brief.md's Nix Environment section) and would need independent nixpkgs-availability verification before adoption; the actual requirement (9.1-9.3, byte-for-byte determinism against a committed fixture) doesn't need file_regression's deliberate-update ergonomics for a small, stable fixture set (Simplification lens).
• Trade-offs: Loses --force-regen's one-flag golden-file update convenience; gains zero new dependencies to verify/pin.
• Follow-up: Revisit if the fixture set grows large enough that manual golden-file regeneration becomes a recurring pain point.

Decision: generic (opcode-agnostic) redirect-following and wire exchange

• Context: Discovery found fill_redirect_reply() only wired for OPCode_join, and two distinct message shapes (Request/Reply-wrapped vs. ProbeConfigTagRequest/Reply direct-struct) both need send/receive.
• Alternatives Considered: 1) An upgradeConfig-specific redirect function, built now and generalized later if dry-run/other opcodes need it. 2) One generic follow_redirects() against the RequestReply shape, and one generic send_request()/framing pair reused by both message shapes.
• Selected Approach: Option 2, generic from the start.
• Rationale: Generalization lens — both consumers (upgradeConfig today, join-shaped mocks in tests, potentially dry-run's direct-struct variant) are instances of the same "read RequestReply-shaped fields, frame-and-exchange bytes" problem; building the opcode-specific version first and generalizing later would mean deleting code, not adding it (the redirect logic and framing logic are trivial regardless — no meaningful cost to being generic from day one).
• Trade-offs: None material.
• Follow-up: None.

Decision: push_config()/probe_config() split into two functions, not one boolean-flagged function (Design Review correction, 2026-07-21, second pass)

• Context: A second /kiro-validate-design pass found the first draft's push_config(..., dry_run: bool) -> PushResult implied a dry-run reply has acked_count, which ProbeConfigTagReply (only a configTag field) does not have — the shared return type would misrepresent one of the two call shapes.
• Alternatives Considered: 1) Keep one function, make PushResult fields Optional for the dry-run case. 2) Two functions, two result types.
• Selected Approach: Option 2 — push_config() -> PushResult and probe_config() -> DryRunResult.
• Rationale: The two wire messages and replies share no fields; a boolean flag silently switching which fields are populated is a Type Safety anti-pattern this project's design-principles.md implicitly warns against ("discriminated unions for error handling" — same spirit applies to result shapes).
• Trade-offs: None material — strictly a clarity improvement over the first draft.
• Follow-up: None.

Decision: schema files are packaged as package_data, not referenced from src/schema/ at runtime

• Context: User question after design approval — does pip install . (Req 8.2, non-editable) actually work? It doesn't as originally drafted: non-editable installs copy files into site-packages, severing any fixed relative path back to src/schema/controller.capnp. A __file__-relative walk (e.g. Path(__file__).parents[3] / "schema") only survives editable installs, where __file__ still resolves into the checked-out repo tree.
• Alternatives Considered: 1) Require --schema-path on every invocation (no default) — rejected, bad ergonomics for the common case. 2) __file__-relative resolution — rejected, breaks for pip install .. 3) Package a synced copy of the schema as package_data, resolved via importlib.resources (packaging-aware, correct for both install modes).
• Selected Approach: Option 3. src/schema/{controller,c}.capnp synced (cp) into src/xceadmin/src/xcecommon/schema_files/ by xce-build.sh's xceadmin) case on every build, packaged via pyproject.toml package_data, resolved at runtime via importlib.resources.files("xcecommon") / "schema_files" / "controller.capnp".
• Rationale: importlib.resources is the standard-library-correct way to locate packaged non-code files regardless of install mode (editable, wheel-installed, zipped) — it doesn't care where site-packages physically is. src/schema/ remains the single source of truth (this spec doesn't own it, doesn't hand-edit the copy); the sync step is mechanical and runs every build, so it can't drift stale.
• Trade-offs: One more build-step to keep in sync (mitigated: it's a single cp, not a generation/compilation step); the packaged copy is momentarily stale between a controller.capnp edit and the next xce-build xceadmin run — acceptable, matches how any other build artifact in this repo already works (nothing is watched/hot-reloaded).
• Follow-up: None.

Risks & Mitigations

• Controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) — no role can process a push today. Mitigation: documented as an explicit Out-of-Boundary/Revalidation-Trigger item in design.md; config push implementation and testing proceed against the documented contract via a mock controller; real-cluster E2E is blocked until a separate fix (outside this spec) lands. Recommend raising this as its own tracked defect.
• Redirect not wired for upgradeConfig — auto-discovery (Requirement 3) is untestable end-to-end against a real cluster for this opcode specifically. Mitigation: generic implementation is correct and forward-compatible; mock-controller tests exercise the logic using synthetic redirectHost/redirectPort replies.
• ProbeConfigTag has no server handler — config push --dry-run cannot be validated live. Mitigation: already Nice-to-Have/Follow-up in requirements.md; client implemented against the schema contract, tested via mock only.
• First Python component in the repo — nix/build/CI integration has no precedent to copy exactly (only an analogous one, xcecli's Go integration). Mitigation: xcecli's meson.build/custom_target() pattern is close enough structurally to de-risk this; flagged for extra scrutiny during task review.
• pycapnp C-extension build inside nix — pip install pycapnp normally compiles a C extension against capnproto; getting this wrong would break nix develop. Mitigation: use python3Packages.pycapnp from nixpkgs (pre-built against nix-provided capnproto), not a bare pip install pycapnp, exactly as brief.md's Nix Environment section already specifies.

Decision: xceadmin is not wired into meson (Design Review correction, 2026-07-21)

• Context: /kiro-validate-design review flagged that the initial design draft's src/xceadmin/meson.build (mirroring xcecli's custom_target()) was architecturally unsound — Go's pattern gates on a single static-binary output; pip install -e . mutates a venv/site-packages tree with no single trackable output for meson to gate on.
• Alternatives Considered: 1) Force it through meson anyway via a stamp-file custom_target(). 2) Skip meson entirely, dispatch purely from xce-build.sh's case statement.
• Selected Approach: Option 2.
• Rationale: brief.md's own harness sketch (xce-build.sh's xceadmin) case) already specified direct pip install/pytest dispatch with no meson compile step — the meson integration was an unjustified addition during design drafting, not something the original harness plan called for.
• Trade-offs: xceadmin doesn't participate in meson compile -C build <targets> the way C/Go components do; acceptable since it has no C-extension compile step of its own that meson would need to orchestrate (pycapnp's C extension is pip/nix's concern, not this project's meson graph).
• Follow-up: None — resolved at design time, component meson option is unchanged.

Decision: bundle the 7 JSON Schema files too, not just controller.capnp (2026-07-21, post-approval user question)

• Context: After design approval, user asked whether the JSON schemas used to validate YAML input should also ship with xceadmin, the same way controller.capnp does. Initial answer was "no, they're operator-supplied input per Req 1.1/1.6" — but on reconsideration, that framing didn't hold up: the JSON schemas are the formal spec of what xcecommon.mapping's hardcoded field-extraction logic (groupname, exclude_members, resname, …) actually expects, not independent operator content. An operator-managed copy that drifts from the shipped mapping code risks a silent failure mode — validation passes on YAML that then maps wrong (or rejects YAML that would map fine) — arguably worse than the capnp case (hard crash) because nothing raises. No requirement or brief text ever described org-custom validation schemas as an actual use case; the configs_dir/schema/ layout in brief.md's original sketch was inherited unreflectively from src/configs/'s example layout, not a deliberate feature.
• Alternatives Considered: 1) Keep operator-supplied, configs_dir/schema/ required (original design). 2) Bundle as package_data, same mechanism as controller.capnp, configs_dir/schema/ becomes irrelevant.
• Selected Approach: Option 2 — confirmed with the user directly (AskUserQuestion, "Yes, bundle" selected over "Keep operator-supplied").
• Rationale: Consistency with the capnp decision's own rationale (version-lock a tool-internal contract to the release that consumes it); removes an entire operator-facing requirement (maintaining a schema/ folder they didn't author) for zero documented benefit forfeited.
• Trade-offs: No mechanism left for an org to enforce custom/stricter YAML validation rules — acceptable, since none was ever a stated requirement; can be revisited as a follow-up feature (e.g. a future --extra-schema overlay) if a real need surfaces.
• Process note: this reopened an approved acceptance criterion (Requirement 1.6) — amended requirements.md directly (added 5.5, rewrote 1.6 as "resolved, superseded") rather than looping back through /kiro-spec-requirements, since the user's direct confirmation in this conversation constitutes the approval for this specific amendment.
• Follow-up: None.

References

• pycapnp GitHub — README, quickstart usage patterns
• pycapnp CHANGELOG — confirms synchronous from_bytes/to_bytes/write/read remain available through 2.2.x after RPC-layer sync support removal in 2.0.0b1
• pycapnp Quickstart (2.2.4 docs)
• capnp.pyx source — verified .write()'s fileno() behavior directly from source
• Click — Commands and Groups
• pytest-regressions / GitHub — evaluated, not adopted (see Design Decisions)
• pytest-golden PyPI — ruled out, YAML-only
• src/schema/controller.capnp, src/schema/c.capnp, src/lib/netio.c, src/lib/request.c, src/controller/{ha,cluster,config_repl,join,myapp,main}.c, src/frontend/xcecli/meson.build, src/nixSupport/xce.nix — primary in-repo sources for all findings above

---

Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)

1. Current State Investigation

All three specs touch the same controller-side config surface (controller.capnp's ClusterConfig, config_repl.c's OP_upgradeConfig handler, cluster.c's dispatch gate) from different angles:

Spec	Phase	Scope
clusterconfig-schema-split	implementation-complete	Remove per-instance/dead fields (port, candidates, perf-tuning) from ClusterConfig
admin-config-tool	implementation-complete (blocked_on stale — see §2)	xceadmin CLI: build/push cluster.bin against the controller
config-upgrade-snapshot	tasks-generated, 18/19 actionable tasks [x]	Snapshot the config alongside cluster state at every upgrade boundary; activation fencing

Verified current schema state (src/schema/controller.capnp:712-726): ClusterConfig has exactly 13 fields (@0–@12: clusterName … configGeneration), matching clusterconfig-schema-split's Requirement 3.1 target exactly. MasterCandidate/HashZArrayConfig/ResmgrConfig are gone (Requirements 1.6/2.5). config-upgrade-snapshot's own schema additions (HostAdminState, QueueAdminState, lines 769/774) are separate message types, not fields on ClusterConfig itself — no ordinal collision or renumbering conflict between the two specs' schema work.

Verified dispatch-gate state (src/controller/cluster.c:669-694): the OPCode_upgradeConfig polarity-inversion bug that admin-config-tool's spec.json.blocked_on and tasks.md (line 155) both cite as blocking real-cluster config push E2E validation is already fixed — commit 959b0cda (bug-1468, landed 2026-07-25 as part of config-upgrade-snapshot task 8.6). The fix's own comment in cluster.c explicitly cross-references admin-config-tool: "Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."

2. Gap Found: admin-config-tool's blocked_on is stale documentation, not a live blocker

admin-config-tool/spec.json's blocked_on array still lists:

"controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"

This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, the same controller-side OP_upgradeConfig path this blocker describes has since been exercised live, repeatedly, and successfully by config-upgrade-snapshot's own --config-upgrade soak (.claude/skills/xce-test-ha/run-ha-soak.sh) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and WAL_CONFIG_MARKER commit. The only difference is the client: the soak drives the RPC via xce_test's hand-rolled upgradeconfig command (built for config-upgrade-snapshot task 8.6, since no other live OP_upgradeConfig client existed at the time), not xceadmin's Python config push. But the controller-side code path is identical for both clients — the dispatch gate doesn't distinguish caller identity.

• Gap classification: Missing (stale tracking data, not missing capability). admin-config-tool's actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with xceadmin config push against a live cluster to close it out formally — the blocking condition itself no longer exists.
• Recommendation for design/task phase: Clear admin-config-tool/spec.json's blocked_on entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. clusterconfig-schema-split's already-resolved entries and config-upgrade-snapshot's own dated Implementation Notes). If the E2E task was left unchecked pending this, run xceadmin config push against a real deployed cluster (/xce-deploy-ha) once to get first-hand xceadmin-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).

3. Gap Found (residual, this session): config-upgrade-snapshot task 8.6 — new commit-timeout issue post-bug-1477

config-upgrade-snapshot's own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see .wolf/buglog.json). Re-running --config-upgrade after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all PASS for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (rc=-110) on the primary — a step this soak scenario has never previously reached (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — not yet root-caused, flagged for the next investigation pass. Task 8.6 remains open pending this.

• Gap classification: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
• Ownership: Likely UNCLEAR until investigated — could be ha-commit-durability/ha-term-safety/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a config-upgrade-snapshot-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.

4. No gap found: clusterconfig-schema-split × admin-config-tool integration

admin-config-tool/requirements.md's own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit 519e11c), and the verified current schema state (§1 above) confirms it: port/candidates/perf-tuning fields are genuinely absent from ClusterConfig, so xceadmin's Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.

5. No gap found: clusterconfig-schema-split × config-upgrade-snapshot schema coexistence

Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of controller.capnp (ClusterConfig's field list vs. new standalone HostAdminState/QueueAdminState structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.

Summary Table

#	Finding	Classification	Effort	Risk	Action
1	admin-config-tool blocked_on stale (bug-1468 already fixed)	Missing (tracking, not capability)	S (update spec.json + one live xceadmin config push run)	Low	Clear blocker, optionally run one live E2E confirmation
2	config-upgrade-snapshot task 8.6: new commit-timeout past force-elect step	Unknown (not yet root-caused)	Unknown until investigated	Medium–High (commit/write-quorum path)	Dedicated debug pass before considering 8.6 closeable
3	clusterconfig-schema-split ↔ admin-config-tool	No gap	—	—	None
4	clusterconfig-schema-split ↔ config-upgrade-snapshot schema	No gap	—	—	None

Research Needed (carried forward)

• Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.

Resolved during this pass (no longer open): admin-config-tool/tasks.md was audited directly — all 23 tasks are [x], none gated on the stale blocker as an active checklist item. The E2E gap is purely spec.json.blocked_on + an informational tasks.md note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.