docs(admin-config-tool): clear stale OP_upgradeConfig dispatch blocker (bug-1468 already fixed)

admin-config-tool's spec.json blocked_on and tasks.md both still described
the OP_upgradeConfig dispatch-gate defect as an open blocker. It was fixed
2026-07-25 (bug-1468, commit 959b0cda) by config-upgrade-snapshot task 8.6,
which independently hit the same wall building a live OP_upgradeConfig
sender -- the fix's own comment in cluster.c already cross-references this
spec. The controller-side path has since been exercised live and
successfully many times via config-upgrade-snapshot's --config-upgrade
soak. All 23 admin-config-tool tasks are already [x]; this was pure
tracking-metadata debt, not a reopened implementation gap.

Also appends a cross-spec gap analysis (admin-config-tool x
config-upgrade-snapshot x clusterconfig-schema-split) to all three specs'
research.md, confirming clean schema integration between the latter two
and flagging the one remaining real gap: a new commit-timeout issue in
config-upgrade-snapshot task 8.6's force-elect step, surfaced now that
bug-1477 is fixed, not yet root-caused.

.kiro/specs/admin-config-tool/research.md

@@ -156,3 +156,60 @@
 - [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/overview.html) / [GitHub](https://github.com/ESSS/pytest-regressions) — evaluated, not adopted (see Design Decisions)
 - [pytest-golden PyPI](https://pypi.org/project/pytest-golden/) — ruled out, YAML-only
 - `src/schema/controller.capnp`, `src/schema/c.capnp`, `src/lib/netio.c`, `src/lib/request.c`, `src/controller/{ha,cluster,config_repl,join,myapp,main}.c`, `src/frontend/xcecli/meson.build`, `src/nixSupport/xce.nix` — primary in-repo sources for all findings above
+
+---
+
+## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)
+
+### 1. Current State Investigation
+
+All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles:
+
+| Spec | Phase | Scope |
+|---|---|---|
+| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` |
+| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller |
+| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing |
+
+**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.**
+
+**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."*
+
+### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker
+
+`admin-config-tool/spec.json`'s `blocked_on` array still lists:
+> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"
+
+This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity.
+
+- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists.
+- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).
+
+### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477
+
+`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this.
+
+- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
+- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.
+
+### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration
+
+`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.
+
+### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence
+
+Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.
+
+### Summary Table
+
+| # | Finding | Classification | Effort | Risk | Action |
+|---|---|---|---|---|---|
+| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation |
+| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable |
+| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None |
+| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None |
+
+### Research Needed (carried forward)
+- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.
+
+**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.

.kiro/specs/admin-config-tool/spec.json

@@ -20,8 +20,8 @@
   },
   "ready_for_implementation": true,
   "blocked_on": [
-    "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation — see research.md Risks",
-    "RESOLVED 2026-07-22 (bug-1461, commit e53c086): controller.capnp's struct Data shadowed the capnp builtin blob type, miscompiling UpgradeConfigRequest.sha256/.bytes — fixed at the schema root (renamed to XceData), unblocking push_config()'s request construction. The dispatch defect above is separate and still open — push_config() remains mock-tested only, no live E2E."
+    "RESOLVED 2026-07-25 (bug-1468, commit 959b0cda): controller OP_upgradeConfig dispatch defect (cluster.c) -- myreq_handler's dispatch guard was inverted (introduced by commit fd93997f, 2026-05-31), rejecting the only role that could ever legitimately reach it and making handle_upgrade_config() permanently unreachable from any network client. Fixed by config-upgrade-snapshot task 8.6 (which independently hit the same wall building a live OP_upgradeConfig sender). The controller-side path this blocker described has since been exercised live and successfully many times via config-upgrade-snapshot's --config-upgrade soak. push_config() itself (xceadmin's TLS/redirect/framing code) remains mock-tested only -- a live `xceadmin config push` run against a deployed cluster would give first-hand confirmation of the CLI's own wire code, but the blocking condition (dispatch unreachable) no longer exists.",
+    "RESOLVED 2026-07-22 (bug-1461, commit e53c086): controller.capnp's struct Data shadowed the capnp builtin blob type, miscompiling UpgradeConfigRequest.sha256/.bytes — fixed at the schema root (renamed to XceData), unblocking push_config()'s request construction. The dispatch defect above (bug-1468) is a separate issue and is now ALSO resolved (see entry above) — push_config() remains mock-tested only, no live xceadmin-specific E2E run yet, but nothing blocks one."
   ],
   "gitea": {
     "milestone_id": 1,

.kiro/specs/admin-config-tool/tasks.md

@@ -152,7 +152,7 @@
 - **2.4**: `errors.py`'s exception subclasses (task 1.2) all take `message` as an explicit first constructor argument in addition to their structured fields (e.g. `ConnectionTimeoutError(message, host, port)`, `ReplyDecodeError(message, reason)`) — design.md's abbreviated `Service Interface` type stubs (`class ConnectionTimeoutError(XceAdminError): host: str; port: int`) read as if only the structured fields were needed; always check the real constructor in `errors.py`, not the design doc's shorthand. `xcecommon.wire` is deliberately opcode-agnostic — `send_request`/`send_frame`/`recv_frame` move raw `bytes` only and never import a capnp type; callers own `.to_bytes()`/`.from_bytes()`. Reused `ha/data/xce.crt`/`xce.key` (the repo's existing dev cert pair, same one `deploy-ha.sh` already copies for controller TLS) as the test fixture for both the TLS test server and the client `cert_dir` — no synthetic cert generation needed.
 - **3.1**: Requirement 1.7's "unrecognized-but-schema-valid field" warning is real and fires on essentially every real build using `src/configs/resources.yaml` — `resources.yaml`'s `type` key is *required* by `resources.schema.json` (`required: [name, type]`) but has no `ClusterConfig` destination (per `xcecommon.mapping`'s documented decision not to guess the `valtype`/`ordtype`/`flags` integer encoding), so every valid resource entry triggers a "field 'type' ... ignored" warning unconditionally, not just for unusual input. This is a disclosed spec tradeoff, not a bug — flagged here so a future pass over `xceadmin.build`'s warning UX (e.g. only warning once per field-name per run instead of once per item) knows the noise is expected, not a regression. `xceadmin.build`'s unmapped-field table (`_UNMAPPED_BUT_VALID_ITEM_KEYS`) was derived by cross-referencing `xcecommon.mapping`'s per-item builder functions against the real `src/configs/schema/*.schema.json` property lists directly (not from prose alone) to avoid drift.
 - **3.2**: design.md's Dependencies list for `xceadmin.discovery` names only `xcecommon.wire`, but decoding a reply to inspect `RequestReply.redirectHost`/`.redirectPort` genuinely requires `xcecommon.schema` too (verified real field names against `controller.capnp:594-595`, which matched design.md exactly this time — unlike task 2.2's `Reply`/`RequestReply` struct-name slip). Treated as a design-doc omission, not license to hand-roll capnp decoding. `follow_redirects`'s `cert_dir` parameter is accepted for signature parity with design.md but unused internally — all I/O, including TLS setup, is delegated to the caller's `send_fn` closure, which is expected to already carry its own `cert_dir`. A connection timeout to any hop (including a non-initial one reached via redirect) raises `NoPrimaryError` naming that specific hop, not the original target.
-- **bug-1461 (out-of-band, between 3.2 and 3.3)**: `controller.capnp` had a real, confirmed schema bug — a locally-declared `struct Data` (an `OPCode_get` query-result tagged union) shadowed Cap'n Proto's builtin `Data` blob type for every later `: Data` field in the file, miscompiling `UpgradeConfigRequest.sha256`/`.bytes` (and 5 other fields across `ReplConfigReply`/`ReplConfigPushRequest`/`ReplEnvelope`/`SnapshotTransfer`/`WalCatchupEntry`/`CtrlHelloRequest`, several HA-critical) as a struct-pointer type instead of a byte blob — blocking `push_config()` from constructing a valid request via pycapnp. Fixed at the root (renamed the local struct to `XceData`, `$C.mapname("xce_data_t")` unchanged so the C typedef is stable) across 20 files spanning controller/lib/scheduler/Go — see commit `e53c086` and `.wolf/buglog.json` bug-1461 for the full investigation and fix. This was NOT an admin-config-tool-boundary change; it's a pre-existing, cross-component schema defect that happened to block this task. `push_config()`'s real controller counterpart remains untested live — a *separate*, already-documented dispatch-gate defect (`ha_dispatch_gate()`/`cluster.c`, spec.json `blocked_on`) still blocks that independently of bug-1461.
+- **bug-1461 (out-of-band, between 3.2 and 3.3)**: `controller.capnp` had a real, confirmed schema bug — a locally-declared `struct Data` (an `OPCode_get` query-result tagged union) shadowed Cap'n Proto's builtin `Data` blob type for every later `: Data` field in the file, miscompiling `UpgradeConfigRequest.sha256`/`.bytes` (and 5 other fields across `ReplConfigReply`/`ReplConfigPushRequest`/`ReplEnvelope`/`SnapshotTransfer`/`WalCatchupEntry`/`CtrlHelloRequest`, several HA-critical) as a struct-pointer type instead of a byte blob — blocking `push_config()` from constructing a valid request via pycapnp. Fixed at the root (renamed the local struct to `XceData`, `$C.mapname("xce_data_t")` unchanged so the C typedef is stable) across 20 files spanning controller/lib/scheduler/Go — see commit `e53c086` and `.wolf/buglog.json` bug-1461 for the full investigation and fix. This was NOT an admin-config-tool-boundary change; it's a pre-existing, cross-component schema defect that happened to block this task. `push_config()`'s real controller counterpart was, separately, blocked by a dispatch-gate defect (`ha_dispatch_gate()`/`cluster.c`) — **RESOLVED 2026-07-25 (bug-1468, commit `959b0cda`)**, fixed by `config-upgrade-snapshot` task 8.6 hitting the same wall building a live `OP_upgradeConfig` sender; that path has since been exercised live and successfully many times via that spec's `--config-upgrade` soak. Nothing blocks a live `push_config()` E2E run anymore, though `xceadmin`'s own TLS/redirect/framing code has not yet had one specifically.
 - **3.3**: `probe_config()` does NOT go through `xceadmin.discovery.follow_redirects` — `ProbeConfigTagReply` is a direct struct with only `configTag`, no `redirectHost`/`.Port` fields exist on it at all, so there is no redirect signal for `follow_redirects` (which unconditionally decodes replies as `RequestReply`) to read; only `push_config()`'s `Request`/`RequestReply`-wrapped exchange supports redirects. `ProbeConfigTagReply` also carries no accept/reject signal — `DryRunResult.accepted` is a client-side inference (true on any successful round-trip), matching the real C reference client (`config_repl.c`'s `xce_probe_config_tag`, which also always sends an empty `name` and returns only a bare tag). pycapnp union field assignment for Void/Text-typed variants (e.g. `reply.data.nodata = None`, `reply.data.debugmsg = "..."`) must use direct assignment, not `.init(name)` — `.init()` without a size is only valid for struct/list/data-typed variants, confirmed via `KjException: init() without a size is only valid for struct and object fields`.
 - **4.1**: `xcecommon.wire.connect()` deliberately lets `ConnectionRefusedError`/other bare `OSError`s propagate unwrapped (documented in `wire.py`'s own module docstring) — `xceadmin.cli` is the layer that catches `OSError` and formats it per Requirement 7.1, alongside its `except XceAdminError` dispatch surface; a task 4.1 implementer must not assume `XceAdminError` alone covers every network failure mode. click 8.3.1's `CliRunner` no longer accepts `mix_stderr=`; `result.stdout`/`.stderr` are separated by default, but `CliRunner`'s output capture decodes as text and is **lossy for arbitrary binary** (confirmed: `bytes(range(256))` does not round-trip through the captured `.output` string) — do not assert exact byte content of a `-o -` binary write through `CliRunner`; that's already covered at the `write_output()` unit level (task 3.1's `test_config_build.py`). A CLI test that omits `-o` falls through to the real default (`cluster.bin`, relative to CWD) — always pass an explicit `-o <tmp_path>` in tests that reach the write step, or the test will write a stray artifact into the actual repo working directory (caught via `git status` before commit, not by the test itself).
 - **4.2**: `deploy-ha.sh` resolves both `xceadmin` (`$SCRIPT_DIR/../xceadmin/.venv/bin/xceadmin`) and the YAML configs source (`$SCRIPT_DIR/../configs`) relative to the script's own location (`$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)`), not the caller's CWD — matches the project's other relative-path scripts and keeps the deploy working regardless of where it's invoked from. `--build-config` is parsed out of `"$@"` before positional-arg assignment so it can appear anywhere in the argument list, not just a fixed position. Verified live against the real `xceadmin` build and `src/configs/`: missing `cluster.bin` auto-builds (8.4), an existing one with no flag is left byte-for-byte untouched — same mtime, same SHA (8.5), and `--build-config` against an existing `cluster.bin` forces a rebuild that reproduces byte-identical content (8.3, confirming 9.1's determinism guarantee holds through this path too).