docs(admin-config-tool): clear stale OP_upgradeConfig dispatch blocker (bug-1468 already fixed)
admin-config-tool's spec.json blocked_on and tasks.md both still described
the OP_upgradeConfig dispatch-gate defect as an open blocker. It was fixed
2026-07-25 (bug-1468, commit 959b0cda) by config-upgrade-snapshot task 8.6,
which independently hit the same wall building a live OP_upgradeConfig
sender -- the fix's own comment in cluster.c already cross-references this
spec. The controller-side path has since been exercised live and
successfully many times via config-upgrade-snapshot's --config-upgrade
soak. All 23 admin-config-tool tasks are already [x]; this was pure
tracking-metadata debt, not a reopened implementation gap.
Also appends a cross-spec gap analysis (admin-config-tool x
config-upgrade-snapshot x clusterconfig-schema-split) to all three specs'
research.md, confirming clean schema integration between the latter two
and flagging the one remaining real gap: a new commit-timeout issue in
config-upgrade-snapshot task 8.6's force-elect step, surfaced now that
bug-1477 is fixed, not yet root-caused.
This commit is contained in:
parent
f1e54f3ba0
commit
aecd8766f4
5 changed files with 225 additions and 3 deletions
|
|
@ -156,3 +156,60 @@
|
|||
- [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/overview.html) / [GitHub](https://github.com/ESSS/pytest-regressions) — evaluated, not adopted (see Design Decisions)
|
||||
- [pytest-golden PyPI](https://pypi.org/project/pytest-golden/) — ruled out, YAML-only
|
||||
- `src/schema/controller.capnp`, `src/schema/c.capnp`, `src/lib/netio.c`, `src/lib/request.c`, `src/controller/{ha,cluster,config_repl,join,myapp,main}.c`, `src/frontend/xcecli/meson.build`, `src/nixSupport/xce.nix` — primary in-repo sources for all findings above
|
||||
|
||||
---
|
||||
|
||||
## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)
|
||||
|
||||
### 1. Current State Investigation
|
||||
|
||||
All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles:
|
||||
|
||||
| Spec | Phase | Scope |
|
||||
|---|---|---|
|
||||
| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` |
|
||||
| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller |
|
||||
| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing |
|
||||
|
||||
**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.**
|
||||
|
||||
**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."*
|
||||
|
||||
### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker
|
||||
|
||||
`admin-config-tool/spec.json`'s `blocked_on` array still lists:
|
||||
> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"
|
||||
|
||||
This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity.
|
||||
|
||||
- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists.
|
||||
- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).
|
||||
|
||||
### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477
|
||||
|
||||
`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this.
|
||||
|
||||
- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
|
||||
- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.
|
||||
|
||||
### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration
|
||||
|
||||
`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.
|
||||
|
||||
### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence
|
||||
|
||||
Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.
|
||||
|
||||
### Summary Table
|
||||
|
||||
| # | Finding | Classification | Effort | Risk | Action |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation |
|
||||
| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable |
|
||||
| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None |
|
||||
| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None |
|
||||
|
||||
### Research Needed (carried forward)
|
||||
- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.
|
||||
|
||||
**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@
|
|||
},
|
||||
"ready_for_implementation": true,
|
||||
"blocked_on": [
|
||||
"controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation — see research.md Risks",
|
||||
"RESOLVED 2026-07-22 (bug-1461, commit e53c086): controller.capnp's struct Data shadowed the capnp builtin blob type, miscompiling UpgradeConfigRequest.sha256/.bytes — fixed at the schema root (renamed to XceData), unblocking push_config()'s request construction. The dispatch defect above is separate and still open — push_config() remains mock-tested only, no live E2E."
|
||||
"RESOLVED 2026-07-25 (bug-1468, commit 959b0cda): controller OP_upgradeConfig dispatch defect (cluster.c) -- myreq_handler's dispatch guard was inverted (introduced by commit fd93997f, 2026-05-31), rejecting the only role that could ever legitimately reach it and making handle_upgrade_config() permanently unreachable from any network client. Fixed by config-upgrade-snapshot task 8.6 (which independently hit the same wall building a live OP_upgradeConfig sender). The controller-side path this blocker described has since been exercised live and successfully many times via config-upgrade-snapshot's --config-upgrade soak. push_config() itself (xceadmin's TLS/redirect/framing code) remains mock-tested only -- a live `xceadmin config push` run against a deployed cluster would give first-hand confirmation of the CLI's own wire code, but the blocking condition (dispatch unreachable) no longer exists.",
|
||||
"RESOLVED 2026-07-22 (bug-1461, commit e53c086): controller.capnp's struct Data shadowed the capnp builtin blob type, miscompiling UpgradeConfigRequest.sha256/.bytes — fixed at the schema root (renamed to XceData), unblocking push_config()'s request construction. The dispatch defect above (bug-1468) is a separate issue and is now ALSO resolved (see entry above) — push_config() remains mock-tested only, no live xceadmin-specific E2E run yet, but nothing blocks one."
|
||||
],
|
||||
"gitea": {
|
||||
"milestone_id": 1,
|
||||
|
|
|
|||
|
|
@ -152,7 +152,7 @@
|
|||
- **2.4**: `errors.py`'s exception subclasses (task 1.2) all take `message` as an explicit first constructor argument in addition to their structured fields (e.g. `ConnectionTimeoutError(message, host, port)`, `ReplyDecodeError(message, reason)`) — design.md's abbreviated `Service Interface` type stubs (`class ConnectionTimeoutError(XceAdminError): host: str; port: int`) read as if only the structured fields were needed; always check the real constructor in `errors.py`, not the design doc's shorthand. `xcecommon.wire` is deliberately opcode-agnostic — `send_request`/`send_frame`/`recv_frame` move raw `bytes` only and never import a capnp type; callers own `.to_bytes()`/`.from_bytes()`. Reused `ha/data/xce.crt`/`xce.key` (the repo's existing dev cert pair, same one `deploy-ha.sh` already copies for controller TLS) as the test fixture for both the TLS test server and the client `cert_dir` — no synthetic cert generation needed.
|
||||
- **3.1**: Requirement 1.7's "unrecognized-but-schema-valid field" warning is real and fires on essentially every real build using `src/configs/resources.yaml` — `resources.yaml`'s `type` key is *required* by `resources.schema.json` (`required: [name, type]`) but has no `ClusterConfig` destination (per `xcecommon.mapping`'s documented decision not to guess the `valtype`/`ordtype`/`flags` integer encoding), so every valid resource entry triggers a "field 'type' ... ignored" warning unconditionally, not just for unusual input. This is a disclosed spec tradeoff, not a bug — flagged here so a future pass over `xceadmin.build`'s warning UX (e.g. only warning once per field-name per run instead of once per item) knows the noise is expected, not a regression. `xceadmin.build`'s unmapped-field table (`_UNMAPPED_BUT_VALID_ITEM_KEYS`) was derived by cross-referencing `xcecommon.mapping`'s per-item builder functions against the real `src/configs/schema/*.schema.json` property lists directly (not from prose alone) to avoid drift.
|
||||
- **3.2**: design.md's Dependencies list for `xceadmin.discovery` names only `xcecommon.wire`, but decoding a reply to inspect `RequestReply.redirectHost`/`.redirectPort` genuinely requires `xcecommon.schema` too (verified real field names against `controller.capnp:594-595`, which matched design.md exactly this time — unlike task 2.2's `Reply`/`RequestReply` struct-name slip). Treated as a design-doc omission, not license to hand-roll capnp decoding. `follow_redirects`'s `cert_dir` parameter is accepted for signature parity with design.md but unused internally — all I/O, including TLS setup, is delegated to the caller's `send_fn` closure, which is expected to already carry its own `cert_dir`. A connection timeout to any hop (including a non-initial one reached via redirect) raises `NoPrimaryError` naming that specific hop, not the original target.
|
||||
- **bug-1461 (out-of-band, between 3.2 and 3.3)**: `controller.capnp` had a real, confirmed schema bug — a locally-declared `struct Data` (an `OPCode_get` query-result tagged union) shadowed Cap'n Proto's builtin `Data` blob type for every later `: Data` field in the file, miscompiling `UpgradeConfigRequest.sha256`/`.bytes` (and 5 other fields across `ReplConfigReply`/`ReplConfigPushRequest`/`ReplEnvelope`/`SnapshotTransfer`/`WalCatchupEntry`/`CtrlHelloRequest`, several HA-critical) as a struct-pointer type instead of a byte blob — blocking `push_config()` from constructing a valid request via pycapnp. Fixed at the root (renamed the local struct to `XceData`, `$C.mapname("xce_data_t")` unchanged so the C typedef is stable) across 20 files spanning controller/lib/scheduler/Go — see commit `e53c086` and `.wolf/buglog.json` bug-1461 for the full investigation and fix. This was NOT an admin-config-tool-boundary change; it's a pre-existing, cross-component schema defect that happened to block this task. `push_config()`'s real controller counterpart remains untested live — a *separate*, already-documented dispatch-gate defect (`ha_dispatch_gate()`/`cluster.c`, spec.json `blocked_on`) still blocks that independently of bug-1461.
|
||||
- **bug-1461 (out-of-band, between 3.2 and 3.3)**: `controller.capnp` had a real, confirmed schema bug — a locally-declared `struct Data` (an `OPCode_get` query-result tagged union) shadowed Cap'n Proto's builtin `Data` blob type for every later `: Data` field in the file, miscompiling `UpgradeConfigRequest.sha256`/`.bytes` (and 5 other fields across `ReplConfigReply`/`ReplConfigPushRequest`/`ReplEnvelope`/`SnapshotTransfer`/`WalCatchupEntry`/`CtrlHelloRequest`, several HA-critical) as a struct-pointer type instead of a byte blob — blocking `push_config()` from constructing a valid request via pycapnp. Fixed at the root (renamed the local struct to `XceData`, `$C.mapname("xce_data_t")` unchanged so the C typedef is stable) across 20 files spanning controller/lib/scheduler/Go — see commit `e53c086` and `.wolf/buglog.json` bug-1461 for the full investigation and fix. This was NOT an admin-config-tool-boundary change; it's a pre-existing, cross-component schema defect that happened to block this task. `push_config()`'s real controller counterpart was, separately, blocked by a dispatch-gate defect (`ha_dispatch_gate()`/`cluster.c`) — **RESOLVED 2026-07-25 (bug-1468, commit `959b0cda`)**, fixed by `config-upgrade-snapshot` task 8.6 hitting the same wall building a live `OP_upgradeConfig` sender; that path has since been exercised live and successfully many times via that spec's `--config-upgrade` soak. Nothing blocks a live `push_config()` E2E run anymore, though `xceadmin`'s own TLS/redirect/framing code has not yet had one specifically.
|
||||
- **3.3**: `probe_config()` does NOT go through `xceadmin.discovery.follow_redirects` — `ProbeConfigTagReply` is a direct struct with only `configTag`, no `redirectHost`/`.Port` fields exist on it at all, so there is no redirect signal for `follow_redirects` (which unconditionally decodes replies as `RequestReply`) to read; only `push_config()`'s `Request`/`RequestReply`-wrapped exchange supports redirects. `ProbeConfigTagReply` also carries no accept/reject signal — `DryRunResult.accepted` is a client-side inference (true on any successful round-trip), matching the real C reference client (`config_repl.c`'s `xce_probe_config_tag`, which also always sends an empty `name` and returns only a bare tag). pycapnp union field assignment for Void/Text-typed variants (e.g. `reply.data.nodata = None`, `reply.data.debugmsg = "..."`) must use direct assignment, not `.init(name)` — `.init()` without a size is only valid for struct/list/data-typed variants, confirmed via `KjException: init() without a size is only valid for struct and object fields`.
|
||||
- **4.1**: `xcecommon.wire.connect()` deliberately lets `ConnectionRefusedError`/other bare `OSError`s propagate unwrapped (documented in `wire.py`'s own module docstring) — `xceadmin.cli` is the layer that catches `OSError` and formats it per Requirement 7.1, alongside its `except XceAdminError` dispatch surface; a task 4.1 implementer must not assume `XceAdminError` alone covers every network failure mode. click 8.3.1's `CliRunner` no longer accepts `mix_stderr=`; `result.stdout`/`.stderr` are separated by default, but `CliRunner`'s output capture decodes as text and is **lossy for arbitrary binary** (confirmed: `bytes(range(256))` does not round-trip through the captured `.output` string) — do not assert exact byte content of a `-o -` binary write through `CliRunner`; that's already covered at the `write_output()` unit level (task 3.1's `test_config_build.py`). A CLI test that omits `-o` falls through to the real default (`cluster.bin`, relative to CWD) — always pass an explicit `-o <tmp_path>` in tests that reach the write step, or the test will write a stray artifact into the actual repo working directory (caught via `git status` before commit, not by the test itself).
|
||||
- **4.2**: `deploy-ha.sh` resolves both `xceadmin` (`$SCRIPT_DIR/../xceadmin/.venv/bin/xceadmin`) and the YAML configs source (`$SCRIPT_DIR/../configs`) relative to the script's own location (`$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)`), not the caller's CWD — matches the project's other relative-path scripts and keeps the deploy working regardless of where it's invoked from. `--build-config` is parsed out of `"$@"` before positional-arg assignment so it can appear anywhere in the argument list, not just a fixed position. Verified live against the real `xceadmin` build and `src/configs/`: missing `cluster.bin` auto-builds (8.4), an existing one with no flag is left byte-for-byte untouched — same mtime, same SHA (8.5), and `--build-config` against an existing `cluster.bin` forces a rebuild that reproduces byte-identical content (8.3, confirming 9.1's determinism guarantee holds through this path too).
|
||||
|
|
|
|||
|
|
@ -94,3 +94,60 @@ The task 1.1 implementer deliberately left `HashZArrayConfig` and `ResmgrConfig`
|
|||
Task 5.3's reviewer, verifying the live single-node scenarios, built and ran the actual pre-spec baseline (commit `94f50ce`, HEAD before this spec's first commit) for comparison — and found single-node startup (`xce_controller` with no `--peers`) was **already exiting with `-EINVAL`**, unconditionally, before this spec began. Root cause: `myapp_run()` (`src/controller/myapp.c`) calls `cluster_open_listener()` before the primary config (`cluster_config`) is loaded from disk, so the pre-spec fallback branch (`cluster_config->port`, only reachable when `cluster_config != NULL`) always hit its `else` branch. Logged as `bug-1457` (unrelated to this spec, not fixed as its own change — task 2.1's rewrite incidentally repaired it, verified live in task 5.3).
|
||||
|
||||
**This invalidates part of this spec's own premise, stated in the Introduction and Requirement 1.7's original correction**: "`cluster_open_listener`... has a live fallback to `cluster_config->port`... the sole port-sourcing mechanism for single-node deployments" and "single-node controller startup SHALL continue to work exactly as it does today." In reality, it did not work today (pre-spec) — for a reason entirely unrelated to `ClusterConfig`'s schema. The spec's actual behavior (task 2.1's hardcoded-default fallback, independent of `cluster_config` load state) is still correct and was still the right thing to build — it just fixed a real bug as a side effect, rather than merely "preserving" already-working behavior as originally framed. Not correcting the historical Introduction/Requirement 1.7 text itself (it accurately reflects what was believed and reviewed at design time); this note is the record of what live verification actually found.
|
||||
|
||||
---
|
||||
|
||||
## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)
|
||||
|
||||
### 1. Current State Investigation
|
||||
|
||||
All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles:
|
||||
|
||||
| Spec | Phase | Scope |
|
||||
|---|---|---|
|
||||
| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` |
|
||||
| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller |
|
||||
| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing |
|
||||
|
||||
**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.**
|
||||
|
||||
**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."*
|
||||
|
||||
### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker
|
||||
|
||||
`admin-config-tool/spec.json`'s `blocked_on` array still lists:
|
||||
> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"
|
||||
|
||||
This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity.
|
||||
|
||||
- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists.
|
||||
- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).
|
||||
|
||||
### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477
|
||||
|
||||
`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this.
|
||||
|
||||
- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
|
||||
- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.
|
||||
|
||||
### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration
|
||||
|
||||
`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.
|
||||
|
||||
### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence
|
||||
|
||||
Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.
|
||||
|
||||
### Summary Table
|
||||
|
||||
| # | Finding | Classification | Effort | Risk | Action |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation |
|
||||
| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable |
|
||||
| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None |
|
||||
| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None |
|
||||
|
||||
### Research Needed (carried forward)
|
||||
- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.
|
||||
|
||||
**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.
|
||||
|
|
|
|||
|
|
@ -312,3 +312,111 @@ Three items, only two are genuinely new build work:
|
|||
(bug-970) already wires `WAL_CONFIG_MARKER` into the live upgrade path, so
|
||||
the bug-857 repro (Req 6.2) is triggerable today with existing tooling —
|
||||
no blocker there.
|
||||
|
||||
---
|
||||
|
||||
## Update — 2026-07-23 (design discovery; requirements extended to 7 requirements/36 ACs)
|
||||
|
||||
Requirements went through several rounds of review this session (host/queue admin-state frames, bug-1418 re-verified fixed, bootstrap-vs-upgrade boundary, Design Invariants section, Requirement 7 added for new-mechanism regression coverage). This update captures the DESIGN-level discovery done to write `design.md`, current as of 2026-07-23, extension-scope (light discovery — existing system, no new external dependency).
|
||||
|
||||
### Discovery: exact hook points for the marker → materialization trigger (Req 1.1, 1.5, 1.8)
|
||||
|
||||
**The single unified apply seam is `apply_committed_entry`** (`persist.c:2751-3132`), confirmed as the ONLY live path for entity mutation on every member — primary online-apply (`persist_apply_pump_run` → `persist_apply_pump_kick`, vtask-pool dispatched, `persist.c:3383`), standby normal apply (`ha_standby_advance_committed_and_apply`, heartbeat-learned commit position, `ha.c:2596-2613/2688-2700` — supersedes the older "apply on receipt" bug-1043 fix as of 2026-07-10), and boot/restart replay (`persist.c:3920`). The sibling dispatcher `apply_wal_entry` (`persist.c:2583-2714`) has **zero live callers** (grep-confirmed) — fully superseded, dead code, not a second seam to worry about. `WAL_CONFIG_MARKER` currently falls into `apply_committed_entry`'s generic `default:` no-op (`persist.c:3126-3131`) and `apply_wal_entry`'s equivalent no-op (`persist.c:2697-2707`, irrelevant given no callers). **Design decision: add one new explicit `case WAL_CONFIG_MARKER:` in `apply_committed_entry` only** — it fires identically for primary and standby, satisfying Req 1.2's "every member... via that member's own local materialization once it has applied the marker itself" for free, with no per-role branching needed.
|
||||
|
||||
**Today's restart+replay behavior being replaced** lives in `do_upgrade_config_with_bytes` (`config_repl.c:1591-1773`, step 9, `config_repl.c:1753-1772`): `atomic_store(&c->restart_requested, 1)` + `cbuf_put(&c->wbuf, &stop_val)`. An existing NOTICE comment at `config_repl.c:1737-1751` already names this exact spec by path and describes the target end-state verbatim — a strong signal the upgrade path's own maintainers already anticipated this design. No function-pointer hook exists at that NOTICE site (grepped, none) — it is a plain fall-through, confirming the real trigger point is the `apply_committed_entry` case above, not this step-9 comment site itself. **Once the new case lands, step 9's restart/replay becomes dead code for the upgrade path and should be removed** (not left as a parallel fallback — Design Invariant "never leave two mechanisms doing the same job").
|
||||
|
||||
### Discovery: checkpoint triggering — no forced/immediate entry point exists yet (Req 1.5)
|
||||
|
||||
`persist_checkpoint()` (`persist.c:2262-2321`) is unconditionally synchronous/blocking (fdatasync-backed `write_snapshot`, directory scans) and has exactly one production trigger path today — `schedule_checkpoint()` (`persist.c:2238-2256`), fired only by a WAL-record-count threshold inside `persist_wal_async()` (`persist.c:921-928`). No wall-clock periodic timer exists in the live code (an older design doc's `checkpoint_requested`/`persist_checkpoint_timer_fn` was never implemented — grep-confirmed absent). **No `force_checkpoint()`/`persist_checkpoint_now()` entry point exists** — but a second direct, unconditional call to `persist_checkpoint()` already exists at `ha.c:3308` inside `ha_snapshot_do_send()` ("Req 4.1: create a FRESH checkpoint..."), proving the function itself already tolerates being force-invoked outside the record-count path (it already handles the `-EBUSY` a concurrent scheduled checkpoint can return, via the shared `checkpoint_in_progress` CAS, `persist.h:415`).
|
||||
|
||||
Two existing patterns already solve "trigger without blocking the caller" in this codebase — `delay_task`/delaymgr enqueue (what `schedule_checkpoint` does; the *actual* checkpoint I/O then runs on the delaymgr thread itself, per direct trace of `delay_worker`/`run_delay_tasks`, `delay.c:60`/`cluster.c:988-991` — delaymgr has no thread-pool hand-off) and the dedicated **slow-lane executor** (what `ha_snapshot_do_send` uses, `ha.c` ~6120-6160, specifically because `persist_checkpoint` is blocking/unbounded — this is the pattern `ha-async-io.md:29` documents as the bug-1198 fix for inline-on-iomgr checkpoint I/O). **Design decision: the new `WAL_CONFIG_MARKER` case in `apply_committed_entry` calls `persist_checkpoint()` directly, not via either existing indirection.** This is safe specifically because `apply_committed_entry` (via the apply pump) already runs on its own vtask-pool worker thread, never the iomgr or delaymgr thread — it IS the async-dispatched context, so no further hand-off is needed for THIS call site (unlike `schedule_checkpoint`'s WAL-threshold trigger, which fires from arbitrary vtask handlers appending WAL entries and genuinely needs delaymgr's async return). This gives Req 1.5's "immediate and deterministic" trigger for free — no new delay/lane plumbing required — while the CAS (`checkpoint_in_progress`) already guards against a racing periodic checkpoint.
|
||||
|
||||
**Note for design (not a requirements gap, an implementation-order note):** `scan_latest_committed_config_ref`/`write_snapshot`'s existing `persist_checkpoint` call sequence (`persist.c:2301/2305`) always stamps `checkpoint_lsn = last_applied_lsn` (`persist.c:2292-2295`) — i.e. a marker-triggered forced checkpoint calling `persist_checkpoint()` at the moment `apply_committed_entry` handles the marker will naturally pin `checkpoint_lsn` to the marker's own `(term,lsn)`, satisfying Req 1.1's "materialize at that marker's boundary" without any new LSN-targeting parameter — `persist_checkpoint()` doesn't need to accept an explicit target LSN; it just needs to run at the right moment, which the new apply-time case guarantees by construction.
|
||||
|
||||
### Discovery: the activation gate has a direct, reusable precedent (Req 3)
|
||||
|
||||
`ha_dispatch_gate` (`ha.c:3809-3975`, single call site `cluster.c:276`, gating every wire request — client and peer alike, both HA replication opcodes and apiserver-relayed job/queue/host opcodes) already composes multiple independent boolean fences in sequence for the `CTRL_PRIMARY` role: role check, then a **config-serve-fence check** (`config_serve_fenced(ha->raft_c)`, `ha.c:3827`), then a write-quorum check. This is a **direct structural precedent already in production** for exactly what Req 3 needs — a config-dependent, state-changing-request fence that is orthogonal to role/quorum, with its own opcode allowlist (replication + read-only ops stay admitted while fenced).
|
||||
|
||||
The existing `config_serve_fence` (`ha.h:307-322`, set/cleared by `config_activate_committed` in `config_repl.c:304-377`, hash-verifies committed marker bytes against locally-held config bytes) is the closer of the two fence precedents found (the other being `recovery_unrecoverable`, which only gates *promotion*, not live serving) because it already: (i) blocks client-request serving via `ha_dispatch_gate`, (ii) also blocks promotion (`ha.c:1425-1440`, `ha_config_promotion_fenced()`), and (iii) self-clears via a fully-wired, non-blocking retry loop (`config_schedule_activate_retry` → delaymgr trampoline → vtask does the actual blocking verify I/O, `config_repl.c:415-507`), triggered from the commit-advance callback (`ha.c:2197-2198`) — the exact "loud ERROR once, stay alive, keep retrying, never self-terminate" shape `controller/CLAUDE.md` mandates, already proven in production. **Design decision: add a new atomic fence field on `ha_state_t` (sibling to `config_serve_fence`, e.g. `upgrade_snapshot_fence`), a matching setter/verify function mirroring `config_activate_committed`'s hash-verify-then-clear shape, and one more fence check inside `ha_dispatch_gate`'s `CTRL_PRIMARY` branch** (composed the same way the existing config-serve-fence check already is — not folded into that same field, since the two conditions are logically independent: config-serve-fence is about byte-mismatch, the new fence is about materialization-not-yet-complete for an otherwise-matching config) — reusing `config_schedule_activate_retry`'s exact trampoline shape for the new fence's own retry loop, and `ha_config_promotion_fenced()`'s pattern for gating promotion the same way.
|
||||
|
||||
### Discovery: durable_lsn is NOT advanced by snapshot-install — real but bounded gap (Req 1.2, 5.x)
|
||||
|
||||
Traced definitively (grep + direct read of every function in the call path): `ha_handle_snapshot_chunk` (`ha.c:3014-3171`) → `persist_load_snapshot`/`load_snapshot` (`persist.c:2456-2575`) never touches `p->durable_lsn` anywhere — only `checkpoint_lsn`/`current_lsn`/`cfgref_*` get set. This is asymmetric with the boot-recovery path (`persist_recover`, `persist.c:4338-4353`), which DOES re-derive `durable_lsn` from the recovered WAL tip. **Verdict: real, confirmed gap, but bounded to this node's OWN promotion-eligibility/self-advertised-freshness bookkeeping during a narrow live-catch-up window** (`persist_durable_floor` consumers: pre-promotion gate `ha.c:1318-1352`, promotion truncate-floor clamp `ha.c:1259-1262`, self-advertised candidate position `ha.c:4540-4546`) — the existing fence-and-retry machinery (`ha_arm_promotion_fence_timer`) already handles this safely (conservative refusal to self-promote, never corruption). It is **NOT** a commit-durability/majority-counting safety gap for OTHER members counting a snapshot-caught-up peer, because `majority_candidate_locked` (`commit.c:167-172`) already excludes any standby with `in_catch_up != 0` from majority evaluation, and that flag is cleared only by a real WalAck reaching `catchup_derive_target` (`repl.c:713-739`, via `ha_walack_on_ack`) — a separate, already-safe gate keyed off real acks, not the snapshot event. **Design decision: out of this spec's scope to fix** (it predates this spec, applies to any snapshot-install today including plain recovery-driven catch-up, not upgrade-specific) — but flag it as a design.md Risk/Note so a future spec can pick it up (advancing `durable_lsn` on snapshot-install the same way boot-recovery does would close it cleanly, mirroring `persist.c:4338-4353`'s ASSIGN semantics).
|
||||
|
||||
### Discovery: test tooling — three reusable as-is, two confirmed genuine new-build gaps
|
||||
|
||||
Confirmed via direct read (not just trusting the 2026-07-21 update above, which was itself a research note, not a verified read):
|
||||
- **`snapverify_path`** (`testsuite/snapverify.c`) — confirmed `has_cfgref` is presence-only (`header[16]`), never decodes tag/sha; zero test assertions on it exist today. Needs extending to decode/assert tag+hash for Req 4.2/Req 2 coverage.
|
||||
- **`snapdiff_paths`** (`testsuite/snapdiff.c`) — confirmed reusable as-is for Req 5.1's determinism check (any two paths, no transport-specific assumptions).
|
||||
- **`walcompare`/`walcompare_report`** (`testsuite/walcompare.c`) — confirmed zero references to `cfgref_tag`/`cfgref_sha`/config bytes anywhere in the file; compares only `(lsn, term, pcrc)`. Req 4.2 (active-config identity across members) has no assertion tool — needs either extending `walcompare` or a sibling tool.
|
||||
- **Fault-injection precedent, exact shape confirmed**: `ha_stream_body_read_fault_hook` (`config_repl.c:2104-2106`, wired at `config_repl.c:2183-2193`) and `xce_snapshot_chunk_hook` (`config_repl.c:2847`, wired at `config_repl.c:2937-2949`) are both `#ifdef XCE_TESTING`-gated, `NULL`-by-default, file-scope function-pointer globals, `extern`-declared and set/reset around the driving call in the test file. This is the exact shape the two NEW deterministic hooks (Req 7.2's marker-apply→forced-checkpoint-durable window, Req 7.2/5.4's leadership-change-mid-distribution window) should mirror — a hook inside the new `apply_committed_entry` `WAL_CONFIG_MARKER` case (or wherever the cross-member install-wait loop lives) taking identifying args (marker lsn/term) so a test can force-fail/force-delay at that exact boundary instead of relying on chaotic-soak timing luck.
|
||||
- **Test fixture pattern, exact shape confirmed**: `test_apply_committed_entry.c` (bare in-memory `cluster_t`, no disk, direct `apply_committed_entry()` calls, spy hooks counting `HOOK_AFTER_EVENT` fires) and `test_replay_apply_seam.c` (real disk-backed `persist_t`, two-phase write-then-recover, proving replay routes through the same seam) are the two patterns new host-frame-restore and marker-apply tests should extend — both already have a `tace_encode_worker`/`trr_encode_worker` local-capnp-encode helper (mirroring `persist.c`'s file-static `capnp_encode_worker`) that a new host-admin-frame test can copy the idiom from.
|
||||
|
||||
### Design Synthesis (per `rules/design-synthesis.md`)
|
||||
|
||||
**1. Generalization**: The host-admin-state frame (Req 1.6/1.9/1.10) and the queue-admin-state frame (existing, now migrated to capnp per Req 1.9) are the same underlying shape — `(name, status)` with independently-extensible fields. Rather than two hand-written near-duplicate collect/restore function pairs, design one generic `collect_admin_state_frames(table, status_mask)`/`restore_admin_state_frame(table, name, status, create_if_missing_fn)` pair parameterized over which table (`workermgr`/`qmgr`) and which create-if-missing behavior (hosts: `makeWorker`-based placeholder; queues: none, always pre-provisioned) applies — the two concrete restore functions (`restore_host_state`, `restore_queue_state`) become thin instantiations, not independent implementations. This directly follows Design Invariant 1 (classify config- vs join-provisioned) by construction: the generic function takes the classification as a parameter rather than assuming one.
|
||||
|
||||
**2. Build vs. Adopt**: Every major piece here is "adopt existing internal mechanism," not "build new" — the snapshot transport (`snapshot_transfer_s`/`ha_handle_snapshot_chunk`), the checkpoint/CAS machinery (`persist_checkpoint`/`checkpoint_in_progress`), the fence-and-retry pattern (`config_serve_fence`), the placeholder-entity pattern (`lost_and_found`'s `makeWorker()` precedent), and the fault-injection hook shape (`ha_stream_body_read_fault_hook`) are all reused verbatim rather than reimplemented. The only genuinely NEW machinery is: the `WAL_CONFIG_MARKER` case in `apply_committed_entry`, the new fence field + verify function (structurally mirroring `config_serve_fence` but a distinct field), the config-bytes-embed in the snapshot header/entity payload, the `HostAdminState`/`QueueAdminState` capnp structs, and the two new test hooks. No external library or new dependency is involved anywhere in this spec.
|
||||
|
||||
**3. Simplification**: Considered whether the new fence needs to be a SEPARATE atomic from `config_serve_fence` — confirmed yes (the two conditions are logically independent: hash-mismatch vs materialization-incomplete, and Req 3.4 needs to distinguish "keep serving under last-verified config" from "no verified config exists yet," which requires knowing WHICH fence is active). Considered whether `apply_wal_entry` needs a matching `WAL_CONFIG_MARKER` case for symmetry with `apply_committed_entry` — confirmed no, it has zero live callers, touching it would be pure speculative-symmetry work with no runtime effect (Design Invariant would call this exactly the "convenient but unjustified" mistake to avoid).
|
||||
|
||||
### Risks carried into design.md
|
||||
- The narrow window between `schedule_checkpoint`'s CAS release (`checkpoint_task_fn`, `persist.c:2222`) and `persist_checkpoint`'s own re-acquire (`persist.c:2265`) allows a second concurrent trigger to slip in and start its own `persist_checkpoint()` call — pre-existing, not introduced by this spec, but the new marker-triggered call site adds a second caller into this same race window and should be noted.
|
||||
- `durable_lsn` not advancing on snapshot-install (above) — pre-existing, out of scope, flagged for a future spec.
|
||||
|
||||
---
|
||||
|
||||
## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)
|
||||
|
||||
### 1. Current State Investigation
|
||||
|
||||
All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles:
|
||||
|
||||
| Spec | Phase | Scope |
|
||||
|---|---|---|
|
||||
| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` |
|
||||
| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller |
|
||||
| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing |
|
||||
|
||||
**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.**
|
||||
|
||||
**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."*
|
||||
|
||||
### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker
|
||||
|
||||
`admin-config-tool/spec.json`'s `blocked_on` array still lists:
|
||||
> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"
|
||||
|
||||
This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity.
|
||||
|
||||
- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists.
|
||||
- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).
|
||||
|
||||
### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477
|
||||
|
||||
`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this.
|
||||
|
||||
- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
|
||||
- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.
|
||||
|
||||
### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration
|
||||
|
||||
`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.
|
||||
|
||||
### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence
|
||||
|
||||
Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.
|
||||
|
||||
### Summary Table
|
||||
|
||||
| # | Finding | Classification | Effort | Risk | Action |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation |
|
||||
| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable |
|
||||
| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None |
|
||||
| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None |
|
||||
|
||||
### Research Needed (carried forward)
|
||||
- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.
|
||||
|
||||
**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue