myxce/.kiro/specs/config-upgrade-snapshot/research.md

# Gap Analysis — config-upgrade-snapshot (2026-06-24)

## Scope

Make a configuration upgrade install one identical, **entity-bearing** state snapshot
(with the new config embedded) on every member at the config-change marker's
`(term, lsn)` boundary, replacing today's config-push + restart + per-node WAL
replay (which can materialize divergent state — **bug-857**). Closes bug-857.

> **GATING BLOCKER (hard dependency): `entity-bearing-snapshots`.**
> Today snapshots carry **zero entity state** (an LSN sentinel). This spec
> *consumes* an entity-bearing snapshot it cannot produce. It **cannot reach a
> testable GREEN until `entity-bearing-snapshots` lands or is co-scoped.** Every
> "Missing" below traces back to this one absence. See §"Gating Blocker".

---

## Requirement-to-Asset Map

### R1 — Full snapshot at the config-change boundary

| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Materialize a snapshot at the marker's `(term,lsn)` | `persist_checkpoint` / `write_snapshot` (`persist.c:1247-1291`) writes a checkpoint snapshot at `checkpoint_lsn` | **Missing** — snapshot is a 16-byte sentinel (`"XCESNAP\0"` + LSN + optional 41-byte config-ref tail); carries no entities. No way to pin it to the marker boundary vs. an arbitrary checkpoint LSN. |
| Snapshot pins identical materialized state | dual-zarray COW: `xzarray_freeze`/`xzarray_thaw` (`zarray.c:695-757`), `test_zarray_cow.c` | **Missing (dormant)** — built + unit-tested but **no production callers**: `kvmgr.c:297` `vfreeze=NULL /* wired in P0-2 */`, `vthaw=NULL`. |
| No member re-derives via replay | upgrade path `do_upgrade_config_with_bytes` (`config_repl.c:1495-1677`); NOTICE at step 9 (`config_repl.c:1641-1655`) | **Constraint (current behavior is the bug)** — step 9 sets `restart_requested`; on restart each node replays its own pre-upgrade WAL under the new config (config-dependent via `lost_and_found`). This is exactly what R1.3 forbids. |
| Transfer out-of-band, not in WAL payloads | `ha_handle_snapshot_chunk` (`ha.c:2424-2515`) + `persist_load_snapshot` (`persist.c:1594`); `OPCode_snapshotTransfer` dispatch-gated (`ha.c:2842/2874/2962`) | **Exists (reusable transport)** — chunked, size-checked, atomic `snap.part`→rename, then `persist_load_snapshot`. Reusable as-is; it only ever moves a sentinel today. |

### R2 — Configuration embedded in the upgrade snapshot

| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Embed new config in snapshot (one artifact) | snapshot config-ref tail (`persist.c:1263-1274`, read-back `1581-1590`): `{present, version_tag(8), sha256(32)}` | **Partial / Missing** — only a `{tag, sha}` **reference** is embedded, not the config **bytes** and not the entity state. Need the actual config bytes + entities in one installable artifact. |
| Supersede ha-term-safety R7 config-push for upgrades | R7 out-of-band byte push driven from `config_commit_change` (`config_repl.c:478-510`) | **Missing** — upgrade still uses the R7 interim byte-push path; no branch that delivers config via the snapshot instead. |
| Keep config bytes out of WAL payloads | `WAL_CONFIG_MARKER` is `{version_tag, sha256}` only — `config_marker_encode` (`config_repl.c:40-69`), bytes never in WAL | **Already satisfied** — marker carries no bytes; invariant holds today. |

### R3 — Activation gating

| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Don't serve/activate new config until snapshot installed | install-before-activate already structural in `ha_handle_snapshot_chunk` (rename+load on full receipt) | **Partial** — transport installs before load, but there is **no activation gate** tying "config active" to "upgrade snapshot for this `(term,lsn)` installed." Today activation = restart-then-replay. |
| Verify embedded config vs marker content hash | `config_committed_marker` (`config_repl.c:216`) yields `{tag, sha}`; hash-verify-before-activate already the rule for R7 byte push (steering `ha-async-io.md` §per-entry-term) | **Missing for snapshot** — the verify-against-marker logic exists for the byte-push artifact; must be extended to the snapshot-embedded config. |
| Reject + keep retrying on mismatch (no self-terminate) | loud-fence-and-stay-alive invariant (`controller/CLAUDE.md` "Process Lifecycle") | **Constraint** — reject path MUST be a loud serve-fence + retry, never `exit`/`abort`. New code must honor this. |

### R4 — Post-upgrade consistency invariant

| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Identical materialized state across members | none — no entity snapshot to compare | **Missing** — depends entirely on R1/R2 landing. |
| Identical active config across members | `config_committed_marker` derives committed config from `(term,lsn)` | **Partial** — config identity is reachable via the marker; **state** identity is not. |
| Invariant holds across upgrade × failover interleave | Raft `(term,lsn)` reconcile + promotion-truncate (`ha-term-safety`, `raft.c`) | **Unknown** — needs design: how a mid-upgrade snapshot interacts with promotion-truncate and catch-up streaming when a leader dies mid-install. |

### R5 — Reproduction and regression

| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Reproduce bug-857 divergence (RED) | `waldump` (`testsuite/waldump.{c,h}`), `/xce-test-ha` chaotic-multi-kill soak, planned `snapdump`/`walcompare` (roadmap Hardening §) | **Partial** — waldump exists; `snapdump`/`walcompare` (entity-less-snapshot + WAL-divergence detectors) are **unbuilt** and are the natural assertion tools here. |
| Pass on fixed code | — | **Missing** — fix does not exist yet. |

---

## Gating Blocker — entity-bearing-snapshots (hard, explicit)

This is the single dependency that gates the whole spec.

- **Evidence it's absent:** `write_snapshot` (`persist.c:1247`) and `load_snapshot`
  (`persist.c:1555`) handle only magic + LSN + a `{tag,sha}` config-ref tail —
  **no job/queue/host/resource entities**. The header comment literally says
  *"Full Cap'n Proto serialization is Task 11 when recovery needs to read it back."*
- **Why it blocks:** R1.1, R2.1, R4.1 all require an artifact that carries
  materialized entity state. Without it, "install the same snapshot on every member"
  delivers an empty sentinel — divergence is unavoidable.
- **What's already paid down (de-risks it):** the COW substrate exists
  (`xzarray_freeze`/`thaw`, `zarray.c:695-757`, O(1) freeze, GC-filtered thaw,
  unit-tested) and the **transport** is reusable (`ha_handle_snapshot_chunk` +
  `persist_load_snapshot`). What's missing is the **payload**: serialize entities
  under per-object locks during the freeze window into the snapshot file, and
  deserialize them on `load_snapshot`. COW is single-generation/slot-level only —
  one consumer owns freeze/thaw (roadmap §entity-bearing-snapshots).

**Consequence for this spec:** requirements/design can proceed, but **tasks cannot
reach implementation** until `entity-bearing-snapshots` is implemented or
co-scoped. `spec.json.blocked_on` already records this.

Also depends on **`ha-term-safety` R7** for the `(term,lsn)` config-change marker
that the snapshot anchors on — that marker machinery **is implemented**
(`config_commit_change`, `config_committed_marker`, persisted config-ref).

---

## Implementation Approach Options

### Option A — Extend the existing checkpoint/upgrade path

Wire `vfreeze`/`vthaw` into `kvmgr`, teach `write_snapshot`/`load_snapshot` to
serialize/deserialize entities, and in `do_upgrade_config_with_bytes` (after the
marker commits, before restart) checkpoint an entity-bearing snapshot at the
marker `(term,lsn)`, embed the config bytes, stream it via `snapshot_transfer_s`,
and gate activation on install. Drop the per-node-replay restart for the upgrade case.

- ✅ Reuses the live transport + COW; minimal new files; one consistent snapshot format.
- ✅ Naturally supersedes the R7 byte-push for upgrades.
- ❌ Bloats `persist.c` snapshot codec and the already-large `config_repl.c` upgrade path.
- ❌ Couples the generic checkpoint snapshot format to upgrade-specific embedding
  (config bytes) unless cleanly factored.

### Option B — New "upgrade snapshot" component

A dedicated module owns: materialize-at-marker, embed config, drive transport,
activation gate, and the mismatch fence — leaving `persist.c` to expose only an
entity (de)serialize primitive and `config_repl.c` to call into it.

- ✅ Clean separation; upgrade-specific logic isolated and unit-testable.
- ✅ Keeps `config_repl.c`/`persist.c` from growing further.
- ❌ More wiring across persist ↔ new module ↔ ha transport ↔ config_repl.
- ❌ Risk of a second snapshot-format path diverging from the generic checkpoint one.

### Option C — Hybrid (recommended)

1. **Foundation (in/with `entity-bearing-snapshots`):** wire COW into kvmgr and add
   the generic entity (de)serialize to `write_snapshot`/`load_snapshot` — *Option A
   style*, shared by recovery + state transfer.
2. **Upgrade layer (this spec):** a thin new helper (Option B style) that, given an
   entity-bearing snapshot capability, materializes at the marker `(term,lsn)`,
   embeds the config bytes, streams via the existing transport, and adds the
   activation gate + hash-verify-vs-marker + loud-fence-on-mismatch. Replace the
   step-9 restart/replay for the upgrade case; keep restart/replay as the fallback
   only until the foundation lands.

- ✅ Generic snapshot work is reused (recovery, faster rebuild) — not upgrade-only.
- ✅ Upgrade-specific gating/verification stays isolated and testable.
- ✅ Phased: foundation can land + be validated before the upgrade layer flips on.
- ❌ Requires careful coordination across the two specs (format ownership, ordering).

---

## Effort & Risk

| Dimension | Rating | One-line justification |
|-----------|--------|------------------------|
| **This spec alone** (assuming entity-bearing snapshots exist) | **L (1–2 wks)** | Transport + marker + COW exist; new work is materialize-at-marker, config embed, activation gate, hash-verify, mismatch-fence, and the failover-interleave regression. |
| **Including the gating dependency** | **XL (2+ wks)** | `entity-bearing-snapshots` is itself non-trivial (entity serialization under per-object locks, COW wiring, recovery read-back) and must land first. |
| **Risk** | **High** | Hard blocker not yet started; touches consensus-adjacent state at a `(term,lsn)` boundary interleaved with failover (R4.3); snapshot/promotion-truncate/catch-up interaction is unverified; must preserve the never-self-terminate invariant. |

---

## Recommendations for Design Phase

**Preferred approach:** **Option C (Hybrid)** — land entity-bearing snapshots as a
shared foundation (COW into kvmgr + generic entity (de)serialize in
`write_snapshot`/`load_snapshot`), then add a thin upgrade layer that materializes
at the marker, embeds the config, reuses `snapshot_transfer_s`, and adds activation
gating + hash-verify + loud-fence.

**Key decisions to settle in design:**
1. **Spec sequencing** — co-scope `entity-bearing-snapshots` into this spec's
   delivery, or hard-gate this spec behind it? (Recommend gate; promote to design
   only once the foundation has a tasks plan — matches `roadmap.md`.)
2. **Snapshot format** — one unified entity-bearing format used by both recovery and
   upgrade, with the config-bytes embed as an optional section? Avoid a second format.
3. **Materialize-at-marker semantics** — snapshot must reflect state at *exactly* the
   marker `(term,lsn)`, not "latest checkpoint." Define how the freeze window aligns
   to the committed marker LSN.
4. **Activation gate** — bind "new config active" to "upgrade snapshot for this
   `(term,lsn)` installed + hash-verified vs `config_committed_marker`"; mismatch ⇒
   loud serve-fence + retry (never exit — `controller/CLAUDE.md`).
5. **Retire the per-node restart/replay** for the upgrade case (step 9,
   `config_repl.c:1657-1676`) once the snapshot path is proven.

**Research-Needed (carry forward):**
- Upgrade × failover interleave (R4.3): how does a mid-upgrade entity snapshot
  interact with promotion-truncate, `(term,lsn)` reconcile, and ACK-driven catch-up
  if the leader dies mid-install? Define the recovery story.
- COW single-generation constraint vs. a checkpoint racing an upgrade snapshot
  (only one freeze/thaw consumer allowed — `zarray.c` / roadmap note).
- Build `snapdump` + `walcompare` (roadmap Hardening §) first — they are the RED/GREEN
  assertion tools for R5 (entity-less snapshot detection + cross-node WAL/state diff).

---

## Update — 2026-07-21 (dependencies resolved; requirements extended; harness gap analysis)

This section is current as of 2026-07-21 and supersedes the stale parts of the
2026-06-24 analysis above (kept for history). Do not re-derive the gating
status or tool inventory from the sections above without reading this first.

### Gating blocker — RESOLVED

- **`entity-bearing-snapshots`** — satisfied 2026-07-09 by `ha-entity-snapshots`
  (21/21 tasks, fully approved). It shipped `write_snapshot`/`load_snapshot`
  (`persist.c`) as a direct Cap'n-Proto entity-encode/decode of job and queue
  records at checkpoint time — **not** the COW `xzarray_freeze`/`thaw` path this
  research doc's Option A/C assumed (that machinery is still dormant,
  `kvmgr.c:297`). Design must build on the real `write_snapshot`/`load_snapshot`
  interface, not the COW one.
- **`ha-term-safety`** (owns the `(term,lsn)` config-change marker this spec
  anchors on) — `implementation-complete`, GO-validated 2026-07-17, 46/46
  acceptance criteria mapped, live chaotic soak clean.
- Net effect: this spec is **unblocked**. `spec.json.blocked_on` should be
  cleared and the phase can move to design once requirements are approved.

### Tool inventory — CORRECTED (contradicts the "unbuilt" note above)

`snapverify`, `snapdiff`, and `walcompare`/`walcompare_report` all **already
exist** (`src/controller/testsuite/`), built 2026-07-02 during bug-1386/1389
triage. The line-57/176 framing above ("planned", "unbuilt", "build first") is
stale — do not re-propose building them.

- `snapverify_path` — structural-only checkpoint-file validator (magic,
  format_version, frame walk to EOF). Reports `has_cfgref` as a bool but does
  **not** decode or assert the tag/hash value.
- `snapdiff_paths` — byte-for-byte comparator between two checkpoint files,
  with chunk-boundary diagnosis. Built for source-vs-received comparison but
  directly reusable for a determinism check (two independently materialized
  snapshots at the same boundary should be byte-identical).
- `walcompare`/`walcompare_report` — cross-member committed-prefix divergence
  detector, compares `(term, pcrc)` per LSN, and already has a **per-member
  checkpoint floor** concept (an LSN below a member's own installed-snapshot
  floor is "legitimately compacted," never flagged divergent — `ha-entity-
  snapshots` task 7.1 / `raft-ha-core-safety` task 5.10a). It does **not**
  compare active configuration across members.

### Requirements extended this session (2026-07-21)

`requirements.md` gained content not reflected in this doc's "Recommendations"
section (items 1-5 above predate these):
- **Req 1.5** — snapshot materialization is triggered synchronously by
  marker-apply but executed through the *existing* async checkpoint mechanism
  (`persist_checkpoint`/`schedule_checkpoint`/`write_snapshot`, delaymgr-driven,
  CAS-guarded single-in-flight), not inlined into the raft apply path — answers
  the "should this be one atomic Raft FSM operation" question: the marker
  commit already is one FSM op; materialization should be triggered by it but
  run via the existing decoupled checkpoint mechanism (never block a vtask
  handler); cross-node install must never be — it's inherently async.
- **Req 2.4** — every checkpoint taken after a config change embeds the
  *current* config, not only the marker-triggered one, so a member joining
  after the upgrade snapshot has rotated away (`rotate_snapshots`, `snap_keep_n`)
  still gets config + state together. Piggybacks on the existing
  `scan_latest_committed_config_ref` (`persist.c:1999`), which already re-stamps
  the latest committed config *reference* into every checkpoint — Req 2 extends
  that from a reference to the full embedded bytes.
- **Req 5 (new)** — Crash-safety and idempotent resumption: deterministic
  snapshot content (any member's independent materialization at the same
  `(term,lsn,config-hash)` is identical), crash-safe resumption if a member
  dies mid-materialization, idempotency guard reusing the existing
  `checkpoint_in_progress` CAS, and leader-change-mid-distribution resumption.
- **Req 6.3 (was Req 5.2)** — regression test must inject a crash/restart and a
  leadership change specifically between marker-commit and snapshot-install-
  complete, not just rely on general chaotic-kill timing.

### Invariant risk carried forward from this session's controller-invariant check (2026-07-21)

Checked the new requirements (esp. Req 3's activation gate) against
`controller/CLAUDE.md`'s TLA+ invariants and Protocol Verification Checklist.
One real precedent-risk, already fixed at the requirements level (Req 1.8) but
flagging for design/implementation attention:

- **bug-1418 (open, NOT FIXED)** is the same failure shape this spec's Req 3
  risks reintroducing in a new flavor. bug-1418: a newly-elected primary whose
  *apply* is behind (`apply_divergent=1`) hits a **liveness deadlock** — the
  existing promotion gate blocks it hard, repeatedly, rather than degrading
  gracefully; the (unimplemented) proposed fix is "become primary, self-apply,
  then serve." Req 3.2/3.3 of this spec assume exactly that graceful pattern
  for *materialization*-lag instead of *apply*-lag. Since the apply-lag version
  of this pattern is proven broken in production today, design must not
  implement Req 3's gate as a hard promotion block (Req 3.1 already commits to
  this — no separate promotion-eligibility mechanism) and must verify the
  "keep serving under last-verified config" fallback (Req 3.3) is genuinely
  implemented, not degrade into the same stall bug-1418 exhibits.
- Root cause of the *need* for Req 1.8 (apply must not itself re-home entities;
  activation is a separate, later-gated step): without that split, Req 3.3
  ("keep serving under last-verified configuration" while materialization is
  incomplete) is incoherent — there would be no old interpretation left to
  serve under once apply had already mutated live entity state. This is the
  same family of apply-ordering subtlety behind bug-1417/1418/1419.
- **Design-verification item (unconfirmed, not yet checked in code):** does
  `persist_load_snapshot`/`load_snapshot` advance `durable_lsn` (not just
  `checkpoint_lsn`/`current_lsn`) when a member catches up via the transport-
  install path (Req 1.2's second branch)? If not, a snapshot-caught-up member
  may not correctly count toward commit majority for LSNs at/below the
  snapshot boundary even though it durably holds that state via the snapshot
  file rather than raw WAL — a possible Commit Durability invariant gap
  specific to the transport-install path. Verify in design before assuming
  Req 1.2's two paths are equally safe.

### Harness gaps for design phase (new, from this session's analysis)

Three items, only two are genuinely new build work:

1. **No cross-member config comparator (new gap).** `walcompare` diffs WAL only.
   Req 4.2 ("every member's active configuration identical") has no assertion
   tool. Extend `walcompare` (or add a sibling) to decode+compare the embedded
   config tag/hash (and eventually bytes) per member, mirroring how it already
   compares `(term,pcrc)`. Also extend `snapverify` to decode/assert the
   cfgref tag+hash value, not just report its presence.
2. **No deterministic fault-injection at the marker-apply → forced-checkpoint-
   durable window (new gap, needed for Req 5.2/6.3).** `/xce-test-ha --chaotic`
   kills at random points — could hit this narrow window eventually, too flaky
   for a CI regression gate. Precedent for the right shape of hook already
   exists twice in `config_repl.c`: `ha_stream_body_read_fault_hook`,
   `xce_snapshot_chunk_hook` (both `#ifdef XCE_TESTING` seams). Add an
   equivalent hook at the marker-apply/forced-checkpoint boundary.
3. **No deterministic leadership-change-mid-distribution injection (new gap,
   Req 5.4/6.3).** Same precedent pattern as #2 — a targeted "kill/step-down
   primary exactly after marker commits, before all standbys ACK install,"
   not reliance on chaos-soak probability.
4. **Reusable, no new work needed:** `snapdiff_paths` for the Req 5.1
   determinism check; `/xce-test-ha --force-snapshot` for Req 2.4's
   fresh-join-after-rotation (already exercises wiped-node-rejoins-empty via
   bug-1386's own test mode — just needs its assertions extended once config
   embedding lands); `config_commit_change`/`do_upgrade_config_with_bytes`
   (bug-970) already wires `WAL_CONFIG_MARKER` into the live upgrade path, so
   the bug-857 repro (Req 6.2) is triggerable today with existing tooling —
   no blocker there.

---

## Update — 2026-07-23 (design discovery; requirements extended to 7 requirements/36 ACs)

Requirements went through several rounds of review this session (host/queue admin-state frames, bug-1418 re-verified fixed, bootstrap-vs-upgrade boundary, Design Invariants section, Requirement 7 added for new-mechanism regression coverage). This update captures the DESIGN-level discovery done to write `design.md`, current as of 2026-07-23, extension-scope (light discovery — existing system, no new external dependency).

### Discovery: exact hook points for the marker → materialization trigger (Req 1.1, 1.5, 1.8)

**The single unified apply seam is `apply_committed_entry`** (`persist.c:2751-3132`), confirmed as the ONLY live path for entity mutation on every member — primary online-apply (`persist_apply_pump_run` → `persist_apply_pump_kick`, vtask-pool dispatched, `persist.c:3383`), standby normal apply (`ha_standby_advance_committed_and_apply`, heartbeat-learned commit position, `ha.c:2596-2613/2688-2700` — supersedes the older "apply on receipt" bug-1043 fix as of 2026-07-10), and boot/restart replay (`persist.c:3920`). The sibling dispatcher `apply_wal_entry` (`persist.c:2583-2714`) has **zero live callers** (grep-confirmed) — fully superseded, dead code, not a second seam to worry about. `WAL_CONFIG_MARKER` currently falls into `apply_committed_entry`'s generic `default:` no-op (`persist.c:3126-3131`) and `apply_wal_entry`'s equivalent no-op (`persist.c:2697-2707`, irrelevant given no callers). **Design decision: add one new explicit `case WAL_CONFIG_MARKER:` in `apply_committed_entry` only** — it fires identically for primary and standby, satisfying Req 1.2's "every member... via that member's own local materialization once it has applied the marker itself" for free, with no per-role branching needed.

**Today's restart+replay behavior being replaced** lives in `do_upgrade_config_with_bytes` (`config_repl.c:1591-1773`, step 9, `config_repl.c:1753-1772`): `atomic_store(&c->restart_requested, 1)` + `cbuf_put(&c->wbuf, &stop_val)`. An existing NOTICE comment at `config_repl.c:1737-1751` already names this exact spec by path and describes the target end-state verbatim — a strong signal the upgrade path's own maintainers already anticipated this design. No function-pointer hook exists at that NOTICE site (grepped, none) — it is a plain fall-through, confirming the real trigger point is the `apply_committed_entry` case above, not this step-9 comment site itself. **Once the new case lands, step 9's restart/replay becomes dead code for the upgrade path and should be removed** (not left as a parallel fallback — Design Invariant "never leave two mechanisms doing the same job").

### Discovery: checkpoint triggering — no forced/immediate entry point exists yet (Req 1.5)

`persist_checkpoint()` (`persist.c:2262-2321`) is unconditionally synchronous/blocking (fdatasync-backed `write_snapshot`, directory scans) and has exactly one production trigger path today — `schedule_checkpoint()` (`persist.c:2238-2256`), fired only by a WAL-record-count threshold inside `persist_wal_async()` (`persist.c:921-928`). No wall-clock periodic timer exists in the live code (an older design doc's `checkpoint_requested`/`persist_checkpoint_timer_fn` was never implemented — grep-confirmed absent). **No `force_checkpoint()`/`persist_checkpoint_now()` entry point exists** — but a second direct, unconditional call to `persist_checkpoint()` already exists at `ha.c:3308` inside `ha_snapshot_do_send()` ("Req 4.1: create a FRESH checkpoint..."), proving the function itself already tolerates being force-invoked outside the record-count path (it already handles the `-EBUSY` a concurrent scheduled checkpoint can return, via the shared `checkpoint_in_progress` CAS, `persist.h:415`).

Two existing patterns already solve "trigger without blocking the caller" in this codebase — `delay_task`/delaymgr enqueue (what `schedule_checkpoint` does; the *actual* checkpoint I/O then runs on the delaymgr thread itself, per direct trace of `delay_worker`/`run_delay_tasks`, `delay.c:60`/`cluster.c:988-991` — delaymgr has no thread-pool hand-off) and the dedicated **slow-lane executor** (what `ha_snapshot_do_send` uses, `ha.c` ~6120-6160, specifically because `persist_checkpoint` is blocking/unbounded — this is the pattern `ha-async-io.md:29` documents as the bug-1198 fix for inline-on-iomgr checkpoint I/O). **Design decision: the new `WAL_CONFIG_MARKER` case in `apply_committed_entry` calls `persist_checkpoint()` directly, not via either existing indirection.** This is safe specifically because `apply_committed_entry` (via the apply pump) already runs on its own vtask-pool worker thread, never the iomgr or delaymgr thread — it IS the async-dispatched context, so no further hand-off is needed for THIS call site (unlike `schedule_checkpoint`'s WAL-threshold trigger, which fires from arbitrary vtask handlers appending WAL entries and genuinely needs delaymgr's async return). This gives Req 1.5's "immediate and deterministic" trigger for free — no new delay/lane plumbing required — while the CAS (`checkpoint_in_progress`) already guards against a racing periodic checkpoint.

**Note for design (not a requirements gap, an implementation-order note):** `scan_latest_committed_config_ref`/`write_snapshot`'s existing `persist_checkpoint` call sequence (`persist.c:2301/2305`) always stamps `checkpoint_lsn = last_applied_lsn` (`persist.c:2292-2295`) — i.e. a marker-triggered forced checkpoint calling `persist_checkpoint()` at the moment `apply_committed_entry` handles the marker will naturally pin `checkpoint_lsn` to the marker's own `(term,lsn)`, satisfying Req 1.1's "materialize at that marker's boundary" without any new LSN-targeting parameter — `persist_checkpoint()` doesn't need to accept an explicit target LSN; it just needs to run at the right moment, which the new apply-time case guarantees by construction.

### Discovery: the activation gate has a direct, reusable precedent (Req 3)

`ha_dispatch_gate` (`ha.c:3809-3975`, single call site `cluster.c:276`, gating every wire request — client and peer alike, both HA replication opcodes and apiserver-relayed job/queue/host opcodes) already composes multiple independent boolean fences in sequence for the `CTRL_PRIMARY` role: role check, then a **config-serve-fence check** (`config_serve_fenced(ha->raft_c)`, `ha.c:3827`), then a write-quorum check. This is a **direct structural precedent already in production** for exactly what Req 3 needs — a config-dependent, state-changing-request fence that is orthogonal to role/quorum, with its own opcode allowlist (replication + read-only ops stay admitted while fenced).

The existing `config_serve_fence` (`ha.h:307-322`, set/cleared by `config_activate_committed` in `config_repl.c:304-377`, hash-verifies committed marker bytes against locally-held config bytes) is the closer of the two fence precedents found (the other being `recovery_unrecoverable`, which only gates *promotion*, not live serving) because it already: (i) blocks client-request serving via `ha_dispatch_gate`, (ii) also blocks promotion (`ha.c:1425-1440`, `ha_config_promotion_fenced()`), and (iii) self-clears via a fully-wired, non-blocking retry loop (`config_schedule_activate_retry` → delaymgr trampoline → vtask does the actual blocking verify I/O, `config_repl.c:415-507`), triggered from the commit-advance callback (`ha.c:2197-2198`) — the exact "loud ERROR once, stay alive, keep retrying, never self-terminate" shape `controller/CLAUDE.md` mandates, already proven in production. **Design decision: add a new atomic fence field on `ha_state_t` (sibling to `config_serve_fence`, e.g. `upgrade_snapshot_fence`), a matching setter/verify function mirroring `config_activate_committed`'s hash-verify-then-clear shape, and one more fence check inside `ha_dispatch_gate`'s `CTRL_PRIMARY` branch** (composed the same way the existing config-serve-fence check already is — not folded into that same field, since the two conditions are logically independent: config-serve-fence is about byte-mismatch, the new fence is about materialization-not-yet-complete for an otherwise-matching config) — reusing `config_schedule_activate_retry`'s exact trampoline shape for the new fence's own retry loop, and `ha_config_promotion_fenced()`'s pattern for gating promotion the same way.

### Discovery: durable_lsn is NOT advanced by snapshot-install — real but bounded gap (Req 1.2, 5.x)

Traced definitively (grep + direct read of every function in the call path): `ha_handle_snapshot_chunk` (`ha.c:3014-3171`) → `persist_load_snapshot`/`load_snapshot` (`persist.c:2456-2575`) never touches `p->durable_lsn` anywhere — only `checkpoint_lsn`/`current_lsn`/`cfgref_*` get set. This is asymmetric with the boot-recovery path (`persist_recover`, `persist.c:4338-4353`), which DOES re-derive `durable_lsn` from the recovered WAL tip. **Verdict: real, confirmed gap, but bounded to this node's OWN promotion-eligibility/self-advertised-freshness bookkeeping during a narrow live-catch-up window** (`persist_durable_floor` consumers: pre-promotion gate `ha.c:1318-1352`, promotion truncate-floor clamp `ha.c:1259-1262`, self-advertised candidate position `ha.c:4540-4546`) — the existing fence-and-retry machinery (`ha_arm_promotion_fence_timer`) already handles this safely (conservative refusal to self-promote, never corruption). It is **NOT** a commit-durability/majority-counting safety gap for OTHER members counting a snapshot-caught-up peer, because `majority_candidate_locked` (`commit.c:167-172`) already excludes any standby with `in_catch_up != 0` from majority evaluation, and that flag is cleared only by a real WalAck reaching `catchup_derive_target` (`repl.c:713-739`, via `ha_walack_on_ack`) — a separate, already-safe gate keyed off real acks, not the snapshot event. **Design decision: out of this spec's scope to fix** (it predates this spec, applies to any snapshot-install today including plain recovery-driven catch-up, not upgrade-specific) — but flag it as a design.md Risk/Note so a future spec can pick it up (advancing `durable_lsn` on snapshot-install the same way boot-recovery does would close it cleanly, mirroring `persist.c:4338-4353`'s ASSIGN semantics).

### Discovery: test tooling — three reusable as-is, two confirmed genuine new-build gaps

Confirmed via direct read (not just trusting the 2026-07-21 update above, which was itself a research note, not a verified read):
- **`snapverify_path`** (`testsuite/snapverify.c`) — confirmed `has_cfgref` is presence-only (`header[16]`), never decodes tag/sha; zero test assertions on it exist today. Needs extending to decode/assert tag+hash for Req 4.2/Req 2 coverage.
- **`snapdiff_paths`** (`testsuite/snapdiff.c`) — confirmed reusable as-is for Req 5.1's determinism check (any two paths, no transport-specific assumptions).
- **`walcompare`/`walcompare_report`** (`testsuite/walcompare.c`) — confirmed zero references to `cfgref_tag`/`cfgref_sha`/config bytes anywhere in the file; compares only `(lsn, term, pcrc)`. Req 4.2 (active-config identity across members) has no assertion tool — needs either extending `walcompare` or a sibling tool.
- **Fault-injection precedent, exact shape confirmed**: `ha_stream_body_read_fault_hook` (`config_repl.c:2104-2106`, wired at `config_repl.c:2183-2193`) and `xce_snapshot_chunk_hook` (`config_repl.c:2847`, wired at `config_repl.c:2937-2949`) are both `#ifdef XCE_TESTING`-gated, `NULL`-by-default, file-scope function-pointer globals, `extern`-declared and set/reset around the driving call in the test file. This is the exact shape the two NEW deterministic hooks (Req 7.2's marker-apply→forced-checkpoint-durable window, Req 7.2/5.4's leadership-change-mid-distribution window) should mirror — a hook inside the new `apply_committed_entry` `WAL_CONFIG_MARKER` case (or wherever the cross-member install-wait loop lives) taking identifying args (marker lsn/term) so a test can force-fail/force-delay at that exact boundary instead of relying on chaotic-soak timing luck.
- **Test fixture pattern, exact shape confirmed**: `test_apply_committed_entry.c` (bare in-memory `cluster_t`, no disk, direct `apply_committed_entry()` calls, spy hooks counting `HOOK_AFTER_EVENT` fires) and `test_replay_apply_seam.c` (real disk-backed `persist_t`, two-phase write-then-recover, proving replay routes through the same seam) are the two patterns new host-frame-restore and marker-apply tests should extend — both already have a `tace_encode_worker`/`trr_encode_worker` local-capnp-encode helper (mirroring `persist.c`'s file-static `capnp_encode_worker`) that a new host-admin-frame test can copy the idiom from.

### Design Synthesis (per `rules/design-synthesis.md`)

**1. Generalization**: The host-admin-state frame (Req 1.6/1.9/1.10) and the queue-admin-state frame (existing, now migrated to capnp per Req 1.9) are the same underlying shape — `(name, status)` with independently-extensible fields. Rather than two hand-written near-duplicate collect/restore function pairs, design one generic `collect_admin_state_frames(table, status_mask)`/`restore_admin_state_frame(table, name, status, create_if_missing_fn)` pair parameterized over which table (`workermgr`/`qmgr`) and which create-if-missing behavior (hosts: `makeWorker`-based placeholder; queues: none, always pre-provisioned) applies — the two concrete restore functions (`restore_host_state`, `restore_queue_state`) become thin instantiations, not independent implementations. This directly follows Design Invariant 1 (classify config- vs join-provisioned) by construction: the generic function takes the classification as a parameter rather than assuming one.

**2. Build vs. Adopt**: Every major piece here is "adopt existing internal mechanism," not "build new" — the snapshot transport (`snapshot_transfer_s`/`ha_handle_snapshot_chunk`), the checkpoint/CAS machinery (`persist_checkpoint`/`checkpoint_in_progress`), the fence-and-retry pattern (`config_serve_fence`), the placeholder-entity pattern (`lost_and_found`'s `makeWorker()` precedent), and the fault-injection hook shape (`ha_stream_body_read_fault_hook`) are all reused verbatim rather than reimplemented. The only genuinely NEW machinery is: the `WAL_CONFIG_MARKER` case in `apply_committed_entry`, the new fence field + verify function (structurally mirroring `config_serve_fence` but a distinct field), the config-bytes-embed in the snapshot header/entity payload, the `HostAdminState`/`QueueAdminState` capnp structs, and the two new test hooks. No external library or new dependency is involved anywhere in this spec.

**3. Simplification**: Considered whether the new fence needs to be a SEPARATE atomic from `config_serve_fence` — confirmed yes (the two conditions are logically independent: hash-mismatch vs materialization-incomplete, and Req 3.4 needs to distinguish "keep serving under last-verified config" from "no verified config exists yet," which requires knowing WHICH fence is active). Considered whether `apply_wal_entry` needs a matching `WAL_CONFIG_MARKER` case for symmetry with `apply_committed_entry` — confirmed no, it has zero live callers, touching it would be pure speculative-symmetry work with no runtime effect (Design Invariant would call this exactly the "convenient but unjustified" mistake to avoid).

### Risks carried into design.md
- The narrow window between `schedule_checkpoint`'s CAS release (`checkpoint_task_fn`, `persist.c:2222`) and `persist_checkpoint`'s own re-acquire (`persist.c:2265`) allows a second concurrent trigger to slip in and start its own `persist_checkpoint()` call — pre-existing, not introduced by this spec, but the new marker-triggered call site adds a second caller into this same race window and should be noted.
- `durable_lsn` not advancing on snapshot-install (above) — pre-existing, out of scope, flagged for a future spec.

---

## Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)

### 1. Current State Investigation

All three specs touch the same controller-side config surface (`controller.capnp`'s `ClusterConfig`, `config_repl.c`'s `OP_upgradeConfig` handler, `cluster.c`'s dispatch gate) from different angles:

| Spec | Phase | Scope |
|---|---|---|
| `clusterconfig-schema-split` | `implementation-complete` | Remove per-instance/dead fields (`port`, `candidates`, perf-tuning) from `ClusterConfig` |
| `admin-config-tool` | `implementation-complete` (blocked_on stale — see §2) | `xceadmin` CLI: build/push `cluster.bin` against the controller |
| `config-upgrade-snapshot` | `tasks-generated`, 18/19 actionable tasks `[x]` | Snapshot the config alongside cluster state at every upgrade boundary; activation fencing |

**Verified current schema state** (`src/schema/controller.capnp:712-726`): `ClusterConfig` has exactly 13 fields (`@0`–`@12`: `clusterName` … `configGeneration`), matching `clusterconfig-schema-split`'s Requirement 3.1 target exactly. `MasterCandidate`/`HashZArrayConfig`/`ResmgrConfig` are gone (Requirements 1.6/2.5). `config-upgrade-snapshot`'s own schema additions (`HostAdminState`, `QueueAdminState`, lines 769/774) are **separate message types**, not fields on `ClusterConfig` itself — **no ordinal collision or renumbering conflict between the two specs' schema work.**

**Verified dispatch-gate state** (`src/controller/cluster.c:669-694`): the `OPCode_upgradeConfig` polarity-inversion bug that `admin-config-tool`'s `spec.json.blocked_on` and `tasks.md` (line 155) both cite as blocking real-cluster `config push` E2E validation is **already fixed** — commit `959b0cda` (bug-1468, landed 2026-07-25 as part of `config-upgrade-snapshot` task 8.6). The fix's own comment in `cluster.c` explicitly cross-references `admin-config-tool`: *"Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."*

### 2. Gap Found: `admin-config-tool`'s `blocked_on` is stale documentation, not a live blocker

`admin-config-tool/spec.json`'s `blocked_on` array still lists:
> "controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"

This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, **the same controller-side `OP_upgradeConfig` path this blocker describes has since been exercised live, repeatedly, and successfully** by `config-upgrade-snapshot`'s own `--config-upgrade` soak (`.claude/skills/xce-test-ha/run-ha-soak.sh`) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and `WAL_CONFIG_MARKER` commit. The only difference is the *client*: the soak drives the RPC via `xce_test`'s hand-rolled `upgradeconfig` command (built for `config-upgrade-snapshot` task 8.6, since no other live `OP_upgradeConfig` client existed at the time), not `xceadmin`'s Python `config push`. But the **controller-side code path is identical** for both clients — the dispatch gate doesn't distinguish caller identity.

- **Gap classification**: Missing (stale tracking data, not missing capability). `admin-config-tool`'s actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with `xceadmin config push` against a live cluster to close it out formally — the blocking condition itself no longer exists.
- **Recommendation for design/task phase**: Clear `admin-config-tool/spec.json`'s `blocked_on` entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. `clusterconfig-schema-split`'s already-resolved entries and `config-upgrade-snapshot`'s own dated Implementation Notes). If the E2E task was left unchecked pending this, run `xceadmin config push` against a real deployed cluster (`/xce-deploy-ha`) once to get first-hand `xceadmin`-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).

### 3. Gap Found (residual, this session): `config-upgrade-snapshot` task 8.6 — new commit-timeout issue post-bug-1477

`config-upgrade-snapshot`'s own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see `.wolf/buglog.json`). Re-running `--config-upgrade` after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all **PASS** for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (`rc=-110`) on the primary — a step this soak scenario has **never previously reached** (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — **not yet root-caused**, flagged for the next investigation pass. Task 8.6 remains open pending this.

- **Gap classification**: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
- **Ownership**: Likely `UNCLEAR` until investigated — could be `ha-commit-durability`/`ha-term-safety`/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a `config-upgrade-snapshot`-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.

### 4. No gap found: `clusterconfig-schema-split` × `admin-config-tool` integration

`admin-config-tool/requirements.md`'s own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit `519e11c`), and the verified current schema state (§1 above) confirms it: `port`/`candidates`/perf-tuning fields are genuinely absent from `ClusterConfig`, so `xceadmin`'s Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.

### 5. No gap found: `clusterconfig-schema-split` × `config-upgrade-snapshot` schema coexistence

Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of `controller.capnp` (`ClusterConfig`'s field list vs. new standalone `HostAdminState`/`QueueAdminState` structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.

### Summary Table

| # | Finding | Classification | Effort | Risk | Action |
|---|---|---|---|---|---|
| 1 | `admin-config-tool` `blocked_on` stale (bug-1468 already fixed) | Missing (tracking, not capability) | S (update spec.json + one live `xceadmin config push` run) | Low | Clear blocker, optionally run one live E2E confirmation |
| 2 | `config-upgrade-snapshot` task 8.6: new commit-timeout past force-elect step | Unknown (not yet root-caused) | Unknown until investigated | Medium–High (commit/write-quorum path) | Dedicated debug pass before considering 8.6 closeable |
| 3 | `clusterconfig-schema-split` ↔ `admin-config-tool` | No gap | — | — | None |
| 4 | `clusterconfig-schema-split` ↔ `config-upgrade-snapshot` schema | No gap | — | — | None |

### Research Needed (carried forward)
- Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.

**Resolved during this pass** (no longer open): `admin-config-tool/tasks.md` was audited directly — all 23 tasks are `[x]`, none gated on the stale blocker as an active checklist item. The E2E gap is purely `spec.json.blocked_on` + an informational `tasks.md` note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.