myxce/.kiro/specs/config-upgrade-snapshot/research.md

Gap Analysis — config-upgrade-snapshot (2026-06-24)

Scope

Make a configuration upgrade install one identical, entity-bearing state snapshot (with the new config embedded) on every member at the config-change marker's (term, lsn) boundary, replacing today's config-push + restart + per-node WAL replay (which can materialize divergent state — bug-857). Closes bug-857.

GATING BLOCKER (hard dependency): entity-bearing-snapshots. Today snapshots carry zero entity state (an LSN sentinel). This spec consumes an entity-bearing snapshot it cannot produce. It cannot reach a testable GREEN until entity-bearing-snapshots lands or is co-scoped. Every "Missing" below traces back to this one absence. See §"Gating Blocker".

---

Requirement-to-Asset Map

R1 — Full snapshot at the config-change boundary

Need	Existing asset (file:line)	Gap
Materialize a snapshot at the marker's (term,lsn)	persist_checkpoint / write_snapshot (persist.c:1247-1291) writes a checkpoint snapshot at checkpoint_lsn	Missing — snapshot is a 16-byte sentinel ("XCESNAP\0" + LSN + optional 41-byte config-ref tail); carries no entities. No way to pin it to the marker boundary vs. an arbitrary checkpoint LSN.
Snapshot pins identical materialized state	dual-zarray COW: xzarray_freeze/xzarray_thaw (zarray.c:695-757), test_zarray_cow.c	Missing (dormant) — built + unit-tested but no production callers: kvmgr.c:297 vfreeze=NULL /* wired in P0-2 */, vthaw=NULL.
No member re-derives via replay	upgrade path do_upgrade_config_with_bytes (config_repl.c:1495-1677); NOTICE at step 9 (config_repl.c:1641-1655)	Constraint (current behavior is the bug) — step 9 sets restart_requested; on restart each node replays its own pre-upgrade WAL under the new config (config-dependent via lost_and_found). This is exactly what R1.3 forbids.
Transfer out-of-band, not in WAL payloads	ha_handle_snapshot_chunk (ha.c:2424-2515) + persist_load_snapshot (persist.c:1594); OPCode_snapshotTransfer dispatch-gated (ha.c:2842/2874/2962)	Exists (reusable transport) — chunked, size-checked, atomic snap.part→rename, then persist_load_snapshot. Reusable as-is; it only ever moves a sentinel today.

R2 — Configuration embedded in the upgrade snapshot

Need	Existing asset (file:line)	Gap
Embed new config in snapshot (one artifact)	snapshot config-ref tail (persist.c:1263-1274, read-back 1581-1590): {present, version_tag(8), sha256(32)}	Partial / Missing — only a {tag, sha} reference is embedded, not the config bytes and not the entity state. Need the actual config bytes + entities in one installable artifact.
Supersede ha-term-safety R7 config-push for upgrades	R7 out-of-band byte push driven from config_commit_change (config_repl.c:478-510)	Missing — upgrade still uses the R7 interim byte-push path; no branch that delivers config via the snapshot instead.
Keep config bytes out of WAL payloads	WAL_CONFIG_MARKER is {version_tag, sha256} only — config_marker_encode (config_repl.c:40-69), bytes never in WAL	Already satisfied — marker carries no bytes; invariant holds today.

R3 — Activation gating

Need	Existing asset (file:line)	Gap
Don't serve/activate new config until snapshot installed	install-before-activate already structural in ha_handle_snapshot_chunk (rename+load on full receipt)	Partial — transport installs before load, but there is no activation gate tying "config active" to "upgrade snapshot for this (term,lsn) installed." Today activation = restart-then-replay.
Verify embedded config vs marker content hash	config_committed_marker (config_repl.c:216) yields {tag, sha}; hash-verify-before-activate already the rule for R7 byte push (steering ha-async-io.md §per-entry-term)	Missing for snapshot — the verify-against-marker logic exists for the byte-push artifact; must be extended to the snapshot-embedded config.
Reject + keep retrying on mismatch (no self-terminate)	loud-fence-and-stay-alive invariant (controller/CLAUDE.md "Process Lifecycle")	Constraint — reject path MUST be a loud serve-fence + retry, never exit/abort. New code must honor this.

R4 — Post-upgrade consistency invariant

Need	Existing asset (file:line)	Gap
Identical materialized state across members	none — no entity snapshot to compare	Missing — depends entirely on R1/R2 landing.
Identical active config across members	config_committed_marker derives committed config from (term,lsn)	Partial — config identity is reachable via the marker; state identity is not.
Invariant holds across upgrade × failover interleave	Raft (term,lsn) reconcile + promotion-truncate (ha-term-safety, raft.c)	Unknown — needs design: how a mid-upgrade snapshot interacts with promotion-truncate and catch-up streaming when a leader dies mid-install.

R5 — Reproduction and regression

Need	Existing asset (file:line)	Gap
Reproduce bug-857 divergence (RED)	waldump (testsuite/waldump.{c,h}), /xce-test-ha chaotic-multi-kill soak, planned snapdump/walcompare (roadmap Hardening §)	Partial — waldump exists; snapdump/walcompare (entity-less-snapshot + WAL-divergence detectors) are unbuilt and are the natural assertion tools here.
Pass on fixed code	—	Missing — fix does not exist yet.

---

Gating Blocker — entity-bearing-snapshots (hard, explicit)

This is the single dependency that gates the whole spec.

• Evidence it's absent: write_snapshot (persist.c:1247) and load_snapshot (persist.c:1555) handle only magic + LSN + a {tag,sha} config-ref tail — no job/queue/host/resource entities. The header comment literally says "Full Cap'n Proto serialization is Task 11 when recovery needs to read it back."
• Why it blocks: R1.1, R2.1, R4.1 all require an artifact that carries materialized entity state. Without it, "install the same snapshot on every member" delivers an empty sentinel — divergence is unavoidable.
• What's already paid down (de-risks it): the COW substrate exists (xzarray_freeze/thaw, zarray.c:695-757, O(1) freeze, GC-filtered thaw, unit-tested) and the transport is reusable (ha_handle_snapshot_chunk + persist_load_snapshot). What's missing is the payload: serialize entities under per-object locks during the freeze window into the snapshot file, and deserialize them on load_snapshot. COW is single-generation/slot-level only — one consumer owns freeze/thaw (roadmap §entity-bearing-snapshots).

Consequence for this spec: requirements/design can proceed, but tasks cannot reach implementation until entity-bearing-snapshots is implemented or co-scoped. spec.json.blocked_on already records this.

Also depends on ha-term-safety R7 for the (term,lsn) config-change marker that the snapshot anchors on — that marker machinery is implemented (config_commit_change, config_committed_marker, persisted config-ref).

---

Implementation Approach Options

Option A — Extend the existing checkpoint/upgrade path

Wire vfreeze/vthaw into kvmgr, teach write_snapshot/load_snapshot to serialize/deserialize entities, and in do_upgrade_config_with_bytes (after the marker commits, before restart) checkpoint an entity-bearing snapshot at the marker (term,lsn), embed the config bytes, stream it via snapshot_transfer_s, and gate activation on install. Drop the per-node-replay restart for the upgrade case.

• ✅ Reuses the live transport + COW; minimal new files; one consistent snapshot format.
• ✅ Naturally supersedes the R7 byte-push for upgrades.
• ❌ Bloats persist.c snapshot codec and the already-large config_repl.c upgrade path.
• ❌ Couples the generic checkpoint snapshot format to upgrade-specific embedding (config bytes) unless cleanly factored.

Option B — New "upgrade snapshot" component

A dedicated module owns: materialize-at-marker, embed config, drive transport, activation gate, and the mismatch fence — leaving persist.c to expose only an entity (de)serialize primitive and config_repl.c to call into it.

• ✅ Clean separation; upgrade-specific logic isolated and unit-testable.
• ✅ Keeps config_repl.c/persist.c from growing further.
• ❌ More wiring across persist ↔ new module ↔ ha transport ↔ config_repl.
• ❌ Risk of a second snapshot-format path diverging from the generic checkpoint one.

Option C — Hybrid (recommended)

1. Foundation (in/with entity-bearing-snapshots): wire COW into kvmgr and add the generic entity (de)serialize to write_snapshot/load_snapshot — Option A style, shared by recovery + state transfer.
2. Upgrade layer (this spec): a thin new helper (Option B style) that, given an entity-bearing snapshot capability, materializes at the marker (term,lsn), embeds the config bytes, streams via the existing transport, and adds the activation gate + hash-verify-vs-marker + loud-fence-on-mismatch. Replace the step-9 restart/replay for the upgrade case; keep restart/replay as the fallback only until the foundation lands.

• ✅ Generic snapshot work is reused (recovery, faster rebuild) — not upgrade-only.
• ✅ Upgrade-specific gating/verification stays isolated and testable.
• ✅ Phased: foundation can land + be validated before the upgrade layer flips on.
• ❌ Requires careful coordination across the two specs (format ownership, ordering).

---

Effort & Risk

Dimension	Rating	One-line justification
This spec alone (assuming entity-bearing snapshots exist)	L (1–2 wks)	Transport + marker + COW exist; new work is materialize-at-marker, config embed, activation gate, hash-verify, mismatch-fence, and the failover-interleave regression.
Including the gating dependency	XL (2+ wks)	entity-bearing-snapshots is itself non-trivial (entity serialization under per-object locks, COW wiring, recovery read-back) and must land first.
Risk	High	Hard blocker not yet started; touches consensus-adjacent state at a (term,lsn) boundary interleaved with failover (R4.3); snapshot/promotion-truncate/catch-up interaction is unverified; must preserve the never-self-terminate invariant.

---

Recommendations for Design Phase

Preferred approach: Option C (Hybrid) — land entity-bearing snapshots as a shared foundation (COW into kvmgr + generic entity (de)serialize in write_snapshot/load_snapshot), then add a thin upgrade layer that materializes at the marker, embeds the config, reuses snapshot_transfer_s, and adds activation gating + hash-verify + loud-fence.

Key decisions to settle in design:

1. Spec sequencing — co-scope entity-bearing-snapshots into this spec's delivery, or hard-gate this spec behind it? (Recommend gate; promote to design only once the foundation has a tasks plan — matches roadmap.md.)
2. Snapshot format — one unified entity-bearing format used by both recovery and upgrade, with the config-bytes embed as an optional section? Avoid a second format.
3. Materialize-at-marker semantics — snapshot must reflect state at exactly the marker (term,lsn), not "latest checkpoint." Define how the freeze window aligns to the committed marker LSN.
4. Activation gate — bind "new config active" to "upgrade snapshot for this (term,lsn) installed + hash-verified vs config_committed_marker"; mismatch ⇒ loud serve-fence + retry (never exit — controller/CLAUDE.md).
5. Retire the per-node restart/replay for the upgrade case (step 9, config_repl.c:1657-1676) once the snapshot path is proven.

Research-Needed (carry forward):

• Upgrade × failover interleave (R4.3): how does a mid-upgrade entity snapshot interact with promotion-truncate, (term,lsn) reconcile, and ACK-driven catch-up if the leader dies mid-install? Define the recovery story.
• COW single-generation constraint vs. a checkpoint racing an upgrade snapshot (only one freeze/thaw consumer allowed — zarray.c / roadmap note).
• Build snapdump + walcompare (roadmap Hardening §) first — they are the RED/GREEN assertion tools for R5 (entity-less snapshot detection + cross-node WAL/state diff).

---

Update — 2026-07-21 (dependencies resolved; requirements extended; harness gap analysis)

This section is current as of 2026-07-21 and supersedes the stale parts of the 2026-06-24 analysis above (kept for history). Do not re-derive the gating status or tool inventory from the sections above without reading this first.

Gating blocker — RESOLVED

• entity-bearing-snapshots — satisfied 2026-07-09 by ha-entity-snapshots (21/21 tasks, fully approved). It shipped write_snapshot/load_snapshot (persist.c) as a direct Cap'n-Proto entity-encode/decode of job and queue records at checkpoint time — not the COW xzarray_freeze/thaw path this research doc's Option A/C assumed (that machinery is still dormant, kvmgr.c:297). Design must build on the real write_snapshot/load_snapshot interface, not the COW one.
• ha-term-safety (owns the (term,lsn) config-change marker this spec anchors on) — implementation-complete, GO-validated 2026-07-17, 46/46 acceptance criteria mapped, live chaotic soak clean.
• Net effect: this spec is unblocked. spec.json.blocked_on should be cleared and the phase can move to design once requirements are approved.

Tool inventory — CORRECTED (contradicts the "unbuilt" note above)

snapverify, snapdiff, and walcompare/walcompare_report all already exist (src/controller/testsuite/), built 2026-07-02 during bug-1386/1389 triage. The line-57/176 framing above ("planned", "unbuilt", "build first") is stale — do not re-propose building them.

• snapverify_path — structural-only checkpoint-file validator (magic, format_version, frame walk to EOF). Reports has_cfgref as a bool but does not decode or assert the tag/hash value.
• snapdiff_paths — byte-for-byte comparator between two checkpoint files, with chunk-boundary diagnosis. Built for source-vs-received comparison but directly reusable for a determinism check (two independently materialized snapshots at the same boundary should be byte-identical).
• walcompare/walcompare_report — cross-member committed-prefix divergence detector, compares (term, pcrc) per LSN, and already has a per-member checkpoint floor concept (an LSN below a member's own installed-snapshot floor is "legitimately compacted," never flagged divergent — ha-entity- snapshots task 7.1 / raft-ha-core-safety task 5.10a). It does not compare active configuration across members.

Requirements extended this session (2026-07-21)

requirements.md gained content not reflected in this doc's "Recommendations" section (items 1-5 above predate these):

• Req 1.5 — snapshot materialization is triggered synchronously by marker-apply but executed through the existing async checkpoint mechanism (persist_checkpoint/schedule_checkpoint/write_snapshot, delaymgr-driven, CAS-guarded single-in-flight), not inlined into the raft apply path — answers the "should this be one atomic Raft FSM operation" question: the marker commit already is one FSM op; materialization should be triggered by it but run via the existing decoupled checkpoint mechanism (never block a vtask handler); cross-node install must never be — it's inherently async.
• Req 2.4 — every checkpoint taken after a config change embeds the current config, not only the marker-triggered one, so a member joining after the upgrade snapshot has rotated away (rotate_snapshots, snap_keep_n) still gets config + state together. Piggybacks on the existing scan_latest_committed_config_ref (persist.c:1999), which already re-stamps the latest committed config reference into every checkpoint — Req 2 extends that from a reference to the full embedded bytes.
• Req 5 (new) — Crash-safety and idempotent resumption: deterministic snapshot content (any member's independent materialization at the same (term,lsn,config-hash) is identical), crash-safe resumption if a member dies mid-materialization, idempotency guard reusing the existing checkpoint_in_progress CAS, and leader-change-mid-distribution resumption.
• Req 6.3 (was Req 5.2) — regression test must inject a crash/restart and a leadership change specifically between marker-commit and snapshot-install- complete, not just rely on general chaotic-kill timing.

Invariant risk carried forward from this session's controller-invariant check (2026-07-21)

Checked the new requirements (esp. Req 3's activation gate) against controller/CLAUDE.md's TLA+ invariants and Protocol Verification Checklist. One real precedent-risk, already fixed at the requirements level (Req 1.8) but flagging for design/implementation attention:

• bug-1418 (open, NOT FIXED) is the same failure shape this spec's Req 3 risks reintroducing in a new flavor. bug-1418: a newly-elected primary whose apply is behind (apply_divergent=1) hits a liveness deadlock — the existing promotion gate blocks it hard, repeatedly, rather than degrading gracefully; the (unimplemented) proposed fix is "become primary, self-apply, then serve." Req 3.2/3.3 of this spec assume exactly that graceful pattern for materialization-lag instead of apply-lag. Since the apply-lag version of this pattern is proven broken in production today, design must not implement Req 3's gate as a hard promotion block (Req 3.1 already commits to this — no separate promotion-eligibility mechanism) and must verify the "keep serving under last-verified config" fallback (Req 3.3) is genuinely implemented, not degrade into the same stall bug-1418 exhibits.
• Root cause of the need for Req 1.8 (apply must not itself re-home entities; activation is a separate, later-gated step): without that split, Req 3.3 ("keep serving under last-verified configuration" while materialization is incomplete) is incoherent — there would be no old interpretation left to serve under once apply had already mutated live entity state. This is the same family of apply-ordering subtlety behind bug-1417/1418/1419.
• Design-verification item (unconfirmed, not yet checked in code): does persist_load_snapshot/load_snapshot advance durable_lsn (not just checkpoint_lsn/current_lsn) when a member catches up via the transport- install path (Req 1.2's second branch)? If not, a snapshot-caught-up member may not correctly count toward commit majority for LSNs at/below the snapshot boundary even though it durably holds that state via the snapshot file rather than raw WAL — a possible Commit Durability invariant gap specific to the transport-install path. Verify in design before assuming Req 1.2's two paths are equally safe.

Harness gaps for design phase (new, from this session's analysis)

Three items, only two are genuinely new build work:

1. No cross-member config comparator (new gap). walcompare diffs WAL only. Req 4.2 ("every member's active configuration identical") has no assertion tool. Extend walcompare (or add a sibling) to decode+compare the embedded config tag/hash (and eventually bytes) per member, mirroring how it already compares (term,pcrc). Also extend snapverify to decode/assert the cfgref tag+hash value, not just report its presence.
2. No deterministic fault-injection at the marker-apply → forced-checkpoint- durable window (new gap, needed for Req 5.2/6.3). /xce-test-ha --chaotic kills at random points — could hit this narrow window eventually, too flaky for a CI regression gate. Precedent for the right shape of hook already exists twice in config_repl.c: ha_stream_body_read_fault_hook, xce_snapshot_chunk_hook (both #ifdef XCE_TESTING seams). Add an equivalent hook at the marker-apply/forced-checkpoint boundary.
3. No deterministic leadership-change-mid-distribution injection (new gap, Req 5.4/6.3). Same precedent pattern as #2 — a targeted "kill/step-down primary exactly after marker commits, before all standbys ACK install," not reliance on chaos-soak probability.
4. Reusable, no new work needed: snapdiff_paths for the Req 5.1 determinism check; /xce-test-ha --force-snapshot for Req 2.4's fresh-join-after-rotation (already exercises wiped-node-rejoins-empty via bug-1386's own test mode — just needs its assertions extended once config embedding lands); config_commit_change/do_upgrade_config_with_bytes (bug-970) already wires WAL_CONFIG_MARKER into the live upgrade path, so the bug-857 repro (Req 6.2) is triggerable today with existing tooling — no blocker there.

---

Update — 2026-07-23 (design discovery; requirements extended to 7 requirements/36 ACs)

Requirements went through several rounds of review this session (host/queue admin-state frames, bug-1418 re-verified fixed, bootstrap-vs-upgrade boundary, Design Invariants section, Requirement 7 added for new-mechanism regression coverage). This update captures the DESIGN-level discovery done to write design.md, current as of 2026-07-23, extension-scope (light discovery — existing system, no new external dependency).

Discovery: exact hook points for the marker → materialization trigger (Req 1.1, 1.5, 1.8)

The single unified apply seam is apply_committed_entry (persist.c:2751-3132), confirmed as the ONLY live path for entity mutation on every member — primary online-apply (persist_apply_pump_run → persist_apply_pump_kick, vtask-pool dispatched, persist.c:3383), standby normal apply (ha_standby_advance_committed_and_apply, heartbeat-learned commit position, ha.c:2596-2613/2688-2700 — supersedes the older "apply on receipt" bug-1043 fix as of 2026-07-10), and boot/restart replay (persist.c:3920). The sibling dispatcher apply_wal_entry (persist.c:2583-2714) has zero live callers (grep-confirmed) — fully superseded, dead code, not a second seam to worry about. WAL_CONFIG_MARKER currently falls into apply_committed_entry's generic default: no-op (persist.c:3126-3131) and apply_wal_entry's equivalent no-op (persist.c:2697-2707, irrelevant given no callers). Design decision: add one new explicit case WAL_CONFIG_MARKER: in apply_committed_entry only — it fires identically for primary and standby, satisfying Req 1.2's "every member... via that member's own local materialization once it has applied the marker itself" for free, with no per-role branching needed.

Today's restart+replay behavior being replaced lives in do_upgrade_config_with_bytes (config_repl.c:1591-1773, step 9, config_repl.c:1753-1772): atomic_store(&c->restart_requested, 1) + cbuf_put(&c->wbuf, &stop_val). An existing NOTICE comment at config_repl.c:1737-1751 already names this exact spec by path and describes the target end-state verbatim — a strong signal the upgrade path's own maintainers already anticipated this design. No function-pointer hook exists at that NOTICE site (grepped, none) — it is a plain fall-through, confirming the real trigger point is the apply_committed_entry case above, not this step-9 comment site itself. Once the new case lands, step 9's restart/replay becomes dead code for the upgrade path and should be removed (not left as a parallel fallback — Design Invariant "never leave two mechanisms doing the same job").

Discovery: checkpoint triggering — no forced/immediate entry point exists yet (Req 1.5)

persist_checkpoint() (persist.c:2262-2321) is unconditionally synchronous/blocking (fdatasync-backed write_snapshot, directory scans) and has exactly one production trigger path today — schedule_checkpoint() (persist.c:2238-2256), fired only by a WAL-record-count threshold inside persist_wal_async() (persist.c:921-928). No wall-clock periodic timer exists in the live code (an older design doc's checkpoint_requested/persist_checkpoint_timer_fn was never implemented — grep-confirmed absent). No force_checkpoint()/persist_checkpoint_now() entry point exists — but a second direct, unconditional call to persist_checkpoint() already exists at ha.c:3308 inside ha_snapshot_do_send() ("Req 4.1: create a FRESH checkpoint..."), proving the function itself already tolerates being force-invoked outside the record-count path (it already handles the -EBUSY a concurrent scheduled checkpoint can return, via the shared checkpoint_in_progress CAS, persist.h:415).

Two existing patterns already solve "trigger without blocking the caller" in this codebase — delay_task/delaymgr enqueue (what schedule_checkpoint does; the actual checkpoint I/O then runs on the delaymgr thread itself, per direct trace of delay_worker/run_delay_tasks, delay.c:60/cluster.c:988-991 — delaymgr has no thread-pool hand-off) and the dedicated slow-lane executor (what ha_snapshot_do_send uses, ha.c ~6120-6160, specifically because persist_checkpoint is blocking/unbounded — this is the pattern ha-async-io.md:29 documents as the bug-1198 fix for inline-on-iomgr checkpoint I/O). Design decision: the new WAL_CONFIG_MARKER case in apply_committed_entry calls persist_checkpoint() directly, not via either existing indirection. This is safe specifically because apply_committed_entry (via the apply pump) already runs on its own vtask-pool worker thread, never the iomgr or delaymgr thread — it IS the async-dispatched context, so no further hand-off is needed for THIS call site (unlike schedule_checkpoint's WAL-threshold trigger, which fires from arbitrary vtask handlers appending WAL entries and genuinely needs delaymgr's async return). This gives Req 1.5's "immediate and deterministic" trigger for free — no new delay/lane plumbing required — while the CAS (checkpoint_in_progress) already guards against a racing periodic checkpoint.

Note for design (not a requirements gap, an implementation-order note): scan_latest_committed_config_ref/write_snapshot's existing persist_checkpoint call sequence (persist.c:2301/2305) always stamps checkpoint_lsn = last_applied_lsn (persist.c:2292-2295) — i.e. a marker-triggered forced checkpoint calling persist_checkpoint() at the moment apply_committed_entry handles the marker will naturally pin checkpoint_lsn to the marker's own (term,lsn), satisfying Req 1.1's "materialize at that marker's boundary" without any new LSN-targeting parameter — persist_checkpoint() doesn't need to accept an explicit target LSN; it just needs to run at the right moment, which the new apply-time case guarantees by construction.

Discovery: the activation gate has a direct, reusable precedent (Req 3)

ha_dispatch_gate (ha.c:3809-3975, single call site cluster.c:276, gating every wire request — client and peer alike, both HA replication opcodes and apiserver-relayed job/queue/host opcodes) already composes multiple independent boolean fences in sequence for the CTRL_PRIMARY role: role check, then a config-serve-fence check (config_serve_fenced(ha->raft_c), ha.c:3827), then a write-quorum check. This is a direct structural precedent already in production for exactly what Req 3 needs — a config-dependent, state-changing-request fence that is orthogonal to role/quorum, with its own opcode allowlist (replication + read-only ops stay admitted while fenced).

The existing config_serve_fence (ha.h:307-322, set/cleared by config_activate_committed in config_repl.c:304-377, hash-verifies committed marker bytes against locally-held config bytes) is the closer of the two fence precedents found (the other being recovery_unrecoverable, which only gates promotion, not live serving) because it already: (i) blocks client-request serving via ha_dispatch_gate, (ii) also blocks promotion (ha.c:1425-1440, ha_config_promotion_fenced()), and (iii) self-clears via a fully-wired, non-blocking retry loop (config_schedule_activate_retry → delaymgr trampoline → vtask does the actual blocking verify I/O, config_repl.c:415-507), triggered from the commit-advance callback (ha.c:2197-2198) — the exact "loud ERROR once, stay alive, keep retrying, never self-terminate" shape controller/CLAUDE.md mandates, already proven in production. Design decision: add a new atomic fence field on ha_state_t (sibling to config_serve_fence, e.g. upgrade_snapshot_fence), a matching setter/verify function mirroring config_activate_committed's hash-verify-then-clear shape, and one more fence check inside ha_dispatch_gate's CTRL_PRIMARY branch (composed the same way the existing config-serve-fence check already is — not folded into that same field, since the two conditions are logically independent: config-serve-fence is about byte-mismatch, the new fence is about materialization-not-yet-complete for an otherwise-matching config) — reusing config_schedule_activate_retry's exact trampoline shape for the new fence's own retry loop, and ha_config_promotion_fenced()'s pattern for gating promotion the same way.

Discovery: durable_lsn is NOT advanced by snapshot-install — real but bounded gap (Req 1.2, 5.x)

Traced definitively (grep + direct read of every function in the call path): ha_handle_snapshot_chunk (ha.c:3014-3171) → persist_load_snapshot/load_snapshot (persist.c:2456-2575) never touches p->durable_lsn anywhere — only checkpoint_lsn/current_lsn/cfgref_* get set. This is asymmetric with the boot-recovery path (persist_recover, persist.c:4338-4353), which DOES re-derive durable_lsn from the recovered WAL tip. Verdict: real, confirmed gap, but bounded to this node's OWN promotion-eligibility/self-advertised-freshness bookkeeping during a narrow live-catch-up window (persist_durable_floor consumers: pre-promotion gate ha.c:1318-1352, promotion truncate-floor clamp ha.c:1259-1262, self-advertised candidate position ha.c:4540-4546) — the existing fence-and-retry machinery (ha_arm_promotion_fence_timer) already handles this safely (conservative refusal to self-promote, never corruption). It is NOT a commit-durability/majority-counting safety gap for OTHER members counting a snapshot-caught-up peer, because majority_candidate_locked (commit.c:167-172) already excludes any standby with in_catch_up != 0 from majority evaluation, and that flag is cleared only by a real WalAck reaching catchup_derive_target (repl.c:713-739, via ha_walack_on_ack) — a separate, already-safe gate keyed off real acks, not the snapshot event. Design decision: out of this spec's scope to fix (it predates this spec, applies to any snapshot-install today including plain recovery-driven catch-up, not upgrade-specific) — but flag it as a design.md Risk/Note so a future spec can pick it up (advancing durable_lsn on snapshot-install the same way boot-recovery does would close it cleanly, mirroring persist.c:4338-4353's ASSIGN semantics).

Discovery: test tooling — three reusable as-is, two confirmed genuine new-build gaps

Confirmed via direct read (not just trusting the 2026-07-21 update above, which was itself a research note, not a verified read):

• snapverify_path (testsuite/snapverify.c) — confirmed has_cfgref is presence-only (header[16]), never decodes tag/sha; zero test assertions on it exist today. Needs extending to decode/assert tag+hash for Req 4.2/Req 2 coverage.
• snapdiff_paths (testsuite/snapdiff.c) — confirmed reusable as-is for Req 5.1's determinism check (any two paths, no transport-specific assumptions).
• walcompare/walcompare_report (testsuite/walcompare.c) — confirmed zero references to cfgref_tag/cfgref_sha/config bytes anywhere in the file; compares only (lsn, term, pcrc). Req 4.2 (active-config identity across members) has no assertion tool — needs either extending walcompare or a sibling tool.
• Fault-injection precedent, exact shape confirmed: ha_stream_body_read_fault_hook (config_repl.c:2104-2106, wired at config_repl.c:2183-2193) and xce_snapshot_chunk_hook (config_repl.c:2847, wired at config_repl.c:2937-2949) are both #ifdef XCE_TESTING-gated, NULL-by-default, file-scope function-pointer globals, extern-declared and set/reset around the driving call in the test file. This is the exact shape the two NEW deterministic hooks (Req 7.2's marker-apply→forced-checkpoint-durable window, Req 7.2/5.4's leadership-change-mid-distribution window) should mirror — a hook inside the new apply_committed_entry WAL_CONFIG_MARKER case (or wherever the cross-member install-wait loop lives) taking identifying args (marker lsn/term) so a test can force-fail/force-delay at that exact boundary instead of relying on chaotic-soak timing luck.
• Test fixture pattern, exact shape confirmed: test_apply_committed_entry.c (bare in-memory cluster_t, no disk, direct apply_committed_entry() calls, spy hooks counting HOOK_AFTER_EVENT fires) and test_replay_apply_seam.c (real disk-backed persist_t, two-phase write-then-recover, proving replay routes through the same seam) are the two patterns new host-frame-restore and marker-apply tests should extend — both already have a tace_encode_worker/trr_encode_worker local-capnp-encode helper (mirroring persist.c's file-static capnp_encode_worker) that a new host-admin-frame test can copy the idiom from.

Design Synthesis (per rules/design-synthesis.md)

1. Generalization: The host-admin-state frame (Req 1.6/1.9/1.10) and the queue-admin-state frame (existing, now migrated to capnp per Req 1.9) are the same underlying shape — (name, status) with independently-extensible fields. Rather than two hand-written near-duplicate collect/restore function pairs, design one generic collect_admin_state_frames(table, status_mask)/restore_admin_state_frame(table, name, status, create_if_missing_fn) pair parameterized over which table (workermgr/qmgr) and which create-if-missing behavior (hosts: makeWorker-based placeholder; queues: none, always pre-provisioned) applies — the two concrete restore functions (restore_host_state, restore_queue_state) become thin instantiations, not independent implementations. This directly follows Design Invariant 1 (classify config- vs join-provisioned) by construction: the generic function takes the classification as a parameter rather than assuming one.

2. Build vs. Adopt: Every major piece here is "adopt existing internal mechanism," not "build new" — the snapshot transport (snapshot_transfer_s/ha_handle_snapshot_chunk), the checkpoint/CAS machinery (persist_checkpoint/checkpoint_in_progress), the fence-and-retry pattern (config_serve_fence), the placeholder-entity pattern (lost_and_found's makeWorker() precedent), and the fault-injection hook shape (ha_stream_body_read_fault_hook) are all reused verbatim rather than reimplemented. The only genuinely NEW machinery is: the WAL_CONFIG_MARKER case in apply_committed_entry, the new fence field + verify function (structurally mirroring config_serve_fence but a distinct field), the config-bytes-embed in the snapshot header/entity payload, the HostAdminState/QueueAdminState capnp structs, and the two new test hooks. No external library or new dependency is involved anywhere in this spec.

3. Simplification: Considered whether the new fence needs to be a SEPARATE atomic from config_serve_fence — confirmed yes (the two conditions are logically independent: hash-mismatch vs materialization-incomplete, and Req 3.4 needs to distinguish "keep serving under last-verified config" from "no verified config exists yet," which requires knowing WHICH fence is active). Considered whether apply_wal_entry needs a matching WAL_CONFIG_MARKER case for symmetry with apply_committed_entry — confirmed no, it has zero live callers, touching it would be pure speculative-symmetry work with no runtime effect (Design Invariant would call this exactly the "convenient but unjustified" mistake to avoid).

Risks carried into design.md

• The narrow window between schedule_checkpoint's CAS release (checkpoint_task_fn, persist.c:2222) and persist_checkpoint's own re-acquire (persist.c:2265) allows a second concurrent trigger to slip in and start its own persist_checkpoint() call — pre-existing, not introduced by this spec, but the new marker-triggered call site adds a second caller into this same race window and should be noted.
• durable_lsn not advancing on snapshot-install (above) — pre-existing, out of scope, flagged for a future spec.

---

Cross-Spec Gap Analysis: admin-config-tool × config-upgrade-snapshot × clusterconfig-schema-split (2026-07-28)

1. Current State Investigation

All three specs touch the same controller-side config surface (controller.capnp's ClusterConfig, config_repl.c's OP_upgradeConfig handler, cluster.c's dispatch gate) from different angles:

Spec	Phase	Scope
clusterconfig-schema-split	implementation-complete	Remove per-instance/dead fields (port, candidates, perf-tuning) from ClusterConfig
admin-config-tool	implementation-complete (blocked_on stale — see §2)	xceadmin CLI: build/push cluster.bin against the controller
config-upgrade-snapshot	tasks-generated, 18/19 actionable tasks [x]	Snapshot the config alongside cluster state at every upgrade boundary; activation fencing

Verified current schema state (src/schema/controller.capnp:712-726): ClusterConfig has exactly 13 fields (@0–@12: clusterName … configGeneration), matching clusterconfig-schema-split's Requirement 3.1 target exactly. MasterCandidate/HashZArrayConfig/ResmgrConfig are gone (Requirements 1.6/2.5). config-upgrade-snapshot's own schema additions (HostAdminState, QueueAdminState, lines 769/774) are separate message types, not fields on ClusterConfig itself — no ordinal collision or renumbering conflict between the two specs' schema work.

Verified dispatch-gate state (src/controller/cluster.c:669-694): the OPCode_upgradeConfig polarity-inversion bug that admin-config-tool's spec.json.blocked_on and tasks.md (line 155) both cite as blocking real-cluster config push E2E validation is already fixed — commit 959b0cda (bug-1468, landed 2026-07-25 as part of config-upgrade-snapshot task 8.6). The fix's own comment in cluster.c explicitly cross-references admin-config-tool: "Independently diagnosed and documented as a blocking risk by the admin-config-tool spec (design.md 'Boundary Commitments' / 'Risks') before config-upgrade-snapshot task 8.6 needed a live OP_upgradeConfig sender and hit the same wall."

2. Gap Found: admin-config-tool's blocked_on is stale documentation, not a live blocker

admin-config-tool/spec.json's blocked_on array still lists:

"controller OP_upgradeConfig dispatch defect (cluster.c:667-679 vs ha.c:3926-3951) blocks real-cluster config push E2E validation"

This is the exact same defect as bug-1468, fixed 2026-07-25. Additionally, the same controller-side OP_upgradeConfig path this blocker describes has since been exercised live, repeatedly, and successfully by config-upgrade-snapshot's own --config-upgrade soak (.claude/skills/xce-test-ha/run-ha-soak.sh) — dozens of live 3-node runs today alone, all showing correct dispatch, config replication, majority-ack, and WAL_CONFIG_MARKER commit. The only difference is the client: the soak drives the RPC via xce_test's hand-rolled upgradeconfig command (built for config-upgrade-snapshot task 8.6, since no other live OP_upgradeConfig client existed at the time), not xceadmin's Python config push. But the controller-side code path is identical for both clients — the dispatch gate doesn't distinguish caller identity.

• Gap classification: Missing (stale tracking data, not missing capability). admin-config-tool's actual E2E validation task (referenced in tasks.md line 155) can very likely be marked done now, or re-run directly with xceadmin config push against a live cluster to close it out formally — the blocking condition itself no longer exists.
• Recommendation for design/task phase: Clear admin-config-tool/spec.json's blocked_on entry (or mark it resolved with a dated note, matching this codebase's own convention — e.g. clusterconfig-schema-split's already-resolved entries and config-upgrade-snapshot's own dated Implementation Notes). If the E2E task was left unchecked pending this, run xceadmin config push against a real deployed cluster (/xce-deploy-ha) once to get first-hand xceadmin-specific confirmation (not just proof the shared controller path works) — the CLI's own TLS/redirect/framing code (Requirements 2–3) has never been proven against the real handler, only mock-tested (per tasks.md's own note).

3. Gap Found (residual, this session): config-upgrade-snapshot task 8.6 — new commit-timeout issue post-bug-1477

config-upgrade-snapshot's own remaining task (8.6, live chaotic-upgrade soak) was blocked on bug-1472 → then bug-1477 (both now fixed and verified this session — see .wolf/buglog.json). Re-running --config-upgrade after the bug-1477 fix shows the soak's core assertions (checkpoint-identity, config-identity, host-admin-status survival — the things bug-1471/1472/1477 blocked) all PASS for the first time. But the same run then fails later, in the force-elect-rotation step, with repeated commit timeouts (rc=-110) on the primary — a step this soak scenario has never previously reached (every prior attempt failed earlier). This is very likely a fourth pre-existing, newly-exposed defect in the same family as bug-1471/1472/1474/1477 (surfaced by clearing the earlier blockers, not caused by them) — not yet root-caused, flagged for the next investigation pass. Task 8.6 remains open pending this.

• Gap classification: Unknown (root cause not yet investigated — flagged, not analyzed, per this skill's "defer deep research to design phase" principle).
• Ownership: Likely UNCLEAR until investigated — could be ha-commit-durability/ha-term-safety/raft-log-replication territory (commit-engine admission capacity or write-quorum under chaos), or a config-upgrade-snapshot-adjacent interaction. Follow this session's established routing discipline (bug-1471→1472→1474→1477 precedent): root-cause first, only fix in-place if narrowly scoped, otherwise route to the owning spec and keep task 8.6 blocked.

4. No gap found: clusterconfig-schema-split × admin-config-tool integration

admin-config-tool/requirements.md's own Adjacent Expectations section already documents this dependency as "RESOLVED, landed" (commit 519e11c), and the verified current schema state (§1 above) confirms it: port/candidates/perf-tuning fields are genuinely absent from ClusterConfig, so xceadmin's Requirements 1.8–1.10 (never source these fields) have nothing to accidentally violate — the fields don't exist to source. No further action needed here.

5. No gap found: clusterconfig-schema-split × config-upgrade-snapshot schema coexistence

Verified directly (§1): the two specs' schema additions/removals touch disjoint parts of controller.capnp (ClusterConfig's field list vs. new standalone HostAdminState/QueueAdminState structs). No ordinal renumbering conflict, no shared struct mutated by both specs in incompatible ways.

Summary Table

#	Finding	Classification	Effort	Risk	Action
1	admin-config-tool blocked_on stale (bug-1468 already fixed)	Missing (tracking, not capability)	S (update spec.json + one live xceadmin config push run)	Low	Clear blocker, optionally run one live E2E confirmation
2	config-upgrade-snapshot task 8.6: new commit-timeout past force-elect step	Unknown (not yet root-caused)	Unknown until investigated	Medium–High (commit/write-quorum path)	Dedicated debug pass before considering 8.6 closeable
3	clusterconfig-schema-split ↔ admin-config-tool	No gap	—	—	None
4	clusterconfig-schema-split ↔ config-upgrade-snapshot schema	No gap	—	—	None

Research Needed (carried forward)

• Root cause of the new post-bug-1477 commit-timeout (finding #2) — not investigated in this pass, flagged only.

Resolved during this pass (no longer open): admin-config-tool/tasks.md was audited directly — all 23 tasks are [x], none gated on the stale blocker as an active checklist item. The E2E gap is purely spec.json.blocked_on + an informational tasks.md note (line 155), not an incomplete task. Confirms finding #1 is pure metadata cleanup, not a reopened implementation gap.