myxce/.kiro/specs/config-upgrade-snapshot/research.md

Gap Analysis — config-upgrade-snapshot (2026-06-24)

Scope

Make a configuration upgrade install one identical, entity-bearing state snapshot (with the new config embedded) on every member at the config-change marker's (term, lsn) boundary, replacing today's config-push + restart + per-node WAL replay (which can materialize divergent state — bug-857). Closes bug-857.

GATING BLOCKER (hard dependency): entity-bearing-snapshots. Today snapshots carry zero entity state (an LSN sentinel). This spec consumes an entity-bearing snapshot it cannot produce. It cannot reach a testable GREEN until entity-bearing-snapshots lands or is co-scoped. Every "Missing" below traces back to this one absence. See §"Gating Blocker".

---

Requirement-to-Asset Map

R1 — Full snapshot at the config-change boundary

Need	Existing asset (file:line)	Gap
Materialize a snapshot at the marker's (term,lsn)	persist_checkpoint / write_snapshot (persist.c:1247-1291) writes a checkpoint snapshot at checkpoint_lsn	Missing — snapshot is a 16-byte sentinel ("XCESNAP\0" + LSN + optional 41-byte config-ref tail); carries no entities. No way to pin it to the marker boundary vs. an arbitrary checkpoint LSN.
Snapshot pins identical materialized state	dual-zarray COW: xzarray_freeze/xzarray_thaw (zarray.c:695-757), test_zarray_cow.c	Missing (dormant) — built + unit-tested but no production callers: kvmgr.c:297 vfreeze=NULL /* wired in P0-2 */, vthaw=NULL.
No member re-derives via replay	upgrade path do_upgrade_config_with_bytes (config_repl.c:1495-1677); NOTICE at step 9 (config_repl.c:1641-1655)	Constraint (current behavior is the bug) — step 9 sets restart_requested; on restart each node replays its own pre-upgrade WAL under the new config (config-dependent via lost_and_found). This is exactly what R1.3 forbids.
Transfer out-of-band, not in WAL payloads	ha_handle_snapshot_chunk (ha.c:2424-2515) + persist_load_snapshot (persist.c:1594); OPCode_snapshotTransfer dispatch-gated (ha.c:2842/2874/2962)	Exists (reusable transport) — chunked, size-checked, atomic snap.part→rename, then persist_load_snapshot. Reusable as-is; it only ever moves a sentinel today.

R2 — Configuration embedded in the upgrade snapshot

Need	Existing asset (file:line)	Gap
Embed new config in snapshot (one artifact)	snapshot config-ref tail (persist.c:1263-1274, read-back 1581-1590): {present, version_tag(8), sha256(32)}	Partial / Missing — only a {tag, sha} reference is embedded, not the config bytes and not the entity state. Need the actual config bytes + entities in one installable artifact.
Supersede ha-term-safety R7 config-push for upgrades	R7 out-of-band byte push driven from config_commit_change (config_repl.c:478-510)	Missing — upgrade still uses the R7 interim byte-push path; no branch that delivers config via the snapshot instead.
Keep config bytes out of WAL payloads	WAL_CONFIG_MARKER is {version_tag, sha256} only — config_marker_encode (config_repl.c:40-69), bytes never in WAL	Already satisfied — marker carries no bytes; invariant holds today.

R3 — Activation gating

Need	Existing asset (file:line)	Gap
Don't serve/activate new config until snapshot installed	install-before-activate already structural in ha_handle_snapshot_chunk (rename+load on full receipt)	Partial — transport installs before load, but there is no activation gate tying "config active" to "upgrade snapshot for this (term,lsn) installed." Today activation = restart-then-replay.
Verify embedded config vs marker content hash	config_committed_marker (config_repl.c:216) yields {tag, sha}; hash-verify-before-activate already the rule for R7 byte push (steering ha-async-io.md §per-entry-term)	Missing for snapshot — the verify-against-marker logic exists for the byte-push artifact; must be extended to the snapshot-embedded config.
Reject + keep retrying on mismatch (no self-terminate)	loud-fence-and-stay-alive invariant (controller/CLAUDE.md "Process Lifecycle")	Constraint — reject path MUST be a loud serve-fence + retry, never exit/abort. New code must honor this.

R4 — Post-upgrade consistency invariant

Need	Existing asset (file:line)	Gap
Identical materialized state across members	none — no entity snapshot to compare	Missing — depends entirely on R1/R2 landing.
Identical active config across members	config_committed_marker derives committed config from (term,lsn)	Partial — config identity is reachable via the marker; state identity is not.
Invariant holds across upgrade × failover interleave	Raft (term,lsn) reconcile + promotion-truncate (ha-term-safety, raft.c)	Unknown — needs design: how a mid-upgrade snapshot interacts with promotion-truncate and catch-up streaming when a leader dies mid-install.

R5 — Reproduction and regression

Need	Existing asset (file:line)	Gap
Reproduce bug-857 divergence (RED)	waldump (testsuite/waldump.{c,h}), /xce-test-ha chaotic-multi-kill soak, planned snapdump/walcompare (roadmap Hardening §)	Partial — waldump exists; snapdump/walcompare (entity-less-snapshot + WAL-divergence detectors) are unbuilt and are the natural assertion tools here.
Pass on fixed code	—	Missing — fix does not exist yet.

---

Gating Blocker — entity-bearing-snapshots (hard, explicit)

This is the single dependency that gates the whole spec.

• Evidence it's absent: write_snapshot (persist.c:1247) and load_snapshot (persist.c:1555) handle only magic + LSN + a {tag,sha} config-ref tail — no job/queue/host/resource entities. The header comment literally says "Full Cap'n Proto serialization is Task 11 when recovery needs to read it back."
• Why it blocks: R1.1, R2.1, R4.1 all require an artifact that carries materialized entity state. Without it, "install the same snapshot on every member" delivers an empty sentinel — divergence is unavoidable.
• What's already paid down (de-risks it): the COW substrate exists (xzarray_freeze/thaw, zarray.c:695-757, O(1) freeze, GC-filtered thaw, unit-tested) and the transport is reusable (ha_handle_snapshot_chunk + persist_load_snapshot). What's missing is the payload: serialize entities under per-object locks during the freeze window into the snapshot file, and deserialize them on load_snapshot. COW is single-generation/slot-level only — one consumer owns freeze/thaw (roadmap §entity-bearing-snapshots).

Consequence for this spec: requirements/design can proceed, but tasks cannot reach implementation until entity-bearing-snapshots is implemented or co-scoped. spec.json.blocked_on already records this.

Also depends on ha-term-safety R7 for the (term,lsn) config-change marker that the snapshot anchors on — that marker machinery is implemented (config_commit_change, config_committed_marker, persisted config-ref).

---

Implementation Approach Options

Option A — Extend the existing checkpoint/upgrade path

Wire vfreeze/vthaw into kvmgr, teach write_snapshot/load_snapshot to serialize/deserialize entities, and in do_upgrade_config_with_bytes (after the marker commits, before restart) checkpoint an entity-bearing snapshot at the marker (term,lsn), embed the config bytes, stream it via snapshot_transfer_s, and gate activation on install. Drop the per-node-replay restart for the upgrade case.

• ✅ Reuses the live transport + COW; minimal new files; one consistent snapshot format.
• ✅ Naturally supersedes the R7 byte-push for upgrades.
• ❌ Bloats persist.c snapshot codec and the already-large config_repl.c upgrade path.
• ❌ Couples the generic checkpoint snapshot format to upgrade-specific embedding (config bytes) unless cleanly factored.

Option B — New "upgrade snapshot" component

A dedicated module owns: materialize-at-marker, embed config, drive transport, activation gate, and the mismatch fence — leaving persist.c to expose only an entity (de)serialize primitive and config_repl.c to call into it.

• ✅ Clean separation; upgrade-specific logic isolated and unit-testable.
• ✅ Keeps config_repl.c/persist.c from growing further.
• ❌ More wiring across persist ↔ new module ↔ ha transport ↔ config_repl.
• ❌ Risk of a second snapshot-format path diverging from the generic checkpoint one.

Option C — Hybrid (recommended)

1. Foundation (in/with entity-bearing-snapshots): wire COW into kvmgr and add the generic entity (de)serialize to write_snapshot/load_snapshot — Option A style, shared by recovery + state transfer.
2. Upgrade layer (this spec): a thin new helper (Option B style) that, given an entity-bearing snapshot capability, materializes at the marker (term,lsn), embeds the config bytes, streams via the existing transport, and adds the activation gate + hash-verify-vs-marker + loud-fence-on-mismatch. Replace the step-9 restart/replay for the upgrade case; keep restart/replay as the fallback only until the foundation lands.

• ✅ Generic snapshot work is reused (recovery, faster rebuild) — not upgrade-only.
• ✅ Upgrade-specific gating/verification stays isolated and testable.
• ✅ Phased: foundation can land + be validated before the upgrade layer flips on.
• ❌ Requires careful coordination across the two specs (format ownership, ordering).

---

Effort & Risk

Dimension	Rating	One-line justification
This spec alone (assuming entity-bearing snapshots exist)	L (1–2 wks)	Transport + marker + COW exist; new work is materialize-at-marker, config embed, activation gate, hash-verify, mismatch-fence, and the failover-interleave regression.
Including the gating dependency	XL (2+ wks)	entity-bearing-snapshots is itself non-trivial (entity serialization under per-object locks, COW wiring, recovery read-back) and must land first.
Risk	High	Hard blocker not yet started; touches consensus-adjacent state at a (term,lsn) boundary interleaved with failover (R4.3); snapshot/promotion-truncate/catch-up interaction is unverified; must preserve the never-self-terminate invariant.

---

Recommendations for Design Phase

Preferred approach: Option C (Hybrid) — land entity-bearing snapshots as a shared foundation (COW into kvmgr + generic entity (de)serialize in write_snapshot/load_snapshot), then add a thin upgrade layer that materializes at the marker, embeds the config, reuses snapshot_transfer_s, and adds activation gating + hash-verify + loud-fence.

Key decisions to settle in design:

1. Spec sequencing — co-scope entity-bearing-snapshots into this spec's delivery, or hard-gate this spec behind it? (Recommend gate; promote to design only once the foundation has a tasks plan — matches roadmap.md.)
2. Snapshot format — one unified entity-bearing format used by both recovery and upgrade, with the config-bytes embed as an optional section? Avoid a second format.
3. Materialize-at-marker semantics — snapshot must reflect state at exactly the marker (term,lsn), not "latest checkpoint." Define how the freeze window aligns to the committed marker LSN.
4. Activation gate — bind "new config active" to "upgrade snapshot for this (term,lsn) installed + hash-verified vs config_committed_marker"; mismatch ⇒ loud serve-fence + retry (never exit — controller/CLAUDE.md).
5. Retire the per-node restart/replay for the upgrade case (step 9, config_repl.c:1657-1676) once the snapshot path is proven.

Research-Needed (carry forward):

• Upgrade × failover interleave (R4.3): how does a mid-upgrade entity snapshot interact with promotion-truncate, (term,lsn) reconcile, and ACK-driven catch-up if the leader dies mid-install? Define the recovery story.
• COW single-generation constraint vs. a checkpoint racing an upgrade snapshot (only one freeze/thaw consumer allowed — zarray.c / roadmap note).
• Build snapdump + walcompare (roadmap Hardening §) first — they are the RED/GREEN assertion tools for R5 (entity-less snapshot detection + cross-node WAL/state diff).