lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(specs): gap re-validation 2026-06-24 (5 HA/raft specs)

Browse source 
/kiro-validate-gap gap-focused sweep. Per-spec gap analysis written to each
spec's research.md (3 new, 2 appended), grounded in file:line evidence:

- config-upgrade-snapshot (new): real open gap — transport + (term,lsn) marker
  + COW substrate exist, but the entity-bearing snapshot PAYLOAD does not
  (16-byte LSN sentinel). HARD-blocked on entity-bearing-snapshots. Closes
  bug-857. Effort L (XL incl. blocker) / Risk High.
- raft-commit-safety (new): CLOSED — G4 Figure-8 commit (raft.c:733) + G6 vote
  freshness (raft.c:157) done, full sim coverage; the "(0,0) stopgap / broken
  build" concern is stale (build green, real vote tuple at cluster.c:435).
- raft-sim-conformance (new): mostly open — raft.tla is reference-only; only
  1 of 4 invariants (Log Matching) checked in code. Missing PRNG/replay,
  message-fault bag, scheduler, 3 invariant checkers, fault switches,
  scenarios. Effort L / Risk Medium.
- ha-repl-hardening (append): RESCOPE/superseded — 31/33 EARS closed; Req 2.3
  + Req 6 obsolete (CTRL_PRIMARY_COMMITTING removed). Residual: docs only.
- ha-convergence-recovery (append): done bar Task 8 (live E2E); bug-1028
  post-failover availability resolved 2026-06-23.

Skipped ha-elect-ctx-lifecycle (no requirements.md).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
This commit is contained in:
Rongsong Shen 2026-06-24 10:04:06 +00:00
parent 50b976aedc
commit 05febbc9a7
5 changed files with 570 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

177
.kiro/specs/config-upgrade-snapshot/research.md Normal file
View file

@ -0,0 +1,177 @@
# Gap Analysis — config-upgrade-snapshot (2026-06-24)
## Scope
Make a configuration upgrade install one identical, **entity-bearing** state snapshot
(with the new config embedded) on every member at the config-change marker's
`(term, lsn)` boundary, replacing today's config-push + restart + per-node WAL
replay (which can materialize divergent state — **bug-857**). Closes bug-857.
> **GATING BLOCKER (hard dependency): `entity-bearing-snapshots`.**
> Today snapshots carry **zero entity state** (an LSN sentinel). This spec
> *consumes* an entity-bearing snapshot it cannot produce. It **cannot reach a
> testable GREEN until `entity-bearing-snapshots` lands or is co-scoped.** Every
> "Missing" below traces back to this one absence. See §"Gating Blocker".
---
## Requirement-to-Asset Map
### R1 — Full snapshot at the config-change boundary
| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Materialize a snapshot at the marker's `(term,lsn)` | `persist_checkpoint` / `write_snapshot` (`persist.c:1247-1291`) writes a checkpoint snapshot at `checkpoint_lsn` | **Missing** — snapshot is a 16-byte sentinel (`"XCESNAP\0"` + LSN + optional 41-byte config-ref tail); carries no entities. No way to pin it to the marker boundary vs. an arbitrary checkpoint LSN. |
| Snapshot pins identical materialized state | dual-zarray COW: `xzarray_freeze`/`xzarray_thaw` (`zarray.c:695-757`), `test_zarray_cow.c` | **Missing (dormant)** — built + unit-tested but **no production callers**: `kvmgr.c:297` `vfreeze=NULL /* wired in P0-2 */`, `vthaw=NULL`. |
| No member re-derives via replay | upgrade path `do_upgrade_config_with_bytes` (`config_repl.c:1495-1677`); NOTICE at step 9 (`config_repl.c:1641-1655`) | **Constraint (current behavior is the bug)** — step 9 sets `restart_requested`; on restart each node replays its own pre-upgrade WAL under the new config (config-dependent via `lost_and_found`). This is exactly what R1.3 forbids. |
| Transfer out-of-band, not in WAL payloads | `ha_handle_snapshot_chunk` (`ha.c:2424-2515`) + `persist_load_snapshot` (`persist.c:1594`); `OPCode_snapshotTransfer` dispatch-gated (`ha.c:2842/2874/2962`) | **Exists (reusable transport)** — chunked, size-checked, atomic `snap.part`→rename, then `persist_load_snapshot`. Reusable as-is; it only ever moves a sentinel today. |
### R2 — Configuration embedded in the upgrade snapshot
| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Embed new config in snapshot (one artifact) | snapshot config-ref tail (`persist.c:1263-1274`, read-back `1581-1590`): `{present, version_tag(8), sha256(32)}` | **Partial / Missing** — only a `{tag, sha}` **reference** is embedded, not the config **bytes** and not the entity state. Need the actual config bytes + entities in one installable artifact. |
| Supersede ha-term-safety R7 config-push for upgrades | R7 out-of-band byte push driven from `config_commit_change` (`config_repl.c:478-510`) | **Missing** — upgrade still uses the R7 interim byte-push path; no branch that delivers config via the snapshot instead. |
| Keep config bytes out of WAL payloads | `WAL_CONFIG_MARKER` is `{version_tag, sha256}` only — `config_marker_encode` (`config_repl.c:40-69`), bytes never in WAL | **Already satisfied** — marker carries no bytes; invariant holds today. |
### R3 — Activation gating
| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Don't serve/activate new config until snapshot installed | install-before-activate already structural in `ha_handle_snapshot_chunk` (rename+load on full receipt) | **Partial** — transport installs before load, but there is **no activation gate** tying "config active" to "upgrade snapshot for this `(term,lsn)` installed." Today activation = restart-then-replay. |
| Verify embedded config vs marker content hash | `config_committed_marker` (`config_repl.c:216`) yields `{tag, sha}`; hash-verify-before-activate already the rule for R7 byte push (steering `ha-async-io.md` §per-entry-term) | **Missing for snapshot** — the verify-against-marker logic exists for the byte-push artifact; must be extended to the snapshot-embedded config. |
| Reject + keep retrying on mismatch (no self-terminate) | loud-fence-and-stay-alive invariant (`controller/CLAUDE.md` "Process Lifecycle") | **Constraint** — reject path MUST be a loud serve-fence + retry, never `exit`/`abort`. New code must honor this. |
### R4 — Post-upgrade consistency invariant
| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Identical materialized state across members | none — no entity snapshot to compare | **Missing** — depends entirely on R1/R2 landing. |
| Identical active config across members | `config_committed_marker` derives committed config from `(term,lsn)` | **Partial** — config identity is reachable via the marker; **state** identity is not. |
| Invariant holds across upgrade × failover interleave | Raft `(term,lsn)` reconcile + promotion-truncate (`ha-term-safety`, `raft.c`) | **Unknown** — needs design: how a mid-upgrade snapshot interacts with promotion-truncate and catch-up streaming when a leader dies mid-install. |
### R5 — Reproduction and regression
| Need | Existing asset (file:line) | Gap |
|------|----------------------------|-----|
| Reproduce bug-857 divergence (RED) | `waldump` (`testsuite/waldump.{c,h}`), `/xce-test-ha` chaotic-multi-kill soak, planned `snapdump`/`walcompare` (roadmap Hardening §) | **Partial** — waldump exists; `snapdump`/`walcompare` (entity-less-snapshot + WAL-divergence detectors) are **unbuilt** and are the natural assertion tools here. |
| Pass on fixed code | — | **Missing** — fix does not exist yet. |
---
## Gating Blocker — entity-bearing-snapshots (hard, explicit)
This is the single dependency that gates the whole spec.
- **Evidence it's absent:** `write_snapshot` (`persist.c:1247`) and `load_snapshot`
(`persist.c:1555`) handle only magic + LSN + a `{tag,sha}` config-ref tail —
**no job/queue/host/resource entities**. The header comment literally says
*"Full Cap'n Proto serialization is Task 11 when recovery needs to read it back."*
- **Why it blocks:** R1.1, R2.1, R4.1 all require an artifact that carries
materialized entity state. Without it, "install the same snapshot on every member"
delivers an empty sentinel — divergence is unavoidable.
- **What's already paid down (de-risks it):** the COW substrate exists
(`xzarray_freeze`/`thaw`, `zarray.c:695-757`, O(1) freeze, GC-filtered thaw,
unit-tested) and the **transport** is reusable (`ha_handle_snapshot_chunk` +
`persist_load_snapshot`). What's missing is the **payload**: serialize entities
under per-object locks during the freeze window into the snapshot file, and
deserialize them on `load_snapshot`. COW is single-generation/slot-level only —
one consumer owns freeze/thaw (roadmap §entity-bearing-snapshots).
**Consequence for this spec:** requirements/design can proceed, but **tasks cannot
reach implementation** until `entity-bearing-snapshots` is implemented or
co-scoped. `spec.json.blocked_on` already records this.
Also depends on **`ha-term-safety` R7** for the `(term,lsn)` config-change marker
that the snapshot anchors on — that marker machinery **is implemented**
(`config_commit_change`, `config_committed_marker`, persisted config-ref).
---
## Implementation Approach Options
### Option A — Extend the existing checkpoint/upgrade path
Wire `vfreeze`/`vthaw` into `kvmgr`, teach `write_snapshot`/`load_snapshot` to
serialize/deserialize entities, and in `do_upgrade_config_with_bytes` (after the
marker commits, before restart) checkpoint an entity-bearing snapshot at the
marker `(term,lsn)`, embed the config bytes, stream it via `snapshot_transfer_s`,
and gate activation on install. Drop the per-node-replay restart for the upgrade case.
- ✅ Reuses the live transport + COW; minimal new files; one consistent snapshot format.
- ✅ Naturally supersedes the R7 byte-push for upgrades.
- ❌ Bloats `persist.c` snapshot codec and the already-large `config_repl.c` upgrade path.
- ❌ Couples the generic checkpoint snapshot format to upgrade-specific embedding
(config bytes) unless cleanly factored.
### Option B — New "upgrade snapshot" component
A dedicated module owns: materialize-at-marker, embed config, drive transport,
activation gate, and the mismatch fence — leaving `persist.c` to expose only an
entity (de)serialize primitive and `config_repl.c` to call into it.
- ✅ Clean separation; upgrade-specific logic isolated and unit-testable.
- ✅ Keeps `config_repl.c`/`persist.c` from growing further.
- ❌ More wiring across persist ↔ new module ↔ ha transport ↔ config_repl.
- ❌ Risk of a second snapshot-format path diverging from the generic checkpoint one.
### Option C — Hybrid (recommended)
1. **Foundation (in/with `entity-bearing-snapshots`):** wire COW into kvmgr and add
the generic entity (de)serialize to `write_snapshot`/`load_snapshot` — *Option A
style*, shared by recovery + state transfer.
2. **Upgrade layer (this spec):** a thin new helper (Option B style) that, given an
entity-bearing snapshot capability, materializes at the marker `(term,lsn)`,
embeds the config bytes, streams via the existing transport, and adds the
activation gate + hash-verify-vs-marker + loud-fence-on-mismatch. Replace the
step-9 restart/replay for the upgrade case; keep restart/replay as the fallback
only until the foundation lands.
- ✅ Generic snapshot work is reused (recovery, faster rebuild) — not upgrade-only.
- ✅ Upgrade-specific gating/verification stays isolated and testable.
- ✅ Phased: foundation can land + be validated before the upgrade layer flips on.
- ❌ Requires careful coordination across the two specs (format ownership, ordering).
---
## Effort & Risk
| Dimension | Rating | One-line justification |
|-----------|--------|------------------------|
| **This spec alone** (assuming entity-bearing snapshots exist) | **L (1–2 wks)** | Transport + marker + COW exist; new work is materialize-at-marker, config embed, activation gate, hash-verify, mismatch-fence, and the failover-interleave regression. |
| **Including the gating dependency** | **XL (2+ wks)** | `entity-bearing-snapshots` is itself non-trivial (entity serialization under per-object locks, COW wiring, recovery read-back) and must land first. |
| **Risk** | **High** | Hard blocker not yet started; touches consensus-adjacent state at a `(term,lsn)` boundary interleaved with failover (R4.3); snapshot/promotion-truncate/catch-up interaction is unverified; must preserve the never-self-terminate invariant. |
---
## Recommendations for Design Phase
**Preferred approach:** **Option C (Hybrid)** — land entity-bearing snapshots as a
shared foundation (COW into kvmgr + generic entity (de)serialize in
`write_snapshot`/`load_snapshot`), then add a thin upgrade layer that materializes
at the marker, embeds the config, reuses `snapshot_transfer_s`, and adds activation
gating + hash-verify + loud-fence.
**Key decisions to settle in design:**
1. **Spec sequencing** — co-scope `entity-bearing-snapshots` into this spec's
delivery, or hard-gate this spec behind it? (Recommend gate; promote to design
only once the foundation has a tasks plan — matches `roadmap.md`.)
2. **Snapshot format** — one unified entity-bearing format used by both recovery and
upgrade, with the config-bytes embed as an optional section? Avoid a second format.
3. **Materialize-at-marker semantics** — snapshot must reflect state at *exactly* the
marker `(term,lsn)`, not "latest checkpoint." Define how the freeze window aligns
to the committed marker LSN.
4. **Activation gate** — bind "new config active" to "upgrade snapshot for this
`(term,lsn)` installed + hash-verified vs `config_committed_marker`"; mismatch ⇒
loud serve-fence + retry (never exit — `controller/CLAUDE.md`).
5. **Retire the per-node restart/replay** for the upgrade case (step 9,
`config_repl.c:1657-1676`) once the snapshot path is proven.
**Research-Needed (carry forward):**
- Upgrade × failover interleave (R4.3): how does a mid-upgrade entity snapshot
interact with promotion-truncate, `(term,lsn)` reconcile, and ACK-driven catch-up
if the leader dies mid-install? Define the recovery story.
- COW single-generation constraint vs. a checkpoint racing an upgrade snapshot
(only one freeze/thaw consumer allowed — `zarray.c` / roadmap note).
- Build `snapdump` + `walcompare` (roadmap Hardening §) first — they are the RED/GREEN
assertion tools for R5 (entity-less snapshot detection + cross-node WAL/state diff).