myxce/.kiro/specs/ha-entity-snapshots/research.md

# Gap Analysis: ha-entity-snapshots

## Summary

This feature is **substantially cheaper than it looks**. Every low-level primitive it needs
already exists in the codebase for an unrelated purpose (WAL replay reconstruction) or is
auto-generated by the existing Cap'n Proto schema. The work is almost entirely **wiring**:
enumerate live entities → serialize into the snapshot file → on load, deserialize and call the
existing restore functions → extend the existing trim-floor guard. No new capnp schema struct
is strictly required (a length-prefixed sequence of existing per-entity blobs works), though a
dedicated `Snapshot` struct would be cleaner. The main open design question is write-side
concurrency: the "proper" mechanism (zarray copy-on-write) exists and is tested, but was never
wired into the job/queue backing store (`kvmgr`) — a deliberate deferral ("wired in P0-2"), not
a bug. This is the one real judgment call for the design phase.

## Requirement-to-Asset Map

| Requirement | Existing Asset | Gap |
|---|---|---|
| R1.1 include every live/retained job+queue record | `collect_jobs`/`collect_queues` (dataex.c:215-330, lock READ → `iter_make`/`iter_next` → unlock); `job_should_retain`/`worker_should_retain` (persist.c:3013+) already implement the retention predicate | **Missing**: nothing enumerates for the *snapshot* purpose today — `write_snapshot` (persist.c ~1477-1519) writes only a 57-byte header. Wiring, not new logic. |
| R1.1 per-entity serialization | `capnp_encode_job` (persist.c:45), `capnp_encode_worker` (persist.c:68); capnp codegen for `QueueConf`/`xce_queue_t` — `encode_QueueConf`, `write_QueueConf`, `new_QueueConf`, `new_QueueConf_list` all already generated (controller.capnp.h) | **Missing**: a `capnp_encode_queue` wrapper — trivial, mirrors `capnp_encode_job` line-for-line (~15 lines). |
| R1.2 internally consistent point-in-time view | zarray COW (`src/lib/zarray.c` `xzarray_freeze`/`xzarray_thaw`, tested in `test_zarray_cow.c`) | **Constraint**: COW exists but is **NOT wired into `kvmgr`** (the hash table backing `c->jobmgr`/`c->qmgr`) — `kvmgr.c:297-298` explicitly stubs `vfreeze`/`vthaw` to NULL with a "wired in P0-2" comment. Consistency must come from a `RTTI_LOCK_READ` held across the full enumeration+encode instead (same lock `collect_jobs` already takes, just held longer). This is the one real design decision — see Options below. |
| R1.3 discard incomplete checkpoint | `write_snapshot`'s existing tmp-file + atomic `rename` pattern (persist.c ~1477+, same pattern as `ha_handle_snapshot_chunk`'s `snap.part` → final rename, ha.c:2589-2594) | **None** — extend the existing tmp+rename-on-success pattern to the entity payload; a partial write never reaches the final path. |
| R1.4 record committed position | Already present (`checkpoint_lsn` in the existing sentinel header, persist.c) | **None.** |
| R2.1/2.2 restore entities before advancing position | `restore_job` (job.c:96, called from WAL replay persist.c:1915) and `restore_queue_state` (queue.c:208, called persist.c:1945) — **both already exist and are proven** (used every restart via `apply_wal_entry`) | **Missing**: `load_snapshot` (persist.c:1811-1847) currently sets `current_lsn`/`checkpoint_lsn` directly (line ~1835) with no entity step. Insert a decode-and-`restore_job`/`restore_queue_state` loop **before** that assignment, reusing the exact WAL-replay call. |
| R2.3 failed reconstruction must not advance position | N/A — new logic, but trivial given restore functions already return rc | **Missing**: gate the `current_lsn`/`checkpoint_lsn` store on the restore loop's success (straightforward given R2.1/2.2's insert point). |
| R2.4 pre-feature checkpoint treated as insufficient | N/A | **Missing**: a format/version discriminator. Cheapest: if the file is exactly the legacy 57-byte sentinel size with no trailing entity payload, treat as position-only (do not claim completeness); a new format adds a marker/count after the existing header. |
| R3 retention never outruns restorability | `repl_trim_lsn` (repl.c:650-683) + the bug-1147 guard in `trim_wal_segments` (persist.c:1615, floor = min(checkpoint_lsn, every active standby's `acked_lsn`, disconnected-slot `last_disconnected_lsn`)) — **this is an EXISTING, working guard**, confirmed by its own comment: "does NOT fix the empty/far-behind join case... requires entity-bearing snapshots (separate spec)" — i.e. this spec is the one it names. | **Extension, not new**: add "a standby is currently mid-install of snapshot at LSN X" as another floor input into `repl_trim_lsn`'s `min_lsn` computation — same shape as the existing acked_lsn/disconnected-slot terms. |
| R4 end-to-end catch-up correctness | `ha_evaluate_repl_action` (ha.c:2644, unchanged by this spec) → `ha_snapshot_send_dispatch`/`ha_snapshot_do_send` (ha.c ~2788+, unchanged) → wire opcode `OPCode_snapshotTransfer` → `handle_snapshot_transfer` (cluster.c:197-229) → **`ha_handle_snapshot_chunk`** (ha.c:2529-2609): assembles `snap.part`, renames to final path, calls `persist_load_snapshot`, then sets `ha->applied_lsn`. | **Insertion point identified, not missing infra**: the entity-restore work belongs entirely inside `persist_load_snapshot`/`load_snapshot` (R2's gap). Once that returns correctly, `ha_handle_snapshot_chunk`'s existing "only advance `applied_lsn` after `persist_load_snapshot` succeeds" structure already gives R4's ordering for free. |
| R4.3 operator-observable catch-up completeness | `walcompare`/`waldump` tooling already used this session to diagnose divergence live | **None new required**; existing tooling already surfaces divergence. Confirm it also reports on entity-snapshot-derived state, not just WAL-visible LSNs (verify in design). |
| R5 safe rollout / old-format checkpoints | `rotate_snapshots` already discriminates by filename pattern (`*.capnp`) | **Missing**: the actual content-based old-vs-new format check (R2.4's gap, restated here for the read path). |

## Implementation Approach Options

### Option A: Extend existing components in place (RECOMMENDED)
- Extend `write_snapshot`/`load_snapshot` (persist.c) directly: after the existing sentinel
  header, append a length-prefixed sequence of capnp-encoded `Job` and `QueueConf` blobs
  (reusing `capnp_encode_job`, a new trivial `capnp_encode_queue`, and on load `decode_Job`/
  `decode_QueueConf` + `restore_job`/`restore_queue_state`).
- Write-side consistency: hold `RTTI_LOCK_READ` on `c->jobmgr`/`c->qmgr` across the full
  enumerate+encode (mirrors `collect_jobs`'s existing lock scope, just held for the whole
  snapshot instead of one collection pass). Do the actual disk write from an in-memory buffer
  built while holding the lock, then release the lock before `fdatasync`+rename — so file I/O
  latency does not extend the lock hold.
- Extend `repl_trim_lsn` with one more floor term (in-flight snapshot-install LSN per standby).
- ✅ Every serialization/restore primitive already exists and is proven (WAL replay uses the
  same `restore_job`/`restore_queue_state`) — minimal new code, maximum reuse of tested paths.
- ✅ No new wire opcode, no new capnp schema struct strictly required.
- ❌ Holding a read lock across a full job-table walk blocks concurrent job mutations for the
  checkpoint's duration — needs a bound (checkpoint interval is already configurable;
  `snap_interval_s` exists) and a measurement of walk cost at realistic job-table sizes.

### Option B: Wire zarray COW into kvmgr first, then build the snapshot on top
- Complete the deferred "P0-2" work (wire `xzarray_freeze`/`xzarray_thaw` into `kvmgr`), then
  `write_snapshot` freezes a shadow, walks it lock-free, and thaws — no read-lock hold during
  the walk.
- ✅ Removes the lock-hold concern entirely; likely benefits other future features too.
- ❌ Meaningfully larger and touches a shared, load-bearing data structure (`kvmgr` backs both
  job and queue tables and possibly other managers) used cluster-wide — higher blast radius and
  risk for a benefit (lock-free enumeration) that may not be needed if Option A's lock-hold
  duration proves acceptable in practice.
- ❌ It was deliberately deferred once already; reviving it is its own scoped effort, arguably a
  separate spec/prerequisite rather than embedded in this one.

### Option C: Hybrid — ship Option A now, leave a documented seam for Option B later
- Build the snapshot feature entirely per Option A, but structure the enumeration behind a
  single seam function (e.g. `persist_collect_snapshot_entities`) so that if kvmgr COW is wired
  later, only that one function's internals change (lock+iterate → freeze+iterate+thaw),
  nothing else in the snapshot read/write/trim/catch-up path needs to change.
- ✅ Ships the actual bug fix now; keeps the door open without committing to the larger
  refactor's risk today.
- ❌ Slightly more upfront interface discipline than just inlining the lock+iterate call.

## Effort & Risk

- **Effort: M (3-7 days).** All hard primitives (encode/decode/restore) pre-exist and are
  proven; the work is wiring four call sites (write, load, trim, and the format-discriminator)
  plus tests. Design and TDD discipline (per `.kiro/steering/testing.md`) will take real time
  given this is HA/persistence-critical code, but there is no unknown technology.
- **Risk: Medium.** Known patterns throughout (Option A), but this is exactly the kind of
  code the `raft-ha-core-safety` family (bug-795/828/851/1306) has shown is easy to get subtly
  wrong under concurrent chaos — the actual risk is in getting R2.3 (failed-restore-must-not-
  advance-position) and R3 (trim-floor extension) correct under a live chaotic soak, not in
  writing the serialization code itself. Recommend the same validation bar the recent session's
  fixes used: unit RED/GREEN + a chaotic multi-round soak showing `committed_divergence=0`.

## Research Needed (carry to design phase)

1. **Snapshot file layout**: a single new capnp `Snapshot { jobs: List(Job); queues:
   List(QueueConf); ... }` struct (cleaner, one `capn` init/write) vs. keeping the existing raw
   header + hand-rolled length-prefixed blob sequence (smaller diff, matches the current
   sentinel's hand-rolled style). Design should pick one; either is buildable on existing codegen.
2. **Lock-hold duration measurement**: before committing to Option A's approach, get an order-
   of-magnitude estimate or measurement of `collect_jobs`+encode cost at a realistic job-table
   size, to confirm the read-lock hold is acceptably short (checkpoints already run on a
   `snap_interval_s` cadence via `delay_task`, off the hot request path).
3. **Format discriminator mechanics**: exact byte-level marker to distinguish old 57-byte
   sentinel-only files from new entity-bearing files without breaking `rotate_snapshots`'s
   existing filename-based selection.
4. **Trim-floor extension bookkeeping**: where to track "standby X mid-install of snapshot LSN
   Y" — likely alongside the existing `standby_conn_t`/disconnected-slot state already used by
   `repl_trim_lsn`, but the exact struct field and its lifecycle (set on `SEND_SNAPSHOT`
   dispatch, cleared on the receiver's ack) needs design-phase detail.
5. **Interaction with `ha_snapshot_offload_enabled()` slow-lane path** (ha.c
   `ha_snapshot_send_dispatch`): confirm the entity payload size doesn't change the offload
   flag-ON vs flag-OFF dispatch decision or chunking assumptions already in place for
   `ha_handle_snapshot_chunk`'s multi-chunk reassembly.

## Recommendation

**Option A** (extend `write_snapshot`/`load_snapshot`/`trim_wal_segments` in place, structured
per **Option C**'s seam discipline) for the design phase. Do not pursue Option B (kvmgr COW
wiring) as part of this spec — it is a legitimate future improvement but was already
deliberately deferred once, and this spec's actual requirements (R1-R5) do not need it: a
bounded read-lock hold during a periodic, off-hot-path checkpoint is a reasonable cost, and it
keeps this spec's blast radius to `persist.c`/`ha.c`/`repl.c`, matching the existing bug-1147
guard's pattern rather than touching the shared kvmgr.

---

## Design Phase — Synthesis & Refinements (2026-07-01)

### Key refinement over the gap analysis
The gap analysis (above) assumed `trim_wal_segments`/`repl_trim_lsn` needed a new floor input
keyed on "a standby is currently mid-install of a snapshot." Closer reading of
`persist_checkpoint` (persist.c:1763) shows this is **already handled**: `write_snapshot` runs
before `trim_wal_segments`, and a non-zero `write_snapshot` return skips both `trim_wal_segments`
and `rotate_snapshots` via the existing `goto done`. Once `write_snapshot` only returns 0 when
the entity payload is fully and durably written (this spec's Requirement 1.3), that existing
sequencing alone satisfies Requirement 3 — **no new code needed in `trim_wal_segments`/
`repl_trim_lsn`**.

The one retention hazard that genuinely has no existing guard is **`rotate_snapshots`** deleting
a checkpoint *file* while a standby is still mid-transfer reading it over the wire (a checkpoint
cadence faster than a slow transfer). This is the actual new mechanism this design adds: a
`snapshot_inflight_lsn[]` array (mirroring the existing `snapshot_inflight[]` flag array in
`ha_state_t`) that `rotate_snapshots` consults before deleting a candidate file.

### Synthesis: Generalization
Requirements 1 and 2 both need "enumerate a live entity of kind K, encode it, and (for restore)
decode+apply it." Rather than writing separate bespoke job-snapshot and queue-snapshot code
paths, the design generalizes the *interface* (a small per-kind {collect, encode, decode+restore}
triple) while keeping the *implementation* scoped to exactly the two kinds Requirement 1 names
(job, queue) — not built out for hypothetical future entity kinds (e.g. workers), consistent with
the requirement's explicit boundary.

### Synthesis: Build vs. Adopt
Adopt, not build: the project's own Cap'n Proto schema and generated codecs for `Job`/
`QueueConf` already exist and are already proven via WAL replay's `restore_job`/
`restore_queue_state`. No external library, no new schema struct, no new wire protocol — this
spec is pure reuse of already-solved, already-tested primitives.

### Synthesis: Simplification
Rejected a new top-level capnp `Snapshot` message struct (would require a schema/build change for
a one-off container) in favor of a hand-rolled length-prefixed frame sequence — matching the
project's own existing WAL entry framing style (`wal_entry_hdr_t` + payload) and avoiding a new
abstraction layer for something with exactly one caller. Rejected reviving the deferred zarray
COW wiring (Option B) as unnecessary for this spec's scope — a bounded `RTTI_LOCK_READ` hold
during an off-hot-path, periodic checkpoint is a reasonable cost that keeps this spec's blast
radius to `persist.c`/`ha.c`, matching the existing bug-1147 guard's precedent rather than
touching the shared `kvmgr`.

### Design Decisions Log
- **On-disk format**: legacy header (unchanged) + `format_version` + `job_count` + length-prefixed
  `Job` frames + `queue_count` + length-prefixed `QueueConf` frames. Legacy-file detection is
  "no bytes remain after the legacy header" — no explicit flag bit needed.
- **Consistency boundary**: per-entity, not cross-entity — jobs and queues are independently
  locked/encoded in two sequential `RTTI_LOCK_READ` sections, never held simultaneously.
- **API change**: `persist_load_snapshot` gains an `out_entities_restored` out-parameter so the
  caller can distinguish "fully restored" from "legacy position-only" without treating the latter
  as an error.
- **New guard location**: `rotate_snapshots` (not `trim_wal_segments`), backed by a new
  `snapshot_inflight_lsn[MAX_HA_CANDIDATES]` array in `ha_state_t`, lifecycle-mirrored to the
  existing `snapshot_inflight[]` flag.

---

## Design Validation Pass — Corrections (2026-07-01)

`/kiro-validate-design` found two real defects in the first design draft, both confirmed against
code and fixed in `design.md`:

1. **Queue entity shape was wrong.** The draft planned to serialize a full `QueueConf` capnp
   entity per queue, but `restore_queue_state(cluster_t*, const char* name, uint32_t status)`
   (queue.c:208) — the only existing queue-restore function — takes just a name and status word;
   its own comment states queues are config-built at startup and only admin status changes are
   WAL-tracked. Confirmed independently: queue *definitions* are already replicated via
   `push_config_to_standbys` (config_repl.c), a separate, already-committed, majority-acked
   channel — entirely out of this spec's boundary (user confirmation: "the basic configuration
   is not lived in cluster state snapshot as we have independent configuration"). **Fix**: the
   queue-side snapshot payload is now a plain `(name, status)` record, no capnp, restored via the
   unchanged `restore_queue_state`. This *simplifies* the design (no queue capnp encoder needed
   at all).
2. **The table-lock-only consistency argument was false.** The original Architecture section
   claimed `RTTI_LOCK_READ` on `c->jobmgr`/`c->qmgr` (the table lock) excludes concurrent entity
   mutation. Verified false: `restore_queue_state` itself takes a *separate* per-entity
   `lock(q, RTTI_LOCK_WRITE)`, and job state-transition sites (job.c:1006/1093/1137) take
   `lock(j, RTTI_LOCK_WRITE)` — a distinct lock object from the table lock. **Fix**: the
   entity-collection helper now nests each entity's own `RTTI_LOCK_READ` inside the table walk,
   around that entity's individual encode call — mirroring `restore_queue_state`'s own
   nested-lock pattern. Added a new test item (concurrent-mutation-during-snapshot consistency)
   to prove this directly.

Both fixes are local repairs to the write-side collection logic and the queue payload format —
the overall Option-A approach, boundary, and file structure plan are unchanged.