myxce/.kiro/specs/ha-entity-snapshots/research.md

Gap Analysis: ha-entity-snapshots

Summary

This feature is substantially cheaper than it looks. Every low-level primitive it needs already exists in the codebase for an unrelated purpose (WAL replay reconstruction) or is auto-generated by the existing Cap'n Proto schema. The work is almost entirely wiring: enumerate live entities → serialize into the snapshot file → on load, deserialize and call the existing restore functions → extend the existing trim-floor guard. No new capnp schema struct is strictly required (a length-prefixed sequence of existing per-entity blobs works), though a dedicated Snapshot struct would be cleaner. The main open design question is write-side concurrency: the "proper" mechanism (zarray copy-on-write) exists and is tested, but was never wired into the job/queue backing store (kvmgr) — a deliberate deferral ("wired in P0-2"), not a bug. This is the one real judgment call for the design phase.

Requirement-to-Asset Map

Requirement	Existing Asset	Gap
R1.1 include every live/retained job+queue record	collect_jobs/collect_queues (dataex.c:215-330, lock READ → iter_make/iter_next → unlock); job_should_retain/worker_should_retain (persist.c:3013+) already implement the retention predicate	Missing: nothing enumerates for the snapshot purpose today — write_snapshot (persist.c ~1477-1519) writes only a 57-byte header. Wiring, not new logic.
R1.1 per-entity serialization	capnp_encode_job (persist.c:45), capnp_encode_worker (persist.c:68); capnp codegen for QueueConf/xce_queue_t — encode_QueueConf, write_QueueConf, new_QueueConf, new_QueueConf_list all already generated (controller.capnp.h)	Missing: a capnp_encode_queue wrapper — trivial, mirrors capnp_encode_job line-for-line (~15 lines).
R1.2 internally consistent point-in-time view	zarray COW (src/lib/zarray.c xzarray_freeze/xzarray_thaw, tested in test_zarray_cow.c)	Constraint: COW exists but is NOT wired into kvmgr (the hash table backing c->jobmgr/c->qmgr) — kvmgr.c:297-298 explicitly stubs vfreeze/vthaw to NULL with a "wired in P0-2" comment. Consistency must come from a RTTI_LOCK_READ held across the full enumeration+encode instead (same lock collect_jobs already takes, just held longer). This is the one real design decision — see Options below.
R1.3 discard incomplete checkpoint	write_snapshot's existing tmp-file + atomic rename pattern (persist.c ~1477+, same pattern as ha_handle_snapshot_chunk's snap.part → final rename, ha.c:2589-2594)	None — extend the existing tmp+rename-on-success pattern to the entity payload; a partial write never reaches the final path.
R1.4 record committed position	Already present (checkpoint_lsn in the existing sentinel header, persist.c)	None.
R2.1/2.2 restore entities before advancing position	restore_job (job.c:96, called from WAL replay persist.c:1915) and restore_queue_state (queue.c:208, called persist.c:1945) — both already exist and are proven (used every restart via apply_wal_entry)	Missing: load_snapshot (persist.c:1811-1847) currently sets current_lsn/checkpoint_lsn directly (line ~1835) with no entity step. Insert a decode-and-restore_job/restore_queue_state loop before that assignment, reusing the exact WAL-replay call.
R2.3 failed reconstruction must not advance position	N/A — new logic, but trivial given restore functions already return rc	Missing: gate the current_lsn/checkpoint_lsn store on the restore loop's success (straightforward given R2.1/2.2's insert point).
R2.4 pre-feature checkpoint treated as insufficient	N/A	Missing: a format/version discriminator. Cheapest: if the file is exactly the legacy 57-byte sentinel size with no trailing entity payload, treat as position-only (do not claim completeness); a new format adds a marker/count after the existing header.
R3 retention never outruns restorability	repl_trim_lsn (repl.c:650-683) + the bug-1147 guard in trim_wal_segments (persist.c:1615, floor = min(checkpoint_lsn, every active standby's acked_lsn, disconnected-slot last_disconnected_lsn)) — this is an EXISTING, working guard, confirmed by its own comment: "does NOT fix the empty/far-behind join case... requires entity-bearing snapshots (separate spec)" — i.e. this spec is the one it names.	Extension, not new: add "a standby is currently mid-install of snapshot at LSN X" as another floor input into repl_trim_lsn's min_lsn computation — same shape as the existing acked_lsn/disconnected-slot terms.
R4 end-to-end catch-up correctness	ha_evaluate_repl_action (ha.c:2644, unchanged by this spec) → ha_snapshot_send_dispatch/ha_snapshot_do_send (ha.c ~2788+, unchanged) → wire opcode OPCode_snapshotTransfer → handle_snapshot_transfer (cluster.c:197-229) → ha_handle_snapshot_chunk (ha.c:2529-2609): assembles snap.part, renames to final path, calls persist_load_snapshot, then sets ha->applied_lsn.	Insertion point identified, not missing infra: the entity-restore work belongs entirely inside persist_load_snapshot/load_snapshot (R2's gap). Once that returns correctly, ha_handle_snapshot_chunk's existing "only advance applied_lsn after persist_load_snapshot succeeds" structure already gives R4's ordering for free.
R4.3 operator-observable catch-up completeness	walcompare/waldump tooling already used this session to diagnose divergence live	None new required; existing tooling already surfaces divergence. Confirm it also reports on entity-snapshot-derived state, not just WAL-visible LSNs (verify in design).
R5 safe rollout / old-format checkpoints	rotate_snapshots already discriminates by filename pattern (*.capnp)	Missing: the actual content-based old-vs-new format check (R2.4's gap, restated here for the read path).

Implementation Approach Options

Option A: Extend existing components in place (RECOMMENDED)

• Extend write_snapshot/load_snapshot (persist.c) directly: after the existing sentinel header, append a length-prefixed sequence of capnp-encoded Job and QueueConf blobs (reusing capnp_encode_job, a new trivial capnp_encode_queue, and on load decode_Job/ decode_QueueConf + restore_job/restore_queue_state).
• Write-side consistency: hold RTTI_LOCK_READ on c->jobmgr/c->qmgr across the full enumerate+encode (mirrors collect_jobs's existing lock scope, just held for the whole snapshot instead of one collection pass). Do the actual disk write from an in-memory buffer built while holding the lock, then release the lock before fdatasync+rename — so file I/O latency does not extend the lock hold.
• Extend repl_trim_lsn with one more floor term (in-flight snapshot-install LSN per standby).
• ✅ Every serialization/restore primitive already exists and is proven (WAL replay uses the same restore_job/restore_queue_state) — minimal new code, maximum reuse of tested paths.
• ✅ No new wire opcode, no new capnp schema struct strictly required.
• ❌ Holding a read lock across a full job-table walk blocks concurrent job mutations for the checkpoint's duration — needs a bound (checkpoint interval is already configurable; snap_interval_s exists) and a measurement of walk cost at realistic job-table sizes.

Option B: Wire zarray COW into kvmgr first, then build the snapshot on top

• Complete the deferred "P0-2" work (wire xzarray_freeze/xzarray_thaw into kvmgr), then write_snapshot freezes a shadow, walks it lock-free, and thaws — no read-lock hold during the walk.
• ✅ Removes the lock-hold concern entirely; likely benefits other future features too.
• ❌ Meaningfully larger and touches a shared, load-bearing data structure (kvmgr backs both job and queue tables and possibly other managers) used cluster-wide — higher blast radius and risk for a benefit (lock-free enumeration) that may not be needed if Option A's lock-hold duration proves acceptable in practice.
• ❌ It was deliberately deferred once already; reviving it is its own scoped effort, arguably a separate spec/prerequisite rather than embedded in this one.

Option C: Hybrid — ship Option A now, leave a documented seam for Option B later

• Build the snapshot feature entirely per Option A, but structure the enumeration behind a single seam function (e.g. persist_collect_snapshot_entities) so that if kvmgr COW is wired later, only that one function's internals change (lock+iterate → freeze+iterate+thaw), nothing else in the snapshot read/write/trim/catch-up path needs to change.
• ✅ Ships the actual bug fix now; keeps the door open without committing to the larger refactor's risk today.
• ❌ Slightly more upfront interface discipline than just inlining the lock+iterate call.

Effort & Risk

• Effort: M (3-7 days). All hard primitives (encode/decode/restore) pre-exist and are proven; the work is wiring four call sites (write, load, trim, and the format-discriminator) plus tests. Design and TDD discipline (per .kiro/steering/testing.md) will take real time given this is HA/persistence-critical code, but there is no unknown technology.
• Risk: Medium. Known patterns throughout (Option A), but this is exactly the kind of code the raft-ha-core-safety family (bug-795/828/851/1306) has shown is easy to get subtly wrong under concurrent chaos — the actual risk is in getting R2.3 (failed-restore-must-not- advance-position) and R3 (trim-floor extension) correct under a live chaotic soak, not in writing the serialization code itself. Recommend the same validation bar the recent session's fixes used: unit RED/GREEN + a chaotic multi-round soak showing committed_divergence=0.

Research Needed (carry to design phase)

1. Snapshot file layout: a single new capnp Snapshot { jobs: List(Job); queues: List(QueueConf); ... } struct (cleaner, one capn init/write) vs. keeping the existing raw header + hand-rolled length-prefixed blob sequence (smaller diff, matches the current sentinel's hand-rolled style). Design should pick one; either is buildable on existing codegen.
2. Lock-hold duration measurement: before committing to Option A's approach, get an order- of-magnitude estimate or measurement of collect_jobs+encode cost at a realistic job-table size, to confirm the read-lock hold is acceptably short (checkpoints already run on a snap_interval_s cadence via delay_task, off the hot request path).
3. Format discriminator mechanics: exact byte-level marker to distinguish old 57-byte sentinel-only files from new entity-bearing files without breaking rotate_snapshots's existing filename-based selection.
4. Trim-floor extension bookkeeping: where to track "standby X mid-install of snapshot LSN Y" — likely alongside the existing standby_conn_t/disconnected-slot state already used by repl_trim_lsn, but the exact struct field and its lifecycle (set on SEND_SNAPSHOT dispatch, cleared on the receiver's ack) needs design-phase detail.
5. Interaction with ha_snapshot_offload_enabled() slow-lane path (ha.c ha_snapshot_send_dispatch): confirm the entity payload size doesn't change the offload flag-ON vs flag-OFF dispatch decision or chunking assumptions already in place for ha_handle_snapshot_chunk's multi-chunk reassembly.

Recommendation

Option A (extend write_snapshot/load_snapshot/trim_wal_segments in place, structured per Option C's seam discipline) for the design phase. Do not pursue Option B (kvmgr COW wiring) as part of this spec — it is a legitimate future improvement but was already deliberately deferred once, and this spec's actual requirements (R1-R5) do not need it: a bounded read-lock hold during a periodic, off-hot-path checkpoint is a reasonable cost, and it keeps this spec's blast radius to persist.c/ha.c/repl.c, matching the existing bug-1147 guard's pattern rather than touching the shared kvmgr.

---

Design Phase — Synthesis & Refinements (2026-07-01)

Key refinement over the gap analysis

The gap analysis (above) assumed trim_wal_segments/repl_trim_lsn needed a new floor input keyed on "a standby is currently mid-install of a snapshot." Closer reading of persist_checkpoint (persist.c:1763) shows this is already handled: write_snapshot runs before trim_wal_segments, and a non-zero write_snapshot return skips both trim_wal_segments and rotate_snapshots via the existing goto done. Once write_snapshot only returns 0 when the entity payload is fully and durably written (this spec's Requirement 1.3), that existing sequencing alone satisfies Requirement 3 — no new code needed in trim_wal_segments/ repl_trim_lsn.

The one retention hazard that genuinely has no existing guard is rotate_snapshots deleting a checkpoint file while a standby is still mid-transfer reading it over the wire (a checkpoint cadence faster than a slow transfer). This is the actual new mechanism this design adds: a snapshot_inflight_lsn[] array (mirroring the existing snapshot_inflight[] flag array in ha_state_t) that rotate_snapshots consults before deleting a candidate file.

Synthesis: Generalization

Requirements 1 and 2 both need "enumerate a live entity of kind K, encode it, and (for restore) decode+apply it." Rather than writing separate bespoke job-snapshot and queue-snapshot code paths, the design generalizes the interface (a small per-kind {collect, encode, decode+restore} triple) while keeping the implementation scoped to exactly the two kinds Requirement 1 names (job, queue) — not built out for hypothetical future entity kinds (e.g. workers), consistent with the requirement's explicit boundary.

Synthesis: Build vs. Adopt

Adopt, not build: the project's own Cap'n Proto schema and generated codecs for Job/ QueueConf already exist and are already proven via WAL replay's restore_job/ restore_queue_state. No external library, no new schema struct, no new wire protocol — this spec is pure reuse of already-solved, already-tested primitives.

Synthesis: Simplification

Rejected a new top-level capnp Snapshot message struct (would require a schema/build change for a one-off container) in favor of a hand-rolled length-prefixed frame sequence — matching the project's own existing WAL entry framing style (wal_entry_hdr_t + payload) and avoiding a new abstraction layer for something with exactly one caller. Rejected reviving the deferred zarray COW wiring (Option B) as unnecessary for this spec's scope — a bounded RTTI_LOCK_READ hold during an off-hot-path, periodic checkpoint is a reasonable cost that keeps this spec's blast radius to persist.c/ha.c, matching the existing bug-1147 guard's precedent rather than touching the shared kvmgr.

Design Decisions Log

• On-disk format: legacy header (unchanged) + format_version + job_count + length-prefixed Job frames + queue_count + length-prefixed QueueConf frames. Legacy-file detection is "no bytes remain after the legacy header" — no explicit flag bit needed.
• Consistency boundary: per-entity, not cross-entity — jobs and queues are independently locked/encoded in two sequential RTTI_LOCK_READ sections, never held simultaneously.
• API change: persist_load_snapshot gains an out_entities_restored out-parameter so the caller can distinguish "fully restored" from "legacy position-only" without treating the latter as an error.
• New guard location: rotate_snapshots (not trim_wal_segments), backed by a new snapshot_inflight_lsn[MAX_HA_CANDIDATES] array in ha_state_t, lifecycle-mirrored to the existing snapshot_inflight[] flag.

---

Design Validation Pass — Corrections (2026-07-01)

/kiro-validate-design found two real defects in the first design draft, both confirmed against code and fixed in design.md:

1. Queue entity shape was wrong. The draft planned to serialize a full QueueConf capnp entity per queue, but restore_queue_state(cluster_t*, const char* name, uint32_t status) (queue.c:208) — the only existing queue-restore function — takes just a name and status word; its own comment states queues are config-built at startup and only admin status changes are WAL-tracked. Confirmed independently: queue definitions are already replicated via push_config_to_standbys (config_repl.c), a separate, already-committed, majority-acked channel — entirely out of this spec's boundary (user confirmation: "the basic configuration is not lived in cluster state snapshot as we have independent configuration"). Fix: the queue-side snapshot payload is now a plain (name, status) record, no capnp, restored via the unchanged restore_queue_state. This simplifies the design (no queue capnp encoder needed at all).
2. The table-lock-only consistency argument was false. The original Architecture section claimed RTTI_LOCK_READ on c->jobmgr/c->qmgr (the table lock) excludes concurrent entity mutation. Verified false: restore_queue_state itself takes a separate per-entity lock(q, RTTI_LOCK_WRITE), and job state-transition sites (job.c:1006/1093/1137) take lock(j, RTTI_LOCK_WRITE) — a distinct lock object from the table lock. Fix: the entity-collection helper now nests each entity's own RTTI_LOCK_READ inside the table walk, around that entity's individual encode call — mirroring restore_queue_state's own nested-lock pattern. Added a new test item (concurrent-mutation-during-snapshot consistency) to prove this directly.

Both fixes are local repairs to the write-side collection logic and the queue payload format — the overall Option-A approach, boundary, and file structure plan are unchanged.