myxce/.kiro/specs/ha-entity-snapshots/design.md

Design Document: ha-entity-snapshots

Overview

Purpose: This feature makes the XCE controller's periodic persistence checkpoint an actually-restorable copy of committed job and queue state, closing the permanent data gap a rejoining or far-behind cluster member hits today (task 5.10a / raft-ha-core-safety; bug-795/ 828/851 family). Today write_snapshot records only a position marker; load_snapshot restores nothing. This design adds the missing entity payload and restore step, and closes the one real retention hazard it introduces (a snapshot file being rotated out from under an in-flight transfer).

Users: The XCE HA controller cluster itself is the direct consumer — every member that rejoins empty or falls behind the retained WAL history. The observable beneficiary is the cluster operator, who currently sees a silently incomplete member after such a rejoin.

Impact: Modifies the existing checkpoint write/read/rotate logic in persist.c and adds one small tracking array in ha.c. No wire protocol change, no new external dependency, no new capnp schema struct — the existing Job schema and its generated codec already carry everything needed for jobs. Queue definitions (name, attrs, priority, thresholds, ACLs) are config- provisioned and already replicated independently via push_config_to_standbys (config_repl.c) — out of this spec's boundary. Only a queue's runtime admin status (open/closed/active/inactive), the one piece WAL_QUEUE_CTRL tracks and config replication does not carry, is this spec's concern — restored via the existing restore_queue_state(name, status), no capnp involved on the queue side at all. (Note: the on-disk wire format was later migrated to a capnp QueueAdminState frame — superseded by config-upgrade-snapshot, see that spec's design.md; restore_queue_state's own (name, status) call contract is unchanged.)

Goals

• A checkpoint written at LSN X contains every live/retained job and queue record as of X.
• Loading that checkpoint reconstructs those records before the member's position advances.
• History is never discarded before it is captured in a checkpoint whose entities are durable.
• A member that rejoins via checkpoint ends up bit-for-bit consistent with the rest of the cluster — no permanent gap.
• A checkpoint produced by a prior version of the binary never crashes or corrupts a member that reads it.

Non-Goals

• Lock-free, copy-on-write entity enumeration (zarray COW exists but is not wired into kvmgr; reviving that wiring is a separate, larger effort — see research.md Option B).
• Any change to the wire transport/chunking protocol that ships a snapshot to a peer.
• Any change to the catch-up decision logic (ha_evaluate_repl_action) that chooses between a snapshot transfer and incremental WAL streaming.
• Worker/compute-node state, cluster configuration document snapshotting, or election/term-safety behavior.
• Queue definition fields (name, attrs, priority, thresholds, windows, ACLs) — these are config-provisioned and already replicated independently (push_config_to_standbys); this spec snapshots only the queue's runtime admin status.

Boundary Commitments

This Spec Owns

• The on-disk content format of a checkpoint (entity payload framing) and the write/read logic that produces and consumes it (write_snapshot, load_snapshot internals).
• Point-in-time collection and encoding of live job and queue entities at checkpoint time.
• The entity-restore step on load and gating the member's committed-position advance on that step's success.
• Distinguishing a legacy (pre-feature) position-only checkpoint from an entity-bearing one.
• Protecting an in-flight-transferring checkpoint file from premature rotation deletion.

Out of Boundary

• The wire opcode and chunk-reassembly protocol that delivers a checkpoint to a peer (OPCode_snapshotTransfer, ha_handle_snapshot_chunk's framing) — reused unchanged.
• The decision of when a member needs a checkpoint vs. incremental streaming (ha_evaluate_repl_action) — unchanged.
• The WAL-segment trim floor's standby-acked-LSN computation (repl_trim_lsn) — unchanged; see Architecture below for why no new floor input is needed there.
• Worker/join state, config-document snapshotting (config-upgrade-snapshot), election and term-safety behavior (raft-ha-core-safety, ha-term-safety).

Allowed Dependencies

• Existing capnp codegen for Job (encode_Job/write_Job/decode_Job, already generated from src/schema/controller.capnp). Queue entries need no capnp — restore_queue_state's contract is a plain (name, status) pair (see Architecture below). (Note: config-upgrade-snapshot later added a capnp QueueAdminState frame as the on-disk encoding of that same pair — see Data Models below.)
• restore_job (job.c:274) and restore_queue_state (queue.c:208) — existing, unmodified entity-restore functions already used by WAL replay.
• The collect_jobs/collect_queues enumeration pattern (dataex.c) and the job_should_retain/worker_should_retain retention predicates (persist.c).
• The existing tmp-file-then-atomic-rename pattern already used by write_snapshot and by ha_handle_snapshot_chunk's snap.part assembly.
• The existing per-peer ha->snapshot_inflight[] array shape (ha.h) as the pattern for the new in-flight-LSN tracking.

Revalidation Triggers

• Any change to the Job capnp schema shape — re-verify encode/decode symmetry and compatibility with restore_job.
• Any change to restore_queue_state's (name, status) contract, or any future need to snapshot more than status for queues — re-verify the queue-side format and the config-vs-WAL boundary this design draws.
• Any change to collect_jobs/collect_queues locking semantics, or to the per-entity RTTI_LOCK_READ/WRITE discipline (e.g. if kvmgr COW is wired later) — re-verify the point-in-time consistency argument in Architecture below.
• Any change to persist_checkpoint's write_snapshot → trim_wal_segments sequencing (currently sequential, trim skipped on write failure) — re-verify Requirement 3.
• Any change to ha_handle_snapshot_chunk's ordering (currently: applied_lsn only advances after persist_load_snapshot returns success) — re-verify Requirement 4.

Architecture

Existing Architecture Analysis

• persist_checkpoint (persist.c:2820) already sequences correctly for Requirement 3: it calls write_snapshot, and only calls trim_wal_segments if write_snapshot returned 0 (if (rc != 0) goto done;, skipping both trim and rotate_snapshots). This means once write_snapshot only returns 0 when the entity payload is fully and durably written, the existing trim gating already satisfies "retention never outruns restorability" — no new floor variable is needed in trim_wal_segments/repl_trim_lsn. This is a refinement over the initial gap analysis (research.md), which assumed a new trim-floor input was needed.
• The one retention hazard write_snapshot's success does not already guard is rotate_snapshots (persist.c:2677): it keeps only snap_keep_n newest snapshot files by filename sort and deletes the rest, with no awareness of an in-flight transfer. If a checkpoint cadence faster than a slow transfer rotates out the file a standby is still reading from a chunked wire transfer, that transfer breaks. This is the one genuinely new guard this design adds.
• ha_handle_snapshot_chunk (ha.c:3635-3794) already has the right shape for Requirement 4: it only advances ha->applied_lsn after persist_load_snapshot returns success. Once persist_load_snapshot/load_snapshot correctly gate their own success on entity restoration (Requirement 2), this call site needs no change.
• collect_jobs/collect_queues (dataex.c) already hold RTTI_LOCK_READ on c->jobmgr/ c->qmgr for the duration of their walk. This table lock is a distinct lock from the per-entity lock — verified via restore_queue_state (queue.c:208), which takes lock(q, RTTI_LOCK_WRITE) on the individual queue_t* after already releasing the table lock from its own key_find, and via job.c's state-transition sites (e.g. lines 1006, 1093, 1137), which take lock(j, RTTI_LOCK_WRITE) on the individual job_t* (line numbers shift as job.c grows; verify by grepping lock(j, RTTI_LOCK_WRITE) if re-checking this). The table lock only excludes concurrent insert/delete of table entries; it does not exclude a concurrent per-entity writer mutating an entity's fields mid-walk. Requirement 1.2's point-in-time consistency therefore requires the entity-collection helper to additionally take each entity's own RTTI_LOCK_READ (mirroring restore_queue_state's own per-entity write-lock pattern) around that entity's individual encode call, nested inside the table-level walk — this excludes a concurrent per-entity writer for exactly the duration of that one entity's encode, without needing the (currently unwired) zarray COW mechanism.

Architecture Pattern & Boundary Map

flowchart TB
    subgraph Primary["Primary — checkpoint write path"]
        PC[persist_checkpoint] --> WS[write_snapshot]
        WS --> CJ["collect+encode jobs\n(table READ lock + per-job READ lock)"]
        WS --> CQ["collect name+status pairs\n(table READ lock + per-queue READ lock,\nnow capnp QueueAdminState --\nconfig-upgrade-snapshot)"]
        CJ --> TMP[tmp file + fdatasync + rename]
        CQ --> TMP
        TMP -->|rc==0| TRIM[trim_wal_segments]
        TMP -->|rc==0| ROT["rotate_snapshots\n(skips in-flight LSNs)"]
        TMP -->|rc!=0| SKIP["skip trim + rotate\n(existing goto done)"]
    end

    subgraph Send["Snapshot transfer (unchanged)"]
        DISPATCH[ha_snapshot_send_dispatch] -->|sets snapshot_inflight_lsn for peer| WIRE[OPCode_snapshotTransfer chunks]
        WIRE --> DONE[clears snapshot_inflight_lsn on completion]
    end

    ROT -.reads.-> DISPATCH

    subgraph Standby["Standby — catch-up receive path"]
        RECV[ha_handle_snapshot_chunk] --> RENAME[assemble + rename final file]
        RENAME --> LS[persist_load_snapshot]
        LS --> RJ["restore_job per entity"]
        LS --> RQ["restore_queue_state per entity"]
        RJ -->|all succeed| ADV["advance current_lsn / checkpoint_lsn"]
        RQ -->|all succeed| ADV
        RJ -->|any fails| FAIL["return error, position unchanged"]
        ADV --> APPLIED["ha->applied_lsn advances\n(existing, unchanged)"]
    end

Architecture Integration:

• Selected pattern: extend the existing checkpoint write/read pipeline in place (Option A from research.md); no new subsystem, no new wire message.
• Domain boundaries: entity content and restoration stay inside persist.c (write/load/rotate); the one cross-file touch is the new snapshot_inflight_lsn[] array in ha.h/ha.c, read by rotate_snapshots and written by the existing send-dispatch path.
• Existing patterns preserved: tmp-file-then-rename atomicity, RTTI_LOCK_READ/WRITE table AND per-entity locking (mirroring restore_queue_state's own nested-lock pattern), the collect_jobs/collect_queues iteration shape, the capnp_encode_job wrapper shape.
• New components rationale: a small entity-frame reader/writer pair (mirrors the existing WAL entry length-prefixed framing style, so no new capnp schema struct or codegen is required — jobs use the existing Job capnp codec, queues use a plain fixed (name, status) record since that is the entirety of restore_queue_state's contract); the in-flight-LSN guard in rotate_snapshots. (Note: "no new capnp schema struct" for queues held at this spec's close but was superseded shortly after by config-upgrade-snapshot, which added a QueueAdminState capnp struct and migrated the queue frame to it — see that spec's design.md.)
• Steering compliance: no fork() used for the snapshot (per product.md's "No fork() for snapshots" constraint) — consistency comes from the table read-lock, not process duplication.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Serialization	Cap'n Proto (existing project-wide IDL)	Per-job Job encode/decode; queues use a plain fixed record (no capnp)	Codegen already exists; no schema change
Persistence	Flat file, tmp+rename (existing pattern)	Checkpoint file on disk	Same pattern as today's sentinel-only file
Concurrency	RTTI_LOCK_READ/WRITE, table AND per-entity (existing)	Point-in-time entity consistency during collection	zarray COW deferred (Non-Goal)
Transport	OPCode_snapshotTransfer (existing, unmodified)	Ships the (now larger) file to a standby	Out of boundary

File Structure Plan

Modified Files

• src/controller/persist.c — write_snapshot: append entity payload after the existing header — job frames (capnp Job) and queue frames (plain (name, status) records, no capnp). (Note: queue frames were later migrated to capnp QueueAdminState, and host frames plus a trailing config-bytes section were appended after them — superseded by config-upgrade-snapshot, see Data Models below and that spec's design.md.) load_snapshot: restore entities before advancing current_lsn/checkpoint_lsn; detect legacy (position-only) files by remaining file size. rotate_snapshots: skip deleting any file whose LSN is currently in-flight. New static helpers: a snapshot entity-frame writer/reader pair, and an entity-collection helper built on collect_jobs/collect_queues that additionally takes each entity's own per-entity RTTI_LOCK_READ around its individual encode/collect call (see Architecture).
• src/include/persist.h — extend persist_load_snapshot's signature with an out-parameter reporting whether entities were restored (see Components below); no other public surface changes.
• src/controller/ha.h — add _Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES] next to the existing snapshot_inflight[MAX_HA_CANDIDATES] array.
• src/controller/ha.c — ha_snapshot_send_dispatch/ha_snapshot_do_send: record the LSN being sent into snapshot_inflight_lsn[peer_idx] alongside the existing flag, clear it on completion (same lifecycle as the existing flag). ha_handle_snapshot_chunk: read the new persist_load_snapshot out-parameter; log (not fail) when entities were not restored, since the caller-side catch-up-completeness signal (Requirement 4.3) is carried by existing status/comparison tooling, not a new error path here.

New Test Files

• src/controller/testsuite/test_persist_snapshot_entities.c — unit coverage for write_snapshot/load_snapshot entity round-trip, partial-failure discard, legacy-file handling, and the rotate_snapshots in-flight guard (see Testing Strategy).

No new production files. This stays entirely inside the existing checkpoint machinery's file set plus one small, symmetric addition in ha.h/ha.c for the rotation guard.

System Flows

See the Architecture Pattern & Boundary Map diagram above for the write/send/receive flow. One additional sequence clarifies the failure-ordering contract that Requirement 2.3 depends on:

sequenceDiagram
    participant Standby
    participant persist_load_snapshot
    participant restore_job
    participant restore_queue_state

    Standby->>persist_load_snapshot: load(path, lsn)
    persist_load_snapshot->>persist_load_snapshot: read header, detect format
    alt legacy (position-only) file
        persist_load_snapshot-->>Standby: rc=0, entities_restored=false
    else entity-bearing file
        loop each job frame
            persist_load_snapshot->>restore_job: decode + restore
            restore_job-->>persist_load_snapshot: rc
            alt rc != 0
                persist_load_snapshot-->>Standby: rc=error, position UNCHANGED
            end
        end
        loop each queue frame
            persist_load_snapshot->>restore_queue_state: restore(name, status)
            restore_queue_state-->>persist_load_snapshot: rc
            alt rc != 0
                persist_load_snapshot-->>Standby: rc=error, position UNCHANGED
            end
        end
        persist_load_snapshot->>persist_load_snapshot: advance current_lsn/checkpoint_lsn
        persist_load_snapshot-->>Standby: rc=0, entities_restored=true
    end

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Checkpoint includes every live/retained job+queue record	write_snapshot, entity-collection helper	collect_jobs/collect_queues + job_should_retain; queue side is (name, status) only (definition is config-owned)	Write path (diagram 1)
1.2	Internally consistent point-in-time view	Entity-collection helper	Table RTTI_LOCK_READ + per-entity RTTI_LOCK_READ nested around each encode	Architecture: Existing Architecture Analysis
1.3	Discard incomplete checkpoint	write_snapshot	Existing tmp+rename pattern	Write path (diagram 1)
1.4	Record committed position	write_snapshot (existing header, unchanged)	—	—
2.1/2.2	Restore entities before advancing position	load_snapshot	restore_job, restore_queue_state	Sequence diagram
2.3	Failed reconstruction must not advance position	load_snapshot	rc propagation from restore_job/restore_queue_state	Sequence diagram
2.4	Legacy checkpoint treated as insufficient	load_snapshot, persist_load_snapshot	New entities_restored out-parameter	Sequence diagram
3.1/3.2/3.3	Retention never outruns restorability	persist_checkpoint (existing sequencing, unchanged)	write_snapshot rc gates trim_wal_segments/rotate_snapshots	Write path (diagram 1)
4.1/4.2	End-to-end catch-up correctness	ha_handle_snapshot_chunk (existing, unchanged ordering)	persist_load_snapshot	Receive path (diagram 1)
4.3	Catch-up completeness observable	Existing walcompare/status tooling (out of boundary; confirmed compatible)	—	—
5.1/5.2	Safe rollout against old checkpoints	load_snapshot format discriminator	File-size-based legacy detection	Sequence diagram

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
write_snapshot (extended)	persist.c	Serialize header + live entities to a durable checkpoint file	1.1-1.4, 3.1-3.3	collect_jobs/collect_queues (P0), capnp_encode_job (P0)	State
load_snapshot (extended)	persist.c	Deserialize and restore entities; gate position advance on success	2.1-2.4, 5.1-5.2	restore_job, restore_queue_state (P0)	State
rotate_snapshots (extended)	persist.c	Delete old checkpoint files, skipping any in-flight LSN	3.1-3.3	ha->snapshot_inflight_lsn[] (P0)	State
snapshot_inflight_lsn[] (new)	ha.h / ha.c	Track which checkpoint LSN each in-flight peer transfer references	3.1-3.3	Existing snapshot_inflight[] lifecycle (P0)	State

Persistence Layer

write_snapshot (extended)

Field	Detail
Intent	Produce a durable, entity-bearing checkpoint file at checkpoint_lsn
Requirements	1.1, 1.2, 1.3, 1.4, 3.1, 3.2, 3.3

Responsibilities & Constraints

• Writes the existing legacy header unchanged (magic, checkpoint_lsn, optional config-ref).
• Appends, in order: a format-version marker, a job count and per-job length-prefixed capnp Job frames, a queue count and per-queue fixed (name, status) frames (no capnp — matches restore_queue_state's exact contract; queue definition is config-owned, out of boundary). (Note: superseded by config-upgrade-snapshot — queue frames are now capnp QueueAdminState, and host frames plus a config-bytes section were appended after the queue section; see Data Models below.)
• Collects under RTTI_LOCK_READ on c->jobmgr (released before the queue pass begins); collects under RTTI_LOCK_READ on c->qmgr for queues. The two table locks are never held simultaneously (no new lock-ordering surface). Within each table's held READ lock, also takes each individual entity's own RTTI_LOCK_READ (lock(j, RTTI_LOCK_READ) / lock(q, RTTI_LOCK_READ)) around that one entity's encode/read call — this is required for Requirement 1.2: the table lock alone does not exclude a concurrent per-entity writer (see Architecture).
• Filters via the existing job_should_retain predicate so the checkpoint's notion of "live" matches the existing GC retention policy (Requirement 1.1). Every queue present in c->qmgr is included (a queue's mere table presence already means config says it exists; no retain filter needed for a plain status word).
• Writes to the existing .tmp path, fdatasyncs, then atomically renames to the final path — identical pattern to today, just with more bytes. Any failure at any step (encode, write, sync) leaves no file at the final path (Requirement 1.3); the tmp file is unlinked.

Contracts: State [x]

State Management

• State model: file on disk, atomically visible only at the final path once fully written.
• Persistence & consistency: fdatasync before rename, matching the existing sentinel-only behavior — durability semantics are unchanged, only the payload grows.
• Concurrency strategy: two sequential table-read-lock sections (jobs, then queues), each with a nested per-entity read lock around the individual encode/read call; no lock is held during the disk write itself.

Implementation Notes

• Integration: called from persist_checkpoint, unchanged call site.
• Validation: unit test asserts a checkpoint containing N jobs + M queues round-trips through load_snapshot with all N+M records field-identical.
• Risks: enumeration+encode duration under RTTI_LOCK_READ blocks concurrent job/queue mutation for that duration. Mitigated by checkpoints already running off the hot path on a configurable snap_interval_s cadence; measure actual duration at a realistic job-table size during implementation (see research.md Research Needed).

load_snapshot (extended)

Field	Detail
Intent	Reconstruct entities from a checkpoint file and gate position advance on success
Requirements	2.1, 2.2, 2.3, 2.4, 5.1, 5.2

Responsibilities & Constraints

• Reads the existing legacy header exactly as today (unchanged).
• Determines format: if no bytes remain in the file after the legacy header, this is a pre-feature checkpoint — do not advance current_lsn/checkpoint_lsn as "fully caught up"; return success with entities_restored=false so the caller falls back to normal catch-up (Requirement 2.4). This never crashes or misinterprets an old file (Requirement 5.1, 5.2) — a short file is a valid, expected legacy shape, not a corruption.
• If entity frames are present, decode each job frame (capnp) and call restore_job; read each queue frame's plain (name, status) pair and call restore_queue_state(name, status) directly — in file order. On any decode or restore failure, abort immediately: return an error, leave current_lsn/checkpoint_lsn unchanged (Requirement 2.3).
• Only after every entity restores successfully does the function advance current_lsn/checkpoint_lsn to the checkpoint's position (Requirement 2.1, 2.2).

Contracts: State [x]

State Management

• State model: in-memory job/queue tables (c->jobmgr/c->qmgr), mutated in place by the existing restore_job/restore_queue_state (unmodified — same functions WAL replay already calls, so no new restore code path exists to diverge from replay's behavior).
• Persistence & consistency: current_lsn/checkpoint_lsn advance is the single commit point for "this checkpoint is now fully applied" — atomic in the sense that no partial-entity state is ever exposed as "caught up."
• Concurrency strategy: runs on the receiving member during catch-up, before it accepts write traffic for those entities — no concurrent mutation expected at this point in the lifecycle (same assumption WAL replay already makes at startup).

Public signature change (src/include/persist.h):

int persist_load_snapshot(persist_t* p, const char* path, uint64_t lsn,
                           bool* out_entities_restored);

• Preconditions: path names a file written by write_snapshot (legacy or current format).
• Postconditions: on rc==0, *out_entities_restored reports whether this call actually restored entity state (true) or only observed a legacy position marker (false); current_lsn/ checkpoint_lsn reflect the checkpoint's position only when entities were fully restored or the file was legacy-format (position-only, as today). On rc!=0, position is unchanged.
• Invariants: *out_entities_restored is only set when rc==0.

Implementation Notes

• Integration: ha_handle_snapshot_chunk passes the new out-parameter through; on entities_restored==false it logs (does not fail) since a legacy file is an expected rollout state, not an error — the member simply continues through the existing incremental catch-up path for the remaining history.
• Validation: unit test with a deliberately truncated/corrupted entity frame asserts current_lsn unchanged and rc!=0; unit test with a legacy-sized file asserts rc==0 and entities_restored==false.

rotate_snapshots (extended)

Field	Detail
Intent	Delete old checkpoint files without breaking an in-flight transfer
Requirements	3.1, 3.2, 3.3

Responsibilities & Constraints

• Unchanged selection logic (keep snap_keep_n newest by filename/LSN).
• Before deleting a candidate file, checks whether its LSN matches any entry in ha->snapshot_inflight_lsn[] (via p->cluster->ha, already reachable from persist_t); if so, skip deletion for this rotation pass and retry on the next checkpoint cycle.

Contracts: State [x]

State Management

• State model: reads ha_state_t's new array; does not itself own it (owned by the send-dispatch path, see below).

Implementation Notes

• Integration: minimal — one lookup added to the existing deletion loop.
• Validation: unit test simulates an in-flight LSN matching an otherwise-rotatable file and asserts it survives rotation.

HA Replication Layer

snapshot_inflight_lsn[] (new array)

Field	Detail
Intent	Record which checkpoint LSN each in-flight peer snapshot transfer references
Requirements	3.1, 3.2, 3.3

Responsibilities & Constraints

• _Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES] in ha_state_t (ha.h), declared next to the existing _Atomic int snapshot_inflight[MAX_HA_CANDIDATES].
• Set by ha_snapshot_send_dispatch/ha_snapshot_do_send at the same point the existing snapshot_inflight[peer_idx] flag is set to 1 (before dispatch); cleared at the same points the existing flag is cleared to 0 (on completion or CAS-release failure) — identical lifecycle, one more field.

Contracts: State [x]

State Management

• State model: plain atomic array, same shape and lifecycle as the existing flag array it sits beside.
• Concurrency strategy: no new locking — atomics only, matching the existing flag's discipline.

Implementation Notes

• Integration: two write sites (set/clear), both already-existing call sites in ha.c.
• Validation: covered by the rotate_snapshots guard test above (integration-level: dispatch a send, assert its LSN is protected, complete it, assert protection is released).

Data Models

Checkpoint File Layout (Physical)

[existing legacy header — unchanged, 16 or 57 bytes depending on config-ref presence]
[uint32_t format_version]      -- absent (EOF here) => legacy, position-only file
[uint32_t job_count]
[job_count * { uint32_t payload_len; uint8_t capnp_Job_bytes[payload_len] }]
[uint32_t queue_count]
[queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }]   -- (Note: this raw
                                                                               --  shape is the
                                                                               --  layout AS OF
                                                                               --  this spec's
                                                                               --  close; see below)

Note (superseded by config-upgrade-snapshot, see that spec's design.md): the physical layout above is this spec's as-shipped shape only. config-upgrade-snapshot subsequently (1) migrated the queue frame from the raw {name_len, name, status} shape above to a length-prefixed capnp QueueAdminState frame (same shape as the job-frame loop, content unchanged: name + status), (2) appended a new host-frames section ([uint32_t host_count][host_count * {payload_len, capnp HostAdminState bytes}]) directly after the queue section, and (3) appended a trailing flat length-prefixed config-bytes blob after that ([uint32_t cfg_len][cfg_len bytes], written on every checkpoint). format_version was bumped to 2 for this change. Current source: write_snapshot/load_snapshot_entities in src/controller/persist.c.

• Job frames are length-prefixed capnp Job messages, matching the framing style already used for WAL entries (wal_entry_hdr_t + payload) — no new schema/codegen for a one-off container.
• Queue frames carry only what restore_queue_state can consume: a name and a status word — no capnp at all. A queue's definition (attrs, priority, thresholds, ACLs) is config-provisioned and already replicated via push_config_to_standbys; encoding it here would duplicate data this spec does not own and has no restore path for. (Note: the wire encoding of this same name+status pair was later moved to capnp — see the note above.)
• A file ending exactly at the legacy header (no bytes remain) is unambiguously the old format — the discriminator is "does more data exist," not a flag bit, so old files need zero reinterpretation.
• format_version is present for forward evolution (a future layout bumps this value and load_snapshot dispatches on it) without breaking this design's readers. (Note: superseded by config-upgrade-snapshot's actual choice, see that spec's design.md — when it bumped format_version to 2 it made load_snapshot hard-reject any non-matching version with -EINVAL rather than dispatch, on the reasoning that a version mismatch must never be silently masked as an expected legacy shape or real corruption could go undetected. The legacy (no-format_version-present) detection this bullet's neighbor describes is unaffected.)

Consistency & Integrity

• Per-entity encode happens while both the enumerating table's RTTI_LOCK_READ AND that entity's own nested RTTI_LOCK_READ are held (see Architecture) — the table lock alone is insufficient, since entity mutation takes a separate per-entity lock; the nested per-entity lock is what excludes a concurrent writer for that entity's encode, giving each entity a single consistent snapshot of its own fields with no entity encoded mid-mutation.
• Cross-entity consistency (jobs relative to queues) is not required by Requirement 1.2, which is scoped to "no record split between its own pre/post-checkpoint value" — jobs and queues are independent aggregates for this feature's purposes, consistent with collect_jobs/ collect_queues already treating them as separately-locked tables today.

Error Handling

Error Strategy

• Write-side failure (encode error, disk I/O error) at any point during write_snapshot: unlink the tmp file, return a non-zero rc, which the existing persist_checkpoint caller already treats as "skip trim and rotate this cycle" — no operational-condition self-termination (per src/controller/CLAUDE.md's controller-never-self-terminates invariant); the next checkpoint cycle simply retries.
• Read-side failure (corrupt/truncated entity frame, decode error, restore error) during load_snapshot: return a non-zero rc with position unchanged; the caller (ha_handle_snapshot_chunk) logs the failure and does not advance applied_lsn — the member stays fenced/behind and the existing catch-up retry mechanics (unchanged by this spec) bring it current on a subsequent attempt. Consistent with the project's fail-loud-and-stay-alive philosophy — never a self-terminate on this operational condition.
• Legacy file encountered: not an error. entities_restored=false with rc==0 is the expected, handled outcome for a file predating this feature.

Monitoring

• Existing WARN/ERROR log lines at each new failure point (matching the codebase's existing logging conventions in persist.c/ha.c); no new metric is introduced by this design, but Requirement 4.3's "observable to an operator" is satisfied by the existing walcompare/ waldump tooling already used this session to diagnose divergence — confirm during implementation that it reports on entity-snapshot-derived state, not only WAL-visible LSNs.

Testing Strategy

• Unit — write_snapshot/load_snapshot round-trip: a checkpoint with N jobs and M queues, loaded back, yields N job records field-identical to the originals and M queue statuses correctly reapplied via restore_queue_state (Requirement 1.1, 1.2, 2.1, 2.2).
• Unit — queue status frame round-trip: a queue's (name, status) pair written and read back matches exactly, and restore_queue_state is invoked with those exact values (Requirement 1.1).
• Unit — concurrent-mutation-during-snapshot consistency: a job actively transitioning state (e.g. via mark_job_start) on a concurrent thread while write_snapshot's enumeration walks it must be captured as either the full pre- or full post-transition value, never a torn mix of fields — proves the nested per-entity lock, not just the table lock, is doing the work (Requirement 1.2).
• Unit — partial write discarded: a simulated encode/write failure partway through leaves no file at the final path (Requirement 1.3).
• Unit — partial restore does not advance position: a deliberately corrupted entity frame causes load_snapshot to return an error with current_lsn/checkpoint_lsn unchanged (Requirement 2.3).
• Unit — legacy file handling: a file sized to end exactly at the legacy header yields rc==0, entities_restored=false, no crash (Requirement 2.4, 5.1, 5.2).
• Unit — rotation guard: a file matching an in-flight LSN survives rotate_snapshots even when it would otherwise fall outside snap_keep_n (Requirement 3.1, 3.2, 3.3).
• Integration — chaotic catch-up soak: extend the existing /xce-test-ha --chaotic soak scenario so at least one round forces a SEND_SNAPSHOT catch-up (empty or far-behind member); assert committed_divergence=0 and committed_job_loss=0 post-round (Requirement 4.1, 4.2) — this is the direct acceptance test for the residual soak failure this spec exists to close.
• Integration — rolling upgrade simulation: a pre-feature-format checkpoint file present on disk, loaded by the new binary, does not crash and correctly falls back to full catch-up rather than reporting false-positive completeness (Requirement 5.1, 5.2).