# Design Document: ha-entity-snapshots

## Overview

**Purpose**: This feature makes the XCE controller's periodic persistence checkpoint an
actually-restorable copy of committed job and queue state, closing the permanent data gap a
rejoining or far-behind cluster member hits today (task 5.10a / raft-ha-core-safety; bug-795/
828/851 family). Today `write_snapshot` records only a position marker; `load_snapshot` restores
nothing. This design adds the missing entity payload and restore step, and closes the one real
retention hazard it introduces (a snapshot file being rotated out from under an in-flight
transfer).

**Users**: The XCE HA controller cluster itself is the direct consumer — every member that
rejoins empty or falls behind the retained WAL history. The observable beneficiary is the
cluster operator, who currently sees a silently incomplete member after such a rejoin.

**Impact**: Modifies the existing checkpoint write/read/rotate logic in `persist.c` and adds one
small tracking array in `ha.c`. No wire protocol change, no new external dependency, no new
capnp schema struct — the existing `Job` schema and its generated codec already carry everything
needed for jobs. Queue *definitions* (name, attrs, priority, thresholds, ACLs) are config-
provisioned and already replicated independently via `push_config_to_standbys`
(`config_repl.c`) — out of this spec's boundary. Only a queue's runtime admin status
(open/closed/active/inactive), the one piece `WAL_QUEUE_CTRL` tracks and config replication does
not carry, is this spec's concern — restored via the existing `restore_queue_state(name, status)`,
no capnp involved on the queue side at all. (Note: the on-disk wire format was later migrated to a
capnp `QueueAdminState` frame — superseded by config-upgrade-snapshot, see that spec's design.md;
`restore_queue_state`'s own `(name, status)` call contract is unchanged.)

### Goals
- A checkpoint written at LSN X contains every live/retained job and queue record as of X.
- Loading that checkpoint reconstructs those records before the member's position advances.
- History is never discarded before it is captured in a checkpoint whose entities are durable.
- A member that rejoins via checkpoint ends up bit-for-bit consistent with the rest of the
  cluster — no permanent gap.
- A checkpoint produced by a prior version of the binary never crashes or corrupts a member
  that reads it.

### Non-Goals
- Lock-free, copy-on-write entity enumeration (zarray COW exists but is not wired into `kvmgr`;
  reviving that wiring is a separate, larger effort — see `research.md` Option B).
- Any change to the wire transport/chunking protocol that ships a snapshot to a peer.
- Any change to the catch-up decision logic (`ha_evaluate_repl_action`) that chooses between a
  snapshot transfer and incremental WAL streaming.
- Worker/compute-node state, cluster configuration document snapshotting, or election/term-safety
  behavior.
- Queue *definition* fields (name, attrs, priority, thresholds, windows, ACLs) — these are
  config-provisioned and already replicated independently (`push_config_to_standbys`); this spec
  snapshots only the queue's runtime admin status.

## Boundary Commitments

### This Spec Owns
- The on-disk content format of a checkpoint (entity payload framing) and the write/read logic
  that produces and consumes it (`write_snapshot`, `load_snapshot` internals).
- Point-in-time collection and encoding of live job and queue entities at checkpoint time.
- The entity-restore step on load and gating the member's committed-position advance on that
  step's success.
- Distinguishing a legacy (pre-feature) position-only checkpoint from an entity-bearing one.
- Protecting an in-flight-transferring checkpoint file from premature rotation deletion.

### Out of Boundary
- The wire opcode and chunk-reassembly protocol that delivers a checkpoint to a peer
  (`OPCode_snapshotTransfer`, `ha_handle_snapshot_chunk`'s framing) — reused unchanged.
- The decision of *when* a member needs a checkpoint vs. incremental streaming
  (`ha_evaluate_repl_action`) — unchanged.
- The WAL-segment trim floor's standby-acked-LSN computation (`repl_trim_lsn`) — unchanged; see
  Architecture below for why no new floor input is needed there.
- Worker/join state, config-document snapshotting (`config-upgrade-snapshot`), election and
  term-safety behavior (`raft-ha-core-safety`, `ha-term-safety`).

### Allowed Dependencies
- Existing capnp codegen for `Job` (`encode_Job`/`write_Job`/`decode_Job`, already generated from
  `src/schema/controller.capnp`). Queue entries need no capnp — `restore_queue_state`'s contract
  is a plain `(name, status)` pair (see Architecture below). (Note: config-upgrade-snapshot later
  added a capnp `QueueAdminState` frame as the on-disk encoding of that same pair — see Data
  Models below.)
- `restore_job` (`job.c:274`) and `restore_queue_state` (`queue.c:208`) — existing, unmodified
  entity-restore functions already used by WAL replay.
- The `collect_jobs`/`collect_queues` enumeration pattern (`dataex.c`) and the
  `job_should_retain`/`worker_should_retain` retention predicates (`persist.c`).
- The existing tmp-file-then-atomic-rename pattern already used by `write_snapshot` and by
  `ha_handle_snapshot_chunk`'s `snap.part` assembly.
- The existing per-peer `ha->snapshot_inflight[]` array shape (`ha.h`) as the pattern for the new
  in-flight-LSN tracking.

### Revalidation Triggers
- Any change to the `Job` capnp schema shape — re-verify encode/decode symmetry and compatibility
  with `restore_job`.
- Any change to `restore_queue_state`'s `(name, status)` contract, or any future need to snapshot
  more than status for queues — re-verify the queue-side format and the config-vs-WAL boundary
  this design draws.
- Any change to `collect_jobs`/`collect_queues` locking semantics, or to the per-entity
  `RTTI_LOCK_READ`/`WRITE` discipline (e.g. if `kvmgr` COW is wired later) — re-verify the
  point-in-time consistency argument in Architecture below.
- Any change to `persist_checkpoint`'s `write_snapshot` → `trim_wal_segments` sequencing
  (currently sequential, trim skipped on write failure) — re-verify Requirement 3.
- Any change to `ha_handle_snapshot_chunk`'s ordering (currently: `applied_lsn` only advances
  after `persist_load_snapshot` returns success) — re-verify Requirement 4.

## Architecture

### Existing Architecture Analysis

- `persist_checkpoint` (persist.c:2820) already sequences correctly for Requirement 3: it calls
  `write_snapshot`, and only calls `trim_wal_segments` if `write_snapshot` returned 0
  (`if (rc != 0) goto done;`, skipping both trim and `rotate_snapshots`). **This means once
  `write_snapshot` only returns 0 when the entity payload is fully and durably written, the
  existing trim gating already satisfies "retention never outruns restorability" — no new floor
  variable is needed in `trim_wal_segments`/`repl_trim_lsn`.** This is a refinement over the
  initial gap analysis (`research.md`), which assumed a new trim-floor input was needed.
- The one retention hazard `write_snapshot`'s success does *not* already guard is
  `rotate_snapshots` (persist.c:2677): it keeps only `snap_keep_n` newest snapshot *files* by
  filename sort and deletes the rest, with no awareness of an in-flight transfer. If a checkpoint
  cadence faster than a slow transfer rotates out the file a standby is still reading from a
  chunked wire transfer, that transfer breaks. This is the one genuinely new guard this design
  adds.
- `ha_handle_snapshot_chunk` (ha.c:3635-3794) already has the right shape for Requirement 4: it
  only advances `ha->applied_lsn` after `persist_load_snapshot` returns success. Once
  `persist_load_snapshot`/`load_snapshot` correctly gate their own success on entity restoration
  (Requirement 2), this call site needs no change.
- `collect_jobs`/`collect_queues` (dataex.c) already hold `RTTI_LOCK_READ` on `c->jobmgr`/
  `c->qmgr` for the duration of their walk. **This table lock is a distinct lock from the
  per-entity lock** — verified via `restore_queue_state` (queue.c:208), which takes
  `lock(q, RTTI_LOCK_WRITE)` on the individual `queue_t*` after already releasing the table lock
  from its own `key_find`, and via `job.c`'s state-transition sites (e.g. lines 1006, 1093, 1137),
  which take `lock(j, RTTI_LOCK_WRITE)` on the individual `job_t*` (line numbers shift as job.c
  grows; verify by grepping `lock(j, RTTI_LOCK_WRITE)` if re-checking this). The table lock only excludes
  concurrent insert/delete of table entries; it does **not** exclude a concurrent per-entity
  writer mutating an entity's fields mid-walk. Requirement 1.2's point-in-time consistency
  therefore requires the entity-collection helper to additionally take each entity's own
  `RTTI_LOCK_READ` (mirroring `restore_queue_state`'s own per-entity write-lock pattern) around
  that entity's individual encode call, nested inside the table-level walk — this excludes a
  concurrent per-entity writer for exactly the duration of that one entity's encode, without
  needing the (currently unwired) zarray COW mechanism.

### Architecture Pattern & Boundary Map

```mermaid
flowchart TB
    subgraph Primary["Primary — checkpoint write path"]
        PC[persist_checkpoint] --> WS[write_snapshot]
        WS --> CJ["collect+encode jobs\n(table READ lock + per-job READ lock)"]
        WS --> CQ["collect name+status pairs\n(table READ lock + per-queue READ lock,\nnow capnp QueueAdminState --\nconfig-upgrade-snapshot)"]
        CJ --> TMP[tmp file + fdatasync + rename]
        CQ --> TMP
        TMP -->|rc==0| TRIM[trim_wal_segments]
        TMP -->|rc==0| ROT["rotate_snapshots\n(skips in-flight LSNs)"]
        TMP -->|rc!=0| SKIP["skip trim + rotate\n(existing goto done)"]
    end

    subgraph Send["Snapshot transfer (unchanged)"]
        DISPATCH[ha_snapshot_send_dispatch] -->|sets snapshot_inflight_lsn for peer| WIRE[OPCode_snapshotTransfer chunks]
        WIRE --> DONE[clears snapshot_inflight_lsn on completion]
    end

    ROT -.reads.-> DISPATCH

    subgraph Standby["Standby — catch-up receive path"]
        RECV[ha_handle_snapshot_chunk] --> RENAME[assemble + rename final file]
        RENAME --> LS[persist_load_snapshot]
        LS --> RJ["restore_job per entity"]
        LS --> RQ["restore_queue_state per entity"]
        RJ -->|all succeed| ADV["advance current_lsn / checkpoint_lsn"]
        RQ -->|all succeed| ADV
        RJ -->|any fails| FAIL["return error, position unchanged"]
        ADV --> APPLIED["ha->applied_lsn advances\n(existing, unchanged)"]
    end
```

**Architecture Integration**:
- Selected pattern: extend the existing checkpoint write/read pipeline in place (Option A from
  `research.md`); no new subsystem, no new wire message.
- Domain boundaries: entity content and restoration stay inside `persist.c`
  (write/load/rotate); the one cross-file touch is the new `snapshot_inflight_lsn[]` array in
  `ha.h`/`ha.c`, read by `rotate_snapshots` and written by the existing send-dispatch path.
- Existing patterns preserved: tmp-file-then-rename atomicity, `RTTI_LOCK_READ`/`WRITE` table AND
  per-entity locking (mirroring `restore_queue_state`'s own nested-lock pattern), the
  `collect_jobs`/`collect_queues` iteration shape, the `capnp_encode_job` wrapper shape.
- New components rationale: a small entity-frame reader/writer pair (mirrors the existing WAL
  entry length-prefixed framing style, so no new capnp schema struct or codegen is required —
  jobs use the existing `Job` capnp codec, queues use a plain fixed `(name, status)` record since
  that is the entirety of `restore_queue_state`'s contract); the in-flight-LSN guard in
  `rotate_snapshots`. (Note: "no new capnp schema struct" for queues held at this spec's close but
  was superseded shortly after by config-upgrade-snapshot, which added a `QueueAdminState` capnp
  struct and migrated the queue frame to it — see that spec's design.md.)
- Steering compliance: no `fork()` used for the snapshot (per `product.md`'s "No fork() for
  snapshots" constraint) — consistency comes from the table read-lock, not process duplication.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Serialization | Cap'n Proto (existing project-wide IDL) | Per-job `Job` encode/decode; queues use a plain fixed record (no capnp) | Codegen already exists; no schema change |
| Persistence | Flat file, tmp+rename (existing pattern) | Checkpoint file on disk | Same pattern as today's sentinel-only file |
| Concurrency | `RTTI_LOCK_READ`/`WRITE`, table AND per-entity (existing) | Point-in-time entity consistency during collection | zarray COW deferred (Non-Goal) |
| Transport | `OPCode_snapshotTransfer` (existing, unmodified) | Ships the (now larger) file to a standby | Out of boundary |

## File Structure Plan

### Modified Files
- `src/controller/persist.c` — `write_snapshot`: append entity payload after the existing
  header — job frames (capnp `Job`) and queue frames (plain `(name, status)` records, no capnp).
  (Note: queue frames were later migrated to capnp `QueueAdminState`, and host frames plus a
  trailing config-bytes section were appended after them — superseded by config-upgrade-snapshot,
  see Data Models below and that spec's design.md.)
  `load_snapshot`: restore entities before advancing `current_lsn`/`checkpoint_lsn`; detect
  legacy (position-only) files by remaining file size. `rotate_snapshots`: skip deleting any file
  whose LSN is currently in-flight. New static helpers: a snapshot entity-frame writer/reader
  pair, and an entity-collection helper built on `collect_jobs`/`collect_queues` that additionally
  takes each entity's own per-entity `RTTI_LOCK_READ` around its individual encode/collect call
  (see Architecture).
- `src/include/persist.h` — extend `persist_load_snapshot`'s signature with an out-parameter
  reporting whether entities were restored (see Components below); no other public surface
  changes.
- `src/controller/ha.h` — add `_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]` next
  to the existing `snapshot_inflight[MAX_HA_CANDIDATES]` array.
- `src/controller/ha.c` — `ha_snapshot_send_dispatch`/`ha_snapshot_do_send`: record the LSN being
  sent into `snapshot_inflight_lsn[peer_idx]` alongside the existing flag, clear it on completion
  (same lifecycle as the existing flag). `ha_handle_snapshot_chunk`: read the new
  `persist_load_snapshot` out-parameter; log (not fail) when entities were not restored, since
  the caller-side catch-up-completeness signal (Requirement 4.3) is carried by existing
  status/comparison tooling, not a new error path here.

### New Test Files
- `src/controller/testsuite/test_persist_snapshot_entities.c` — unit coverage for
  `write_snapshot`/`load_snapshot` entity round-trip, partial-failure discard, legacy-file
  handling, and the `rotate_snapshots` in-flight guard (see Testing Strategy).

> No new production files. This stays entirely inside the existing checkpoint machinery's file
> set plus one small, symmetric addition in `ha.h`/`ha.c` for the rotation guard.

## System Flows

See the Architecture Pattern & Boundary Map diagram above for the write/send/receive flow. One
additional sequence clarifies the failure-ordering contract that Requirement 2.3 depends on:

```mermaid
sequenceDiagram
    participant Standby
    participant persist_load_snapshot
    participant restore_job
    participant restore_queue_state

    Standby->>persist_load_snapshot: load(path, lsn)
    persist_load_snapshot->>persist_load_snapshot: read header, detect format
    alt legacy (position-only) file
        persist_load_snapshot-->>Standby: rc=0, entities_restored=false
    else entity-bearing file
        loop each job frame
            persist_load_snapshot->>restore_job: decode + restore
            restore_job-->>persist_load_snapshot: rc
            alt rc != 0
                persist_load_snapshot-->>Standby: rc=error, position UNCHANGED
            end
        end
        loop each queue frame
            persist_load_snapshot->>restore_queue_state: restore(name, status)
            restore_queue_state-->>persist_load_snapshot: rc
            alt rc != 0
                persist_load_snapshot-->>Standby: rc=error, position UNCHANGED
            end
        end
        persist_load_snapshot->>persist_load_snapshot: advance current_lsn/checkpoint_lsn
        persist_load_snapshot-->>Standby: rc=0, entities_restored=true
    end
```

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|---|---|---|---|---|
| 1.1 | Checkpoint includes every live/retained job+queue record | `write_snapshot`, entity-collection helper | `collect_jobs`/`collect_queues` + `job_should_retain`; queue side is `(name, status)` only (definition is config-owned) | Write path (diagram 1) |
| 1.2 | Internally consistent point-in-time view | Entity-collection helper | Table `RTTI_LOCK_READ` + per-entity `RTTI_LOCK_READ` nested around each encode | Architecture: Existing Architecture Analysis |
| 1.3 | Discard incomplete checkpoint | `write_snapshot` | Existing tmp+rename pattern | Write path (diagram 1) |
| 1.4 | Record committed position | `write_snapshot` (existing header, unchanged) | — | — |
| 2.1/2.2 | Restore entities before advancing position | `load_snapshot` | `restore_job`, `restore_queue_state` | Sequence diagram |
| 2.3 | Failed reconstruction must not advance position | `load_snapshot` | rc propagation from `restore_job`/`restore_queue_state` | Sequence diagram |
| 2.4 | Legacy checkpoint treated as insufficient | `load_snapshot`, `persist_load_snapshot` | New `entities_restored` out-parameter | Sequence diagram |
| 3.1/3.2/3.3 | Retention never outruns restorability | `persist_checkpoint` (existing sequencing, unchanged) | `write_snapshot` rc gates `trim_wal_segments`/`rotate_snapshots` | Write path (diagram 1) |
| 4.1/4.2 | End-to-end catch-up correctness | `ha_handle_snapshot_chunk` (existing, unchanged ordering) | `persist_load_snapshot` | Receive path (diagram 1) |
| 4.3 | Catch-up completeness observable | Existing `walcompare`/status tooling (out of boundary; confirmed compatible) | — | — |
| 5.1/5.2 | Safe rollout against old checkpoints | `load_snapshot` format discriminator | File-size-based legacy detection | Sequence diagram |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|---|---|---|---|---|---|
| `write_snapshot` (extended) | persist.c | Serialize header + live entities to a durable checkpoint file | 1.1-1.4, 3.1-3.3 | `collect_jobs`/`collect_queues` (P0), `capnp_encode_job` (P0) | State |
| `load_snapshot` (extended) | persist.c | Deserialize and restore entities; gate position advance on success | 2.1-2.4, 5.1-5.2 | `restore_job`, `restore_queue_state` (P0) | State |
| `rotate_snapshots` (extended) | persist.c | Delete old checkpoint files, skipping any in-flight LSN | 3.1-3.3 | `ha->snapshot_inflight_lsn[]` (P0) | State |
| `snapshot_inflight_lsn[]` (new) | ha.h / ha.c | Track which checkpoint LSN each in-flight peer transfer references | 3.1-3.3 | Existing `snapshot_inflight[]` lifecycle (P0) | State |

### Persistence Layer

#### `write_snapshot` (extended)

| Field | Detail |
|-------|--------|
| Intent | Produce a durable, entity-bearing checkpoint file at `checkpoint_lsn` |
| Requirements | 1.1, 1.2, 1.3, 1.4, 3.1, 3.2, 3.3 |

**Responsibilities & Constraints**
- Writes the existing legacy header unchanged (magic, `checkpoint_lsn`, optional config-ref).
- Appends, in order: a format-version marker, a job count and per-job length-prefixed capnp `Job`
  frames, a queue count and per-queue fixed `(name, status)` frames (no capnp — matches
  `restore_queue_state`'s exact contract; queue definition is config-owned, out of boundary).
  (Note: superseded by config-upgrade-snapshot — queue frames are now capnp `QueueAdminState`,
  and host frames plus a config-bytes section were appended after the queue section; see Data
  Models below.)
- Collects under `RTTI_LOCK_READ` on `c->jobmgr` (released before the queue pass begins); collects
  under `RTTI_LOCK_READ` on `c->qmgr` for queues. The two table locks are never held
  simultaneously (no new lock-ordering surface). **Within each table's held READ lock, also takes
  each individual entity's own `RTTI_LOCK_READ`** (`lock(j, RTTI_LOCK_READ)` /
  `lock(q, RTTI_LOCK_READ)`) around that one entity's encode/read call — this is required for
  Requirement 1.2: the table lock alone does not exclude a concurrent per-entity writer (see
  Architecture).
- Filters via the existing `job_should_retain` predicate so the checkpoint's notion of "live"
  matches the existing GC retention policy (Requirement 1.1). Every queue present in `c->qmgr` is
  included (a queue's mere table presence already means config says it exists; no retain filter
  needed for a plain status word).
- Writes to the existing `.tmp` path, `fdatasync`s, then atomically renames to the final path —
  identical pattern to today, just with more bytes. Any failure at any step (encode, write,
  sync) leaves no file at the final path (Requirement 1.3); the tmp file is unlinked.

**Contracts**: State [x]

##### State Management
- State model: file on disk, atomically visible only at the final path once fully written.
- Persistence & consistency: `fdatasync` before rename, matching the existing sentinel-only
  behavior — durability semantics are unchanged, only the payload grows.
- Concurrency strategy: two sequential table-read-lock sections (jobs, then queues), each with a
  nested per-entity read lock around the individual encode/read call; no lock is held during the
  disk write itself.

**Implementation Notes**
- Integration: called from `persist_checkpoint`, unchanged call site.
- Validation: unit test asserts a checkpoint containing N jobs + M queues round-trips through
  `load_snapshot` with all N+M records field-identical.
- Risks: enumeration+encode duration under `RTTI_LOCK_READ` blocks concurrent job/queue mutation
  for that duration. Mitigated by checkpoints already running off the hot path on a configurable
  `snap_interval_s` cadence; measure actual duration at a realistic job-table size during
  implementation (see `research.md` Research Needed).

#### `load_snapshot` (extended)

| Field | Detail |
|-------|--------|
| Intent | Reconstruct entities from a checkpoint file and gate position advance on success |
| Requirements | 2.1, 2.2, 2.3, 2.4, 5.1, 5.2 |

**Responsibilities & Constraints**
- Reads the existing legacy header exactly as today (unchanged).
- Determines format: if no bytes remain in the file after the legacy header, this is a
  pre-feature checkpoint — do not advance `current_lsn`/`checkpoint_lsn` as "fully caught up";
  return success with `entities_restored=false` so the caller falls back to normal catch-up
  (Requirement 2.4). This never crashes or misinterprets an old file (Requirement 5.1, 5.2) — a
  short file is a valid, expected legacy shape, not a corruption.
- If entity frames are present, decode each job frame (capnp) and call `restore_job`; read each
  queue frame's plain `(name, status)` pair and call `restore_queue_state(name, status)` directly
  — in file order. On any decode or restore failure, abort immediately: return an error, leave
  `current_lsn`/`checkpoint_lsn` unchanged (Requirement 2.3).
- Only after every entity restores successfully does the function advance
  `current_lsn`/`checkpoint_lsn` to the checkpoint's position (Requirement 2.1, 2.2).

**Contracts**: State [x]

##### State Management
- State model: in-memory job/queue tables (`c->jobmgr`/`c->qmgr`), mutated in place by the
  existing `restore_job`/`restore_queue_state` (unmodified — same functions WAL replay already
  calls, so no new restore code path exists to diverge from replay's behavior).
- Persistence & consistency: `current_lsn`/`checkpoint_lsn` advance is the single commit point
  for "this checkpoint is now fully applied" — atomic in the sense that no partial-entity state
  is ever exposed as "caught up."
- Concurrency strategy: runs on the receiving member during catch-up, before it accepts write
  traffic for those entities — no concurrent mutation expected at this point in the lifecycle
  (same assumption WAL replay already makes at startup).

**Public signature change** (`src/include/persist.h`):
```c
int persist_load_snapshot(persist_t* p, const char* path, uint64_t lsn,
                           bool* out_entities_restored);
```
- Preconditions: `path` names a file written by `write_snapshot` (legacy or current format).
- Postconditions: on rc==0, `*out_entities_restored` reports whether this call actually restored
  entity state (`true`) or only observed a legacy position marker (`false`); `current_lsn`/
  `checkpoint_lsn` reflect the checkpoint's position only when entities were fully restored or
  the file was legacy-format (position-only, as today). On rc!=0, position is unchanged.
- Invariants: `*out_entities_restored` is only set when rc==0.

**Implementation Notes**
- Integration: `ha_handle_snapshot_chunk` passes the new out-parameter through; on
  `entities_restored==false` it logs (does not fail) since a legacy file is an expected rollout
  state, not an error — the member simply continues through the existing incremental catch-up
  path for the remaining history.
- Validation: unit test with a deliberately truncated/corrupted entity frame asserts
  `current_lsn` unchanged and rc!=0; unit test with a legacy-sized file asserts rc==0 and
  `entities_restored==false`.

#### `rotate_snapshots` (extended)

| Field | Detail |
|-------|--------|
| Intent | Delete old checkpoint files without breaking an in-flight transfer |
| Requirements | 3.1, 3.2, 3.3 |

**Responsibilities & Constraints**
- Unchanged selection logic (keep `snap_keep_n` newest by filename/LSN).
- Before deleting a candidate file, checks whether its LSN matches any entry in
  `ha->snapshot_inflight_lsn[]` (via `p->cluster->ha`, already reachable from `persist_t`); if so,
  skip deletion for this rotation pass and retry on the next checkpoint cycle.

**Contracts**: State [x]

##### State Management
- State model: reads `ha_state_t`'s new array; does not itself own it (owned by the send-dispatch
  path, see below).

**Implementation Notes**
- Integration: minimal — one lookup added to the existing deletion loop.
- Validation: unit test simulates an in-flight LSN matching an otherwise-rotatable file and
  asserts it survives rotation.

### HA Replication Layer

#### `snapshot_inflight_lsn[]` (new array)

| Field | Detail |
|-------|--------|
| Intent | Record which checkpoint LSN each in-flight peer snapshot transfer references |
| Requirements | 3.1, 3.2, 3.3 |

**Responsibilities & Constraints**
- `_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]` in `ha_state_t` (ha.h), declared
  next to the existing `_Atomic int snapshot_inflight[MAX_HA_CANDIDATES]`.
- Set by `ha_snapshot_send_dispatch`/`ha_snapshot_do_send` at the same point the existing
  `snapshot_inflight[peer_idx]` flag is set to 1 (before dispatch); cleared at the same points
  the existing flag is cleared to 0 (on completion or CAS-release failure) — identical lifecycle,
  one more field.

**Contracts**: State [x]

##### State Management
- State model: plain atomic array, same shape and lifecycle as the existing flag array it sits
  beside.
- Concurrency strategy: no new locking — atomics only, matching the existing flag's discipline.

**Implementation Notes**
- Integration: two write sites (set/clear), both already-existing call sites in `ha.c`.
- Validation: covered by the `rotate_snapshots` guard test above (integration-level: dispatch a
  send, assert its LSN is protected, complete it, assert protection is released).

## Data Models

### Checkpoint File Layout (Physical)

```
[existing legacy header — unchanged, 16 or 57 bytes depending on config-ref presence]
[uint32_t format_version]      -- absent (EOF here) => legacy, position-only file
[uint32_t job_count]
[job_count * { uint32_t payload_len; uint8_t capnp_Job_bytes[payload_len] }]
[uint32_t queue_count]
[queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }]   -- (Note: this raw
                                                                               --  shape is the
                                                                               --  layout AS OF
                                                                               --  this spec's
                                                                               --  close; see below)
```

> **Note (superseded by config-upgrade-snapshot, see that spec's design.md):** the physical layout
> above is this spec's as-shipped shape only. config-upgrade-snapshot subsequently (1) migrated the
> queue frame from the raw `{name_len, name, status}` shape above to a length-prefixed capnp
> `QueueAdminState` frame (same shape as the job-frame loop, content unchanged: name + status),
> (2) appended a new host-frames section (`[uint32_t host_count][host_count * {payload_len,
> capnp HostAdminState bytes}]`) directly after the queue section, and (3) appended a trailing
> flat length-prefixed config-bytes blob after that (`[uint32_t cfg_len][cfg_len bytes]`, written
> on every checkpoint). `format_version` was bumped to `2` for this change. Current source:
> `write_snapshot`/`load_snapshot_entities` in `src/controller/persist.c`.

- Job frames are length-prefixed capnp `Job` messages, matching the framing style already used
  for WAL entries (`wal_entry_hdr_t` + payload) — no new schema/codegen for a one-off container.
- Queue frames carry only what `restore_queue_state` can consume: a name and a status word — no
  capnp at all. A queue's *definition* (attrs, priority, thresholds, ACLs) is config-provisioned
  and already replicated via `push_config_to_standbys`; encoding it here would duplicate data this
  spec does not own and has no restore path for. (Note: the wire encoding of this same
  name+status pair was later moved to capnp — see the note above.)
- A file ending exactly at the legacy header (no bytes remain) is unambiguously the old format —
  the discriminator is "does more data exist," not a flag bit, so old files need zero
  reinterpretation.
- `format_version` is present for forward evolution (a future layout bumps this value and
  `load_snapshot` dispatches on it) without breaking this design's readers. (Note: superseded by
  config-upgrade-snapshot's actual choice, see that spec's design.md — when it bumped
  `format_version` to `2` it made `load_snapshot` hard-reject any non-matching version with
  `-EINVAL` rather than dispatch, on the reasoning that a version mismatch must never be silently
  masked as an expected legacy shape or real corruption could go undetected. The legacy
  (no-`format_version`-present) detection this bullet's neighbor describes is unaffected.)

### Consistency & Integrity
- Per-entity encode happens while both the enumerating table's `RTTI_LOCK_READ` AND that entity's
  own nested `RTTI_LOCK_READ` are held (see Architecture) — the table lock alone is insufficient,
  since entity mutation takes a separate per-entity lock; the nested per-entity lock is what
  excludes a concurrent writer for that entity's encode, giving each entity a single consistent
  snapshot of its own fields with no entity encoded mid-mutation.
- Cross-entity consistency (jobs relative to queues) is not required by Requirement 1.2, which
  is scoped to "no record split between its own pre/post-checkpoint value" — jobs and queues are
  independent aggregates for this feature's purposes, consistent with `collect_jobs`/
  `collect_queues` already treating them as separately-locked tables today.

## Error Handling

### Error Strategy
- **Write-side failure** (encode error, disk I/O error) at any point during `write_snapshot`:
  unlink the tmp file, return a non-zero rc, which the existing `persist_checkpoint` caller
  already treats as "skip trim and rotate this cycle" — no operational-condition self-termination
  (per `src/controller/CLAUDE.md`'s controller-never-self-terminates invariant); the next
  checkpoint cycle simply retries.
- **Read-side failure** (corrupt/truncated entity frame, decode error, restore error) during
  `load_snapshot`: return a non-zero rc with position unchanged; the caller
  (`ha_handle_snapshot_chunk`) logs the failure and does not advance `applied_lsn` — the member
  stays fenced/behind and the existing catch-up retry mechanics (unchanged by this spec) bring it
  current on a subsequent attempt. Consistent with the project's fail-loud-and-stay-alive
  philosophy — never a self-terminate on this operational condition.
- **Legacy file encountered**: not an error. `entities_restored=false` with rc==0 is the expected,
  handled outcome for a file predating this feature.

### Monitoring
- Existing `WARN`/`ERROR` log lines at each new failure point (matching the codebase's existing
  logging conventions in `persist.c`/`ha.c`); no new metric is introduced by this design, but
  Requirement 4.3's "observable to an operator" is satisfied by the existing `walcompare`/
  `waldump` tooling already used this session to diagnose divergence — confirm during
  implementation that it reports on entity-snapshot-derived state, not only WAL-visible LSNs.

## Testing Strategy

- **Unit — `write_snapshot`/`load_snapshot` round-trip**: a checkpoint with N jobs and M queues,
  loaded back, yields N job records field-identical to the originals and M queue statuses
  correctly reapplied via `restore_queue_state` (Requirement 1.1, 1.2, 2.1, 2.2).
- **Unit — queue status frame round-trip**: a queue's `(name, status)` pair written and read back
  matches exactly, and `restore_queue_state` is invoked with those exact values (Requirement 1.1).
- **Unit — concurrent-mutation-during-snapshot consistency**: a job actively transitioning state
  (e.g. via `mark_job_start`) on a concurrent thread while `write_snapshot`'s enumeration walks it
  must be captured as either the full pre- or full post-transition value, never a torn mix of
  fields — proves the nested per-entity lock, not just the table lock, is doing the work
  (Requirement 1.2).
- **Unit — partial write discarded**: a simulated encode/write failure partway through leaves no
  file at the final path (Requirement 1.3).
- **Unit — partial restore does not advance position**: a deliberately corrupted entity frame
  causes `load_snapshot` to return an error with `current_lsn`/`checkpoint_lsn` unchanged
  (Requirement 2.3).
- **Unit — legacy file handling**: a file sized to end exactly at the legacy header yields rc==0,
  `entities_restored=false`, no crash (Requirement 2.4, 5.1, 5.2).
- **Unit — rotation guard**: a file matching an in-flight LSN survives `rotate_snapshots` even
  when it would otherwise fall outside `snap_keep_n` (Requirement 3.1, 3.2, 3.3).
- **Integration — chaotic catch-up soak**: extend the existing `/xce-test-ha --chaotic` soak
  scenario so at least one round forces a `SEND_SNAPSHOT` catch-up (empty or far-behind member);
  assert `committed_divergence=0` and `committed_job_loss=0` post-round (Requirement 4.1, 4.2) —
  this is the direct acceptance test for the residual soak failure this spec exists to close.
- **Integration — rolling upgrade simulation**: a pre-feature-format checkpoint file present on
  disk, loaded by the new binary, does not crash and correctly falls back to full catch-up
  rather than reporting false-positive completeness (Requirement 5.1, 5.2).