559 lines
35 KiB
Markdown
559 lines
35 KiB
Markdown
# Design Document: ha-entity-snapshots
|
|
|
|
## Overview
|
|
|
|
**Purpose**: This feature makes the XCE controller's periodic persistence checkpoint an
|
|
actually-restorable copy of committed job and queue state, closing the permanent data gap a
|
|
rejoining or far-behind cluster member hits today (task 5.10a / raft-ha-core-safety; bug-795/
|
|
828/851 family). Today `write_snapshot` records only a position marker; `load_snapshot` restores
|
|
nothing. This design adds the missing entity payload and restore step, and closes the one real
|
|
retention hazard it introduces (a snapshot file being rotated out from under an in-flight
|
|
transfer).
|
|
|
|
**Users**: The XCE HA controller cluster itself is the direct consumer — every member that
|
|
rejoins empty or falls behind the retained WAL history. The observable beneficiary is the
|
|
cluster operator, who currently sees a silently incomplete member after such a rejoin.
|
|
|
|
**Impact**: Modifies the existing checkpoint write/read/rotate logic in `persist.c` and adds one
|
|
small tracking array in `ha.c`. No wire protocol change, no new external dependency, no new
|
|
capnp schema struct — the existing `Job` schema and its generated codec already carry everything
|
|
needed for jobs. Queue *definitions* (name, attrs, priority, thresholds, ACLs) are config-
|
|
provisioned and already replicated independently via `push_config_to_standbys`
|
|
(`config_repl.c`) — out of this spec's boundary. Only a queue's runtime admin status
|
|
(open/closed/active/inactive), the one piece `WAL_QUEUE_CTRL` tracks and config replication does
|
|
not carry, is this spec's concern — restored via the existing `restore_queue_state(name, status)`,
|
|
no capnp involved on the queue side at all. (Note: the on-disk wire format was later migrated to a
|
|
capnp `QueueAdminState` frame — superseded by config-upgrade-snapshot, see that spec's design.md;
|
|
`restore_queue_state`'s own `(name, status)` call contract is unchanged.)
|
|
|
|
### Goals
|
|
- A checkpoint written at LSN X contains every live/retained job and queue record as of X.
|
|
- Loading that checkpoint reconstructs those records before the member's position advances.
|
|
- History is never discarded before it is captured in a checkpoint whose entities are durable.
|
|
- A member that rejoins via checkpoint ends up bit-for-bit consistent with the rest of the
|
|
cluster — no permanent gap.
|
|
- A checkpoint produced by a prior version of the binary never crashes or corrupts a member
|
|
that reads it.
|
|
|
|
### Non-Goals
|
|
- Lock-free, copy-on-write entity enumeration (zarray COW exists but is not wired into `kvmgr`;
|
|
reviving that wiring is a separate, larger effort — see `research.md` Option B).
|
|
- Any change to the wire transport/chunking protocol that ships a snapshot to a peer.
|
|
- Any change to the catch-up decision logic (`ha_evaluate_repl_action`) that chooses between a
|
|
snapshot transfer and incremental WAL streaming.
|
|
- Worker/compute-node state, cluster configuration document snapshotting, or election/term-safety
|
|
behavior.
|
|
- Queue *definition* fields (name, attrs, priority, thresholds, windows, ACLs) — these are
|
|
config-provisioned and already replicated independently (`push_config_to_standbys`); this spec
|
|
snapshots only the queue's runtime admin status.
|
|
|
|
## Boundary Commitments
|
|
|
|
### This Spec Owns
|
|
- The on-disk content format of a checkpoint (entity payload framing) and the write/read logic
|
|
that produces and consumes it (`write_snapshot`, `load_snapshot` internals).
|
|
- Point-in-time collection and encoding of live job and queue entities at checkpoint time.
|
|
- The entity-restore step on load and gating the member's committed-position advance on that
|
|
step's success.
|
|
- Distinguishing a legacy (pre-feature) position-only checkpoint from an entity-bearing one.
|
|
- Protecting an in-flight-transferring checkpoint file from premature rotation deletion.
|
|
|
|
### Out of Boundary
|
|
- The wire opcode and chunk-reassembly protocol that delivers a checkpoint to a peer
|
|
(`OPCode_snapshotTransfer`, `ha_handle_snapshot_chunk`'s framing) — reused unchanged.
|
|
- The decision of *when* a member needs a checkpoint vs. incremental streaming
|
|
(`ha_evaluate_repl_action`) — unchanged.
|
|
- The WAL-segment trim floor's standby-acked-LSN computation (`repl_trim_lsn`) — unchanged; see
|
|
Architecture below for why no new floor input is needed there.
|
|
- Worker/join state, config-document snapshotting (`config-upgrade-snapshot`), election and
|
|
term-safety behavior (`raft-ha-core-safety`, `ha-term-safety`).
|
|
|
|
### Allowed Dependencies
|
|
- Existing capnp codegen for `Job` (`encode_Job`/`write_Job`/`decode_Job`, already generated from
|
|
`src/schema/controller.capnp`). Queue entries need no capnp — `restore_queue_state`'s contract
|
|
is a plain `(name, status)` pair (see Architecture below). (Note: config-upgrade-snapshot later
|
|
added a capnp `QueueAdminState` frame as the on-disk encoding of that same pair — see Data
|
|
Models below.)
|
|
- `restore_job` (`job.c:274`) and `restore_queue_state` (`queue.c:208`) — existing, unmodified
|
|
entity-restore functions already used by WAL replay.
|
|
- The `collect_jobs`/`collect_queues` enumeration pattern (`dataex.c`) and the
|
|
`job_should_retain`/`worker_should_retain` retention predicates (`persist.c`).
|
|
- The existing tmp-file-then-atomic-rename pattern already used by `write_snapshot` and by
|
|
`ha_handle_snapshot_chunk`'s `snap.part` assembly.
|
|
- The existing per-peer `ha->snapshot_inflight[]` array shape (`ha.h`) as the pattern for the new
|
|
in-flight-LSN tracking.
|
|
|
|
### Revalidation Triggers
|
|
- Any change to the `Job` capnp schema shape — re-verify encode/decode symmetry and compatibility
|
|
with `restore_job`.
|
|
- Any change to `restore_queue_state`'s `(name, status)` contract, or any future need to snapshot
|
|
more than status for queues — re-verify the queue-side format and the config-vs-WAL boundary
|
|
this design draws.
|
|
- Any change to `collect_jobs`/`collect_queues` locking semantics, or to the per-entity
|
|
`RTTI_LOCK_READ`/`WRITE` discipline (e.g. if `kvmgr` COW is wired later) — re-verify the
|
|
point-in-time consistency argument in Architecture below.
|
|
- Any change to `persist_checkpoint`'s `write_snapshot` → `trim_wal_segments` sequencing
|
|
(currently sequential, trim skipped on write failure) — re-verify Requirement 3.
|
|
- Any change to `ha_handle_snapshot_chunk`'s ordering (currently: `applied_lsn` only advances
|
|
after `persist_load_snapshot` returns success) — re-verify Requirement 4.
|
|
|
|
## Architecture
|
|
|
|
### Existing Architecture Analysis
|
|
|
|
- `persist_checkpoint` (persist.c:2820) already sequences correctly for Requirement 3: it calls
|
|
`write_snapshot`, and only calls `trim_wal_segments` if `write_snapshot` returned 0
|
|
(`if (rc != 0) goto done;`, skipping both trim and `rotate_snapshots`). **This means once
|
|
`write_snapshot` only returns 0 when the entity payload is fully and durably written, the
|
|
existing trim gating already satisfies "retention never outruns restorability" — no new floor
|
|
variable is needed in `trim_wal_segments`/`repl_trim_lsn`.** This is a refinement over the
|
|
initial gap analysis (`research.md`), which assumed a new trim-floor input was needed.
|
|
- The one retention hazard `write_snapshot`'s success does *not* already guard is
|
|
`rotate_snapshots` (persist.c:2677): it keeps only `snap_keep_n` newest snapshot *files* by
|
|
filename sort and deletes the rest, with no awareness of an in-flight transfer. If a checkpoint
|
|
cadence faster than a slow transfer rotates out the file a standby is still reading from a
|
|
chunked wire transfer, that transfer breaks. This is the one genuinely new guard this design
|
|
adds.
|
|
- `ha_handle_snapshot_chunk` (ha.c:3635-3794) already has the right shape for Requirement 4: it
|
|
only advances `ha->applied_lsn` after `persist_load_snapshot` returns success. Once
|
|
`persist_load_snapshot`/`load_snapshot` correctly gate their own success on entity restoration
|
|
(Requirement 2), this call site needs no change.
|
|
- `collect_jobs`/`collect_queues` (dataex.c) already hold `RTTI_LOCK_READ` on `c->jobmgr`/
|
|
`c->qmgr` for the duration of their walk. **This table lock is a distinct lock from the
|
|
per-entity lock** — verified via `restore_queue_state` (queue.c:208), which takes
|
|
`lock(q, RTTI_LOCK_WRITE)` on the individual `queue_t*` after already releasing the table lock
|
|
from its own `key_find`, and via `job.c`'s state-transition sites (e.g. lines 1006, 1093, 1137),
|
|
which take `lock(j, RTTI_LOCK_WRITE)` on the individual `job_t*` (line numbers shift as job.c
|
|
grows; verify by grepping `lock(j, RTTI_LOCK_WRITE)` if re-checking this). The table lock only excludes
|
|
concurrent insert/delete of table entries; it does **not** exclude a concurrent per-entity
|
|
writer mutating an entity's fields mid-walk. Requirement 1.2's point-in-time consistency
|
|
therefore requires the entity-collection helper to additionally take each entity's own
|
|
`RTTI_LOCK_READ` (mirroring `restore_queue_state`'s own per-entity write-lock pattern) around
|
|
that entity's individual encode call, nested inside the table-level walk — this excludes a
|
|
concurrent per-entity writer for exactly the duration of that one entity's encode, without
|
|
needing the (currently unwired) zarray COW mechanism.
|
|
|
|
### Architecture Pattern & Boundary Map
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph Primary["Primary — checkpoint write path"]
|
|
PC[persist_checkpoint] --> WS[write_snapshot]
|
|
WS --> CJ["collect+encode jobs\n(table READ lock + per-job READ lock)"]
|
|
WS --> CQ["collect name+status pairs\n(table READ lock + per-queue READ lock,\nnow capnp QueueAdminState --\nconfig-upgrade-snapshot)"]
|
|
CJ --> TMP[tmp file + fdatasync + rename]
|
|
CQ --> TMP
|
|
TMP -->|rc==0| TRIM[trim_wal_segments]
|
|
TMP -->|rc==0| ROT["rotate_snapshots\n(skips in-flight LSNs)"]
|
|
TMP -->|rc!=0| SKIP["skip trim + rotate\n(existing goto done)"]
|
|
end
|
|
|
|
subgraph Send["Snapshot transfer (unchanged)"]
|
|
DISPATCH[ha_snapshot_send_dispatch] -->|sets snapshot_inflight_lsn for peer| WIRE[OPCode_snapshotTransfer chunks]
|
|
WIRE --> DONE[clears snapshot_inflight_lsn on completion]
|
|
end
|
|
|
|
ROT -.reads.-> DISPATCH
|
|
|
|
subgraph Standby["Standby — catch-up receive path"]
|
|
RECV[ha_handle_snapshot_chunk] --> RENAME[assemble + rename final file]
|
|
RENAME --> LS[persist_load_snapshot]
|
|
LS --> RJ["restore_job per entity"]
|
|
LS --> RQ["restore_queue_state per entity"]
|
|
RJ -->|all succeed| ADV["advance current_lsn / checkpoint_lsn"]
|
|
RQ -->|all succeed| ADV
|
|
RJ -->|any fails| FAIL["return error, position unchanged"]
|
|
ADV --> APPLIED["ha->applied_lsn advances\n(existing, unchanged)"]
|
|
end
|
|
```
|
|
|
|
**Architecture Integration**:
|
|
- Selected pattern: extend the existing checkpoint write/read pipeline in place (Option A from
|
|
`research.md`); no new subsystem, no new wire message.
|
|
- Domain boundaries: entity content and restoration stay inside `persist.c`
|
|
(write/load/rotate); the one cross-file touch is the new `snapshot_inflight_lsn[]` array in
|
|
`ha.h`/`ha.c`, read by `rotate_snapshots` and written by the existing send-dispatch path.
|
|
- Existing patterns preserved: tmp-file-then-rename atomicity, `RTTI_LOCK_READ`/`WRITE` table AND
|
|
per-entity locking (mirroring `restore_queue_state`'s own nested-lock pattern), the
|
|
`collect_jobs`/`collect_queues` iteration shape, the `capnp_encode_job` wrapper shape.
|
|
- New components rationale: a small entity-frame reader/writer pair (mirrors the existing WAL
|
|
entry length-prefixed framing style, so no new capnp schema struct or codegen is required —
|
|
jobs use the existing `Job` capnp codec, queues use a plain fixed `(name, status)` record since
|
|
that is the entirety of `restore_queue_state`'s contract); the in-flight-LSN guard in
|
|
`rotate_snapshots`. (Note: "no new capnp schema struct" for queues held at this spec's close but
|
|
was superseded shortly after by config-upgrade-snapshot, which added a `QueueAdminState` capnp
|
|
struct and migrated the queue frame to it — see that spec's design.md.)
|
|
- Steering compliance: no `fork()` used for the snapshot (per `product.md`'s "No fork() for
|
|
snapshots" constraint) — consistency comes from the table read-lock, not process duplication.
|
|
|
|
### Technology Stack
|
|
|
|
| Layer | Choice / Version | Role in Feature | Notes |
|
|
|-------|------------------|-----------------|-------|
|
|
| Serialization | Cap'n Proto (existing project-wide IDL) | Per-job `Job` encode/decode; queues use a plain fixed record (no capnp) | Codegen already exists; no schema change |
|
|
| Persistence | Flat file, tmp+rename (existing pattern) | Checkpoint file on disk | Same pattern as today's sentinel-only file |
|
|
| Concurrency | `RTTI_LOCK_READ`/`WRITE`, table AND per-entity (existing) | Point-in-time entity consistency during collection | zarray COW deferred (Non-Goal) |
|
|
| Transport | `OPCode_snapshotTransfer` (existing, unmodified) | Ships the (now larger) file to a standby | Out of boundary |
|
|
|
|
## File Structure Plan
|
|
|
|
### Modified Files
|
|
- `src/controller/persist.c` — `write_snapshot`: append entity payload after the existing
|
|
header — job frames (capnp `Job`) and queue frames (plain `(name, status)` records, no capnp).
|
|
(Note: queue frames were later migrated to capnp `QueueAdminState`, and host frames plus a
|
|
trailing config-bytes section were appended after them — superseded by config-upgrade-snapshot,
|
|
see Data Models below and that spec's design.md.)
|
|
`load_snapshot`: restore entities before advancing `current_lsn`/`checkpoint_lsn`; detect
|
|
legacy (position-only) files by remaining file size. `rotate_snapshots`: skip deleting any file
|
|
whose LSN is currently in-flight. New static helpers: a snapshot entity-frame writer/reader
|
|
pair, and an entity-collection helper built on `collect_jobs`/`collect_queues` that additionally
|
|
takes each entity's own per-entity `RTTI_LOCK_READ` around its individual encode/collect call
|
|
(see Architecture).
|
|
- `src/include/persist.h` — extend `persist_load_snapshot`'s signature with an out-parameter
|
|
reporting whether entities were restored (see Components below); no other public surface
|
|
changes.
|
|
- `src/controller/ha.h` — add `_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]` next
|
|
to the existing `snapshot_inflight[MAX_HA_CANDIDATES]` array.
|
|
- `src/controller/ha.c` — `ha_snapshot_send_dispatch`/`ha_snapshot_do_send`: record the LSN being
|
|
sent into `snapshot_inflight_lsn[peer_idx]` alongside the existing flag, clear it on completion
|
|
(same lifecycle as the existing flag). `ha_handle_snapshot_chunk`: read the new
|
|
`persist_load_snapshot` out-parameter; log (not fail) when entities were not restored, since
|
|
the caller-side catch-up-completeness signal (Requirement 4.3) is carried by existing
|
|
status/comparison tooling, not a new error path here.
|
|
|
|
### New Test Files
|
|
- `src/controller/testsuite/test_persist_snapshot_entities.c` — unit coverage for
|
|
`write_snapshot`/`load_snapshot` entity round-trip, partial-failure discard, legacy-file
|
|
handling, and the `rotate_snapshots` in-flight guard (see Testing Strategy).
|
|
|
|
> No new production files. This stays entirely inside the existing checkpoint machinery's file
|
|
> set plus one small, symmetric addition in `ha.h`/`ha.c` for the rotation guard.
|
|
|
|
## System Flows
|
|
|
|
See the Architecture Pattern & Boundary Map diagram above for the write/send/receive flow. One
|
|
additional sequence clarifies the failure-ordering contract that Requirement 2.3 depends on:
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Standby
|
|
participant persist_load_snapshot
|
|
participant restore_job
|
|
participant restore_queue_state
|
|
|
|
Standby->>persist_load_snapshot: load(path, lsn)
|
|
persist_load_snapshot->>persist_load_snapshot: read header, detect format
|
|
alt legacy (position-only) file
|
|
persist_load_snapshot-->>Standby: rc=0, entities_restored=false
|
|
else entity-bearing file
|
|
loop each job frame
|
|
persist_load_snapshot->>restore_job: decode + restore
|
|
restore_job-->>persist_load_snapshot: rc
|
|
alt rc != 0
|
|
persist_load_snapshot-->>Standby: rc=error, position UNCHANGED
|
|
end
|
|
end
|
|
loop each queue frame
|
|
persist_load_snapshot->>restore_queue_state: restore(name, status)
|
|
restore_queue_state-->>persist_load_snapshot: rc
|
|
alt rc != 0
|
|
persist_load_snapshot-->>Standby: rc=error, position UNCHANGED
|
|
end
|
|
end
|
|
persist_load_snapshot->>persist_load_snapshot: advance current_lsn/checkpoint_lsn
|
|
persist_load_snapshot-->>Standby: rc=0, entities_restored=true
|
|
end
|
|
```
|
|
|
|
## Requirements Traceability
|
|
|
|
| Requirement | Summary | Components | Interfaces | Flows |
|
|
|---|---|---|---|---|
|
|
| 1.1 | Checkpoint includes every live/retained job+queue record | `write_snapshot`, entity-collection helper | `collect_jobs`/`collect_queues` + `job_should_retain`; queue side is `(name, status)` only (definition is config-owned) | Write path (diagram 1) |
|
|
| 1.2 | Internally consistent point-in-time view | Entity-collection helper | Table `RTTI_LOCK_READ` + per-entity `RTTI_LOCK_READ` nested around each encode | Architecture: Existing Architecture Analysis |
|
|
| 1.3 | Discard incomplete checkpoint | `write_snapshot` | Existing tmp+rename pattern | Write path (diagram 1) |
|
|
| 1.4 | Record committed position | `write_snapshot` (existing header, unchanged) | — | — |
|
|
| 2.1/2.2 | Restore entities before advancing position | `load_snapshot` | `restore_job`, `restore_queue_state` | Sequence diagram |
|
|
| 2.3 | Failed reconstruction must not advance position | `load_snapshot` | rc propagation from `restore_job`/`restore_queue_state` | Sequence diagram |
|
|
| 2.4 | Legacy checkpoint treated as insufficient | `load_snapshot`, `persist_load_snapshot` | New `entities_restored` out-parameter | Sequence diagram |
|
|
| 3.1/3.2/3.3 | Retention never outruns restorability | `persist_checkpoint` (existing sequencing, unchanged) | `write_snapshot` rc gates `trim_wal_segments`/`rotate_snapshots` | Write path (diagram 1) |
|
|
| 4.1/4.2 | End-to-end catch-up correctness | `ha_handle_snapshot_chunk` (existing, unchanged ordering) | `persist_load_snapshot` | Receive path (diagram 1) |
|
|
| 4.3 | Catch-up completeness observable | Existing `walcompare`/status tooling (out of boundary; confirmed compatible) | — | — |
|
|
| 5.1/5.2 | Safe rollout against old checkpoints | `load_snapshot` format discriminator | File-size-based legacy detection | Sequence diagram |
|
|
|
|
## Components and Interfaces
|
|
|
|
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
|
|---|---|---|---|---|---|
|
|
| `write_snapshot` (extended) | persist.c | Serialize header + live entities to a durable checkpoint file | 1.1-1.4, 3.1-3.3 | `collect_jobs`/`collect_queues` (P0), `capnp_encode_job` (P0) | State |
|
|
| `load_snapshot` (extended) | persist.c | Deserialize and restore entities; gate position advance on success | 2.1-2.4, 5.1-5.2 | `restore_job`, `restore_queue_state` (P0) | State |
|
|
| `rotate_snapshots` (extended) | persist.c | Delete old checkpoint files, skipping any in-flight LSN | 3.1-3.3 | `ha->snapshot_inflight_lsn[]` (P0) | State |
|
|
| `snapshot_inflight_lsn[]` (new) | ha.h / ha.c | Track which checkpoint LSN each in-flight peer transfer references | 3.1-3.3 | Existing `snapshot_inflight[]` lifecycle (P0) | State |
|
|
|
|
### Persistence Layer
|
|
|
|
#### `write_snapshot` (extended)
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Produce a durable, entity-bearing checkpoint file at `checkpoint_lsn` |
|
|
| Requirements | 1.1, 1.2, 1.3, 1.4, 3.1, 3.2, 3.3 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Writes the existing legacy header unchanged (magic, `checkpoint_lsn`, optional config-ref).
|
|
- Appends, in order: a format-version marker, a job count and per-job length-prefixed capnp `Job`
|
|
frames, a queue count and per-queue fixed `(name, status)` frames (no capnp — matches
|
|
`restore_queue_state`'s exact contract; queue definition is config-owned, out of boundary).
|
|
(Note: superseded by config-upgrade-snapshot — queue frames are now capnp `QueueAdminState`,
|
|
and host frames plus a config-bytes section were appended after the queue section; see Data
|
|
Models below.)
|
|
- Collects under `RTTI_LOCK_READ` on `c->jobmgr` (released before the queue pass begins); collects
|
|
under `RTTI_LOCK_READ` on `c->qmgr` for queues. The two table locks are never held
|
|
simultaneously (no new lock-ordering surface). **Within each table's held READ lock, also takes
|
|
each individual entity's own `RTTI_LOCK_READ`** (`lock(j, RTTI_LOCK_READ)` /
|
|
`lock(q, RTTI_LOCK_READ)`) around that one entity's encode/read call — this is required for
|
|
Requirement 1.2: the table lock alone does not exclude a concurrent per-entity writer (see
|
|
Architecture).
|
|
- Filters via the existing `job_should_retain` predicate so the checkpoint's notion of "live"
|
|
matches the existing GC retention policy (Requirement 1.1). Every queue present in `c->qmgr` is
|
|
included (a queue's mere table presence already means config says it exists; no retain filter
|
|
needed for a plain status word).
|
|
- Writes to the existing `.tmp` path, `fdatasync`s, then atomically renames to the final path —
|
|
identical pattern to today, just with more bytes. Any failure at any step (encode, write,
|
|
sync) leaves no file at the final path (Requirement 1.3); the tmp file is unlinked.
|
|
|
|
**Contracts**: State [x]
|
|
|
|
##### State Management
|
|
- State model: file on disk, atomically visible only at the final path once fully written.
|
|
- Persistence & consistency: `fdatasync` before rename, matching the existing sentinel-only
|
|
behavior — durability semantics are unchanged, only the payload grows.
|
|
- Concurrency strategy: two sequential table-read-lock sections (jobs, then queues), each with a
|
|
nested per-entity read lock around the individual encode/read call; no lock is held during the
|
|
disk write itself.
|
|
|
|
**Implementation Notes**
|
|
- Integration: called from `persist_checkpoint`, unchanged call site.
|
|
- Validation: unit test asserts a checkpoint containing N jobs + M queues round-trips through
|
|
`load_snapshot` with all N+M records field-identical.
|
|
- Risks: enumeration+encode duration under `RTTI_LOCK_READ` blocks concurrent job/queue mutation
|
|
for that duration. Mitigated by checkpoints already running off the hot path on a configurable
|
|
`snap_interval_s` cadence; measure actual duration at a realistic job-table size during
|
|
implementation (see `research.md` Research Needed).
|
|
|
|
#### `load_snapshot` (extended)
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Reconstruct entities from a checkpoint file and gate position advance on success |
|
|
| Requirements | 2.1, 2.2, 2.3, 2.4, 5.1, 5.2 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Reads the existing legacy header exactly as today (unchanged).
|
|
- Determines format: if no bytes remain in the file after the legacy header, this is a
|
|
pre-feature checkpoint — do not advance `current_lsn`/`checkpoint_lsn` as "fully caught up";
|
|
return success with `entities_restored=false` so the caller falls back to normal catch-up
|
|
(Requirement 2.4). This never crashes or misinterprets an old file (Requirement 5.1, 5.2) — a
|
|
short file is a valid, expected legacy shape, not a corruption.
|
|
- If entity frames are present, decode each job frame (capnp) and call `restore_job`; read each
|
|
queue frame's plain `(name, status)` pair and call `restore_queue_state(name, status)` directly
|
|
— in file order. On any decode or restore failure, abort immediately: return an error, leave
|
|
`current_lsn`/`checkpoint_lsn` unchanged (Requirement 2.3).
|
|
- Only after every entity restores successfully does the function advance
|
|
`current_lsn`/`checkpoint_lsn` to the checkpoint's position (Requirement 2.1, 2.2).
|
|
|
|
**Contracts**: State [x]
|
|
|
|
##### State Management
|
|
- State model: in-memory job/queue tables (`c->jobmgr`/`c->qmgr`), mutated in place by the
|
|
existing `restore_job`/`restore_queue_state` (unmodified — same functions WAL replay already
|
|
calls, so no new restore code path exists to diverge from replay's behavior).
|
|
- Persistence & consistency: `current_lsn`/`checkpoint_lsn` advance is the single commit point
|
|
for "this checkpoint is now fully applied" — atomic in the sense that no partial-entity state
|
|
is ever exposed as "caught up."
|
|
- Concurrency strategy: runs on the receiving member during catch-up, before it accepts write
|
|
traffic for those entities — no concurrent mutation expected at this point in the lifecycle
|
|
(same assumption WAL replay already makes at startup).
|
|
|
|
**Public signature change** (`src/include/persist.h`):
|
|
```c
|
|
int persist_load_snapshot(persist_t* p, const char* path, uint64_t lsn,
|
|
bool* out_entities_restored);
|
|
```
|
|
- Preconditions: `path` names a file written by `write_snapshot` (legacy or current format).
|
|
- Postconditions: on rc==0, `*out_entities_restored` reports whether this call actually restored
|
|
entity state (`true`) or only observed a legacy position marker (`false`); `current_lsn`/
|
|
`checkpoint_lsn` reflect the checkpoint's position only when entities were fully restored or
|
|
the file was legacy-format (position-only, as today). On rc!=0, position is unchanged.
|
|
- Invariants: `*out_entities_restored` is only set when rc==0.
|
|
|
|
**Implementation Notes**
|
|
- Integration: `ha_handle_snapshot_chunk` passes the new out-parameter through; on
|
|
`entities_restored==false` it logs (does not fail) since a legacy file is an expected rollout
|
|
state, not an error — the member simply continues through the existing incremental catch-up
|
|
path for the remaining history.
|
|
- Validation: unit test with a deliberately truncated/corrupted entity frame asserts
|
|
`current_lsn` unchanged and rc!=0; unit test with a legacy-sized file asserts rc==0 and
|
|
`entities_restored==false`.
|
|
|
|
#### `rotate_snapshots` (extended)
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Delete old checkpoint files without breaking an in-flight transfer |
|
|
| Requirements | 3.1, 3.2, 3.3 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- Unchanged selection logic (keep `snap_keep_n` newest by filename/LSN).
|
|
- Before deleting a candidate file, checks whether its LSN matches any entry in
|
|
`ha->snapshot_inflight_lsn[]` (via `p->cluster->ha`, already reachable from `persist_t`); if so,
|
|
skip deletion for this rotation pass and retry on the next checkpoint cycle.
|
|
|
|
**Contracts**: State [x]
|
|
|
|
##### State Management
|
|
- State model: reads `ha_state_t`'s new array; does not itself own it (owned by the send-dispatch
|
|
path, see below).
|
|
|
|
**Implementation Notes**
|
|
- Integration: minimal — one lookup added to the existing deletion loop.
|
|
- Validation: unit test simulates an in-flight LSN matching an otherwise-rotatable file and
|
|
asserts it survives rotation.
|
|
|
|
### HA Replication Layer
|
|
|
|
#### `snapshot_inflight_lsn[]` (new array)
|
|
|
|
| Field | Detail |
|
|
|-------|--------|
|
|
| Intent | Record which checkpoint LSN each in-flight peer snapshot transfer references |
|
|
| Requirements | 3.1, 3.2, 3.3 |
|
|
|
|
**Responsibilities & Constraints**
|
|
- `_Atomic uint64_t snapshot_inflight_lsn[MAX_HA_CANDIDATES]` in `ha_state_t` (ha.h), declared
|
|
next to the existing `_Atomic int snapshot_inflight[MAX_HA_CANDIDATES]`.
|
|
- Set by `ha_snapshot_send_dispatch`/`ha_snapshot_do_send` at the same point the existing
|
|
`snapshot_inflight[peer_idx]` flag is set to 1 (before dispatch); cleared at the same points
|
|
the existing flag is cleared to 0 (on completion or CAS-release failure) — identical lifecycle,
|
|
one more field.
|
|
|
|
**Contracts**: State [x]
|
|
|
|
##### State Management
|
|
- State model: plain atomic array, same shape and lifecycle as the existing flag array it sits
|
|
beside.
|
|
- Concurrency strategy: no new locking — atomics only, matching the existing flag's discipline.
|
|
|
|
**Implementation Notes**
|
|
- Integration: two write sites (set/clear), both already-existing call sites in `ha.c`.
|
|
- Validation: covered by the `rotate_snapshots` guard test above (integration-level: dispatch a
|
|
send, assert its LSN is protected, complete it, assert protection is released).
|
|
|
|
## Data Models
|
|
|
|
### Checkpoint File Layout (Physical)
|
|
|
|
```
|
|
[existing legacy header — unchanged, 16 or 57 bytes depending on config-ref presence]
|
|
[uint32_t format_version] -- absent (EOF here) => legacy, position-only file
|
|
[uint32_t job_count]
|
|
[job_count * { uint32_t payload_len; uint8_t capnp_Job_bytes[payload_len] }]
|
|
[uint32_t queue_count]
|
|
[queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }] -- (Note: this raw
|
|
-- shape is the
|
|
-- layout AS OF
|
|
-- this spec's
|
|
-- close; see below)
|
|
```
|
|
|
|
> **Note (superseded by config-upgrade-snapshot, see that spec's design.md):** the physical layout
|
|
> above is this spec's as-shipped shape only. config-upgrade-snapshot subsequently (1) migrated the
|
|
> queue frame from the raw `{name_len, name, status}` shape above to a length-prefixed capnp
|
|
> `QueueAdminState` frame (same shape as the job-frame loop, content unchanged: name + status),
|
|
> (2) appended a new host-frames section (`[uint32_t host_count][host_count * {payload_len,
|
|
> capnp HostAdminState bytes}]`) directly after the queue section, and (3) appended a trailing
|
|
> flat length-prefixed config-bytes blob after that (`[uint32_t cfg_len][cfg_len bytes]`, written
|
|
> on every checkpoint). `format_version` was bumped to `2` for this change. Current source:
|
|
> `write_snapshot`/`load_snapshot_entities` in `src/controller/persist.c`.
|
|
|
|
- Job frames are length-prefixed capnp `Job` messages, matching the framing style already used
|
|
for WAL entries (`wal_entry_hdr_t` + payload) — no new schema/codegen for a one-off container.
|
|
- Queue frames carry only what `restore_queue_state` can consume: a name and a status word — no
|
|
capnp at all. A queue's *definition* (attrs, priority, thresholds, ACLs) is config-provisioned
|
|
and already replicated via `push_config_to_standbys`; encoding it here would duplicate data this
|
|
spec does not own and has no restore path for. (Note: the wire encoding of this same
|
|
name+status pair was later moved to capnp — see the note above.)
|
|
- A file ending exactly at the legacy header (no bytes remain) is unambiguously the old format —
|
|
the discriminator is "does more data exist," not a flag bit, so old files need zero
|
|
reinterpretation.
|
|
- `format_version` is present for forward evolution (a future layout bumps this value and
|
|
`load_snapshot` dispatches on it) without breaking this design's readers. (Note: superseded by
|
|
config-upgrade-snapshot's actual choice, see that spec's design.md — when it bumped
|
|
`format_version` to `2` it made `load_snapshot` hard-reject any non-matching version with
|
|
`-EINVAL` rather than dispatch, on the reasoning that a version mismatch must never be silently
|
|
masked as an expected legacy shape or real corruption could go undetected. The legacy
|
|
(no-`format_version`-present) detection this bullet's neighbor describes is unaffected.)
|
|
|
|
### Consistency & Integrity
|
|
- Per-entity encode happens while both the enumerating table's `RTTI_LOCK_READ` AND that entity's
|
|
own nested `RTTI_LOCK_READ` are held (see Architecture) — the table lock alone is insufficient,
|
|
since entity mutation takes a separate per-entity lock; the nested per-entity lock is what
|
|
excludes a concurrent writer for that entity's encode, giving each entity a single consistent
|
|
snapshot of its own fields with no entity encoded mid-mutation.
|
|
- Cross-entity consistency (jobs relative to queues) is not required by Requirement 1.2, which
|
|
is scoped to "no record split between its own pre/post-checkpoint value" — jobs and queues are
|
|
independent aggregates for this feature's purposes, consistent with `collect_jobs`/
|
|
`collect_queues` already treating them as separately-locked tables today.
|
|
|
|
## Error Handling
|
|
|
|
### Error Strategy
|
|
- **Write-side failure** (encode error, disk I/O error) at any point during `write_snapshot`:
|
|
unlink the tmp file, return a non-zero rc, which the existing `persist_checkpoint` caller
|
|
already treats as "skip trim and rotate this cycle" — no operational-condition self-termination
|
|
(per `src/controller/CLAUDE.md`'s controller-never-self-terminates invariant); the next
|
|
checkpoint cycle simply retries.
|
|
- **Read-side failure** (corrupt/truncated entity frame, decode error, restore error) during
|
|
`load_snapshot`: return a non-zero rc with position unchanged; the caller
|
|
(`ha_handle_snapshot_chunk`) logs the failure and does not advance `applied_lsn` — the member
|
|
stays fenced/behind and the existing catch-up retry mechanics (unchanged by this spec) bring it
|
|
current on a subsequent attempt. Consistent with the project's fail-loud-and-stay-alive
|
|
philosophy — never a self-terminate on this operational condition.
|
|
- **Legacy file encountered**: not an error. `entities_restored=false` with rc==0 is the expected,
|
|
handled outcome for a file predating this feature.
|
|
|
|
### Monitoring
|
|
- Existing `WARN`/`ERROR` log lines at each new failure point (matching the codebase's existing
|
|
logging conventions in `persist.c`/`ha.c`); no new metric is introduced by this design, but
|
|
Requirement 4.3's "observable to an operator" is satisfied by the existing `walcompare`/
|
|
`waldump` tooling already used this session to diagnose divergence — confirm during
|
|
implementation that it reports on entity-snapshot-derived state, not only WAL-visible LSNs.
|
|
|
|
## Testing Strategy
|
|
|
|
- **Unit — `write_snapshot`/`load_snapshot` round-trip**: a checkpoint with N jobs and M queues,
|
|
loaded back, yields N job records field-identical to the originals and M queue statuses
|
|
correctly reapplied via `restore_queue_state` (Requirement 1.1, 1.2, 2.1, 2.2).
|
|
- **Unit — queue status frame round-trip**: a queue's `(name, status)` pair written and read back
|
|
matches exactly, and `restore_queue_state` is invoked with those exact values (Requirement 1.1).
|
|
- **Unit — concurrent-mutation-during-snapshot consistency**: a job actively transitioning state
|
|
(e.g. via `mark_job_start`) on a concurrent thread while `write_snapshot`'s enumeration walks it
|
|
must be captured as either the full pre- or full post-transition value, never a torn mix of
|
|
fields — proves the nested per-entity lock, not just the table lock, is doing the work
|
|
(Requirement 1.2).
|
|
- **Unit — partial write discarded**: a simulated encode/write failure partway through leaves no
|
|
file at the final path (Requirement 1.3).
|
|
- **Unit — partial restore does not advance position**: a deliberately corrupted entity frame
|
|
causes `load_snapshot` to return an error with `current_lsn`/`checkpoint_lsn` unchanged
|
|
(Requirement 2.3).
|
|
- **Unit — legacy file handling**: a file sized to end exactly at the legacy header yields rc==0,
|
|
`entities_restored=false`, no crash (Requirement 2.4, 5.1, 5.2).
|
|
- **Unit — rotation guard**: a file matching an in-flight LSN survives `rotate_snapshots` even
|
|
when it would otherwise fall outside `snap_keep_n` (Requirement 3.1, 3.2, 3.3).
|
|
- **Integration — chaotic catch-up soak**: extend the existing `/xce-test-ha --chaotic` soak
|
|
scenario so at least one round forces a `SEND_SNAPSHOT` catch-up (empty or far-behind member);
|
|
assert `committed_divergence=0` and `committed_job_loss=0` post-round (Requirement 4.1, 4.2) —
|
|
this is the direct acceptance test for the residual soak failure this spec exists to close.
|
|
- **Integration — rolling upgrade simulation**: a pre-feature-format checkpoint file present on
|
|
disk, loaded by the new binary, does not crash and correctly falls back to full catch-up
|
|
rather than reporting false-positive completeness (Requirement 5.1, 5.2).
|