myxce/.kiro/specs/ha-entity-snapshots/requirements.md

Requirements Document

Project Description (Input)

ha-entity-snapshots: Job/queue entity-bearing WAL snapshots so a rejoining or empty-restart node's catch-up gap (task 5.10a / raft-ha-core-safety, bug-795/828/851 family) closes without permanent committed-prefix loss. Today write_snapshot (persist.c ~1051) writes a 16-byte sentinel with ZERO job/queue entities and load_snapshot only advances the LSN watermark past entries it never received (restores nothing); trim_wal_segments deletes the WAL prefix below a checkpoint. A member that rejoins empty or behind a trimmed prefix is streamed only [watermark..committed] with no snapshot backfill for [1..watermark-1] — a permanent gap (observed live: 10-round chaotic soak shows committed_divergence>0 every round from this catch-up lag). Scope: write_snapshot Cap'n-Proto-serializes all live job/queue entities at checkpoint_lsn; load_snapshot deserializes + restores them BEFORE advancing current_lsn; trim_wal_segments must not trim below the LSN covered by a durable entity-bearing snapshot; install-snapshot-before-stream on the catch-up receive side. Full investigation at .kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md (VERDICT section) and .kiro/specs/raft-ha-core-safety/tasks.md task 5.10(a).

Introduction

XCE cluster members that rejoin the cluster empty, or that fall behind the oldest history the cluster still retains directly, currently receive no way to recover the committed job and queue history that existed before their position. The periodic checkpoint the cluster produces carries only a position marker, not the actual job/queue data, and older history is deleted once a checkpoint is taken. As a result, a member that catches up this way ends up reporting itself as caught up while permanently missing part of the committed record — a gap that persists indefinitely and that surfaces as data divergence across the cluster. This feature makes the checkpoint a complete, restorable copy of committed job and queue state, and ensures history is never discarded before that copy exists, so that every member — however it rejoins — ends up holding the same complete committed state as the rest of the cluster.

Boundary Context

• In scope:
  • The content of a checkpoint: it must represent the actual job and queue records committed as of that point, not merely a position.
  • Restoring a checkpoint: reconstructing the actual job and queue state it contains, not merely advancing a position.
  • Coordinating history retention so that log data is never discarded before its content is captured in a checkpoint that has been confirmed complete and durable.
  • The end-to-end outcome for a member that rejoins empty or significantly behind: it must reach the same complete committed job/queue state as an already-consistent member.
  • Safe behavior when a member encounters a checkpoint produced before this capability existed (rolling upgrade).
• Out of scope:
  • Worker/compute-node registration and state — workers re-establish their state via the existing join protocol, independent of this feature.
  • The mechanics of transferring a checkpoint to a peer over the network — that transport already exists and is unchanged by this feature.
  • Cluster configuration document snapshotting/versioning — owned by the config-upgrade-snapshot feature.
  • Election, vote-freshness, and term-safety behavior — owned by raft-ha-core-safety / ha-term-safety.
• Adjacent expectations: This feature assumes the existing catch-up decision logic (choosing between an incremental replay and a full checkpoint transfer for a member whose position is below the cluster's retained history) continues to trigger correctly and is unchanged by this feature; what changes is that the checkpoint it delivers is now complete, and the member's outcome after catch-up is now correct.

Requirements

Requirement 1: Complete entity-bearing checkpoint content

Objective: As a cluster operator, I want a checkpoint to contain the full set of committed job and queue records as of that point, so that a member restoring it receives actual data rather than only a position marker.

Acceptance Criteria

1. When the system produces a checkpoint at a committed position, the Persistence Service shall include in the checkpoint every job and queue record that is live or retained under the existing retention policy as of that position.
2. The Persistence Service shall produce a checkpoint that reflects a single, internally consistent point-in-time view of job and queue state, with no record split between its pre-checkpoint and post-checkpoint value.
3. If a checkpoint cannot be produced completely (for example, an I/O failure partway through), then the Persistence Service shall discard the incomplete checkpoint and shall not present it as a valid restore point.
4. The Persistence Service shall record, as part of the checkpoint, the committed position the contained entity state corresponds to.

Requirement 2: Full restoration from a checkpoint

Objective: As a rejoining or newly provisioned cluster member, I want loading a checkpoint to reconstruct the actual job and queue state it contains, so that I hold complete data rather than an empty state with an advanced position.

Acceptance Criteria

1. When a member loads a checkpoint, the Persistence Service shall reconstruct every job and queue record contained in the checkpoint into the member's live state before advancing the member's committed position.
2. The Persistence Service shall consider the member's committed position advanced to the checkpoint's position only after all contained records have been reconstructed.
3. If reconstruction fails partway through, then the Persistence Service shall not advance the member's committed position and shall report the failure rather than silently proceeding with partial state.
4. Where a loaded checkpoint carries no entity state (a checkpoint produced before this capability existed), the Persistence Service shall treat that checkpoint as insufficient on its own and shall require the member to obtain the missing history through the normal catch-up path rather than treating the checkpoint as complete.

Requirement 3: Retention never outruns restorability

Objective: As a cluster operator, I want history retention (discarding older log data) to never outrun the point at which that history is fully captured in a restorable checkpoint, so that a rejoining member is never left with a permanent, unrecoverable gap.

Acceptance Criteria

1. While a portion of history has not yet been captured in a checkpoint whose entity content is confirmed durable, the Persistence Service shall retain the corresponding log data.
2. The Persistence Service shall discard log data only when the history it covers is fully represented in a durable, entity-bearing checkpoint at or beyond that point.
3. If retention is requested for a position not yet covered by a durable, entity-bearing checkpoint, then the Persistence Service shall reject or defer that retention rather than discarding the data.

Requirement 4: End-to-end catch-up correctness for a behind or empty member

Objective: As a cluster operator, I want a member that rejoins empty or significantly behind to end up holding the same committed job and queue state as the rest of the cluster, so that a failover onto that member never serves incomplete or divergent data.

Acceptance Criteria

1. When a member's position is behind the oldest history the cluster still retains directly, the system shall bring that member to full committed-consistent job and queue state through a checkpoint transfer followed by any remaining incremental history.
2. After a member completes catch-up, whether through a checkpoint, incremental replay, or both, the system shall hold the same committed job and queue records as an already-consistent member, with no permanent gap.
3. The system shall make the completeness, or incompleteness, of a member's catch-up observable to an operator through existing status and comparison tooling, rather than reporting a member as fully caught up when it is not.

Requirement 5: Safe rollout against existing deployments

Objective: As a cluster operator upgrading an existing deployment, I want the new checkpoint format to coexist safely with checkpoints produced by the prior version, so that upgrading never corrupts state or crashes a member.

Acceptance Criteria

1. When a member reads a checkpoint file, the Persistence Service shall determine whether it is an entity-bearing checkpoint or a prior-format, position-only checkpoint before deciding how to use it.
2. The Persistence Service shall never crash, corrupt state, or fabricate job or queue records when reading a prior-format checkpoint.