# Requirements Document ## Introduction XCE controller operators running HA clusters face three replication gaps that prevent the system from reaching a safe majority-consistent state after election. The majority-ack fast path (`ha_repl_ack_received`) exists in code but is unreachable because no standby reports its catch-up status back to the primary. The committing timeout (60s degraded-mode fallback, task 5.3) is the safety net when majority convergence fails. Snapshot transfer for standbys that fall behind the WAL retention window is wired via the payload data plane (task 7.2). WAL catch-up streams entries one `xce_request` per entry, blocking a vtaskmgr thread for the entire duration. This feature implements a heartbeat-driven state synchronization protocol: the primary pushes config and WAL entries to standbys, each standby acknowledges with its current `(config_tag, applied_lsn)` in heartbeat replies, and the primary transitions to `CTRL_PRIMARY` when a majority of standbys report state converged with the primary. Separate catch-up-done and per-entry ack opcodes are eliminated in favor of piggybacked state reporting. ## Boundary Context - **In scope**: standby-side handling of `OPCode_walCatchupEntry` with local WAL application, heartbeat reply carrying standby state `(config_tag, applied_lsn)`, primary tracking per-standby state and determining majority convergence, snapshot creation and transfer when a standby predates the WAL retention window, batch transfer of multiple WAL entries per message, and gating the `CTRL_PRIMARY` transition on majority state convergence rather than a timeout - **Out of scope**: config push optimization (already implemented — config is pushed only when tags differ), WAL segment retention policy changes, dynamic cluster membership, mTLS between controllers, the election protocol itself, new wire protocol opcodes (the feature reuses existing heartbeat and ctrlhello messages extended with state fields) - **Adjacent expectations**: the primary already sends heartbeats to standbys via `ha_start_primary_heartbeat` during `CTRL_PRIMARY_COMMITTING`; `CtrlHelloReply` already carries `committed_lsn` and `config_tag` fields in the capnp schema; `apply_wal_entry` already exists and can replay individual WAL entries; `persist_checkpoint` already exists and can create a consistent snapshot ## Requirements ### Requirement 1: Heartbeat Reply Carries Standby State **Objective:** As a cluster operator, I want each standby to report its current replication state in every heartbeat reply, so that the primary can determine when the cluster has reached majority consistency without a separate catch-up-complete signal. #### Acceptance Criteria 1. When a standby receives a heartbeat message from the primary, the system shall reply with a heartbeat acknowledgement containing the standby's current config version tag and applied WAL log sequence number. 2. When the primary receives a heartbeat reply from a standby, the system shall record that standby's `(config_tag, applied_lsn)` and evaluate whether the standby has converged with the primary's committed state. 3. A standby is considered converged when its reported config tag is greater than or equal to the primary's committed config tag and its reported applied log sequence number is greater than or equal to the primary's committed log sequence number. 4. If a standby has not yet received the current configuration, the system shall report its config tag as zero in heartbeat replies until the config push completes. --- ### Requirement 2: Primary Tracks Per-Standby Convergence State **Objective:** As a cluster operator, I want the primary to continuously track each standby's reported state, so that it can determine when a majority of standbys have converged and the cluster is safe to enter `CTRL_PRIMARY`. #### Acceptance Criteria 1. When the primary enters `CTRL_PRIMARY_COMMITTING` state, the system shall initialize per-standby tracking with each standby marked as not-converged. 2. When a heartbeat reply arrives from a standby and the standby's reported state satisfies the convergence criteria, the system shall mark that standby as converged and re-evaluate the majority threshold. 3. When the number of converged standbys reaches the majority threshold `n_peers / 2` (C integer division — primary + converged standbys form a cluster majority; n=3→1 standby, n=5→2 standbys; raft rejects even n_peers > 1), the system shall transition the primary to `CTRL_PRIMARY` immediately. *(formula reconciled 2026-06-09)* 4. When the primary transitions to `CTRL_PRIMARY` via majority convergence, the system shall log at INFO level the number of converged standbys and the quorum size. 5. If a converged standby later reports a config tag or applied log sequence number that no longer satisfies the convergence criteria (e.g., after a restart), the system shall mark that standby as not-converged and re-evaluate the majority threshold. --- ### Requirement 3: Standby Applies WAL Catch-Up Entries to Local State **Objective:** As a cluster operator, I want a standby that receives WAL catch-up entries from the primary to apply them to its local durable state, so that the standby's state converges with the primary's and the standby is eligible to become primary on failover. #### Acceptance Criteria 1. When a standby controller receives an `OPCode_walCatchupEntry` message, the system shall decode the WAL entry and apply it to local persistent storage. 2. The system shall update the standby's applied log sequence number after each successfully applied WAL entry, so that the next heartbeat reply reflects the latest state. 3. If a WAL entry fails to apply (corrupt payload, disk error), the system shall log an ERROR and continue processing subsequent entries without crashing. 4. While the standby is processing WAL catch-up entries, the system shall remain in `CTRL_STANDBY` state and shall not accept job dispatch or queue management requests. 5. The system shall process WAL catch-up entries in log sequence number order, preserving the same ordering the primary committed them in. --- ### Requirement 4: Snapshot Transfer for Standbys Outside WAL Retention Window **Objective:** As a cluster operator, I want a standby whose applied log sequence number predates the oldest retained WAL segment to receive a full state snapshot, so that the standby can catch up and rejoin the replication set rather than being permanently excluded. #### Acceptance Criteria 1. When the primary determines that a standby's applied log sequence number predates the oldest retained WAL segment, the system shall create a consistent checkpoint of its current in-memory state and transfer the resulting snapshot data to the standby. 2. When a standby receives a snapshot transfer, the system shall load the snapshot to replace its local state, set its applied log sequence number to the snapshot's log sequence number, and then report the updated log sequence number in subsequent heartbeat replies. 3. When a standby receives a snapshot transfer, the system shall not accept job dispatch or queue management requests until the snapshot is fully loaded and all subsequent WAL catch-up entries are applied. 4. If snapshot transfer fails (connection drops, disk full), the standby shall log an ERROR and retry by reconnecting to the primary and initiating a new ctrlhello handshake. 5. When the cluster is a single node, the system shall not initiate snapshot transfer (no standbys exist). --- ### Requirement 5: Batch WAL Catch-Up Entry Transfer **Objective:** As a cluster operator, I want the primary to transfer multiple WAL entries in a single message during standby catch-up, so that a fresh standby joining a long-running cluster can catch up in reasonable time rather than being blocked by per-entry overhead. #### Acceptance Criteria 1. When the primary streams WAL entries to a catching-up standby, the system shall pack multiple entries into each `OPCode_walCatchupEntry` message rather than sending one message per entry. 2. The system shall accumulate entries up to a configurable maximum batch size (in bytes or entry count) before sending. 3. When a standby receives a batch of WAL catch-up entries, the system shall apply each entry in order and update the applied log sequence number after the entire batch is durably written, so that the next heartbeat reply reflects the post-batch state. 4. If the primary has fewer entries remaining than the batch size, the system shall send a partial batch without waiting for more entries. 5. The batch transfer mechanism shall not change the ordering or durability guarantees of WAL entries — entries within a batch are applied in log sequence number order, and all entries in the batch are durably written before the applied log sequence number is advanced. --- ### Requirement 6: COMMITTING Phase Gated on Majority Convergence **Objective:** As a cluster operator, I want the elected primary to remain in `CTRL_PRIMARY_COMMITTING` state until a majority of standbys report converged state via heartbeat replies, so that the cluster is guaranteed to be in a majority-consistent state before accepting client traffic. #### Acceptance Criteria 1. While in `CTRL_PRIMARY_COMMITTING` state, the system shall continue sending heartbeats to all standbys and processing heartbeat replies to track convergence. 2. While in `CTRL_PRIMARY_COMMITTING` state, the system shall accept only replication-related opcodes (WAL ack, config push ack, ctrlhello) and shall reply with a not-ready error to job dispatch, queue management, and resource management opcodes. 3. When a majority of standbys have reported converged state via heartbeat replies, the system shall transition to `CTRL_PRIMARY` immediately without waiting for any timeout. 4. If the committing timeout expires before majority convergence is reached, the system shall log a WARN and may transition to `CTRL_PRIMARY` in degraded mode, allowing the cluster to make progress when standbys are genuinely unavailable. 5. When the cluster is a single node, the system shall consider the convergence criterion satisfied immediately (no standbys needed). --- ### Requirement 7: Observability for Replication Progress **Objective:** As a cluster operator, I want structured log events for replication state transitions, so that catch-up progress, snapshot transfers, and majority-convergence status can be monitored without external tooling. #### Acceptance Criteria 1. When a standby begins WAL catch-up, the system shall log at INFO level the starting and target log sequence numbers. 2. When the primary detects that a standby has converged via heartbeat reply, the system shall log at INFO level the standby's identity and its reported config tag and applied log sequence number. 3. When a snapshot transfer begins and completes, the system shall log at INFO level the snapshot log sequence number and the transfer size. 4. When the primary reaches majority convergence and transitions to `CTRL_PRIMARY`, the system shall log at INFO level the number of standbys that have converged and the quorum size. 5. When the committing timeout fires before majority convergence is reached, the system shall log at WARN level indicating the cluster is entering degraded mode, including the number of standbys that have converged and the number still pending.