Rongsong Shen
06135954ff
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
62 lines
4.3 KiB
Markdown
62 lines
4.3 KiB
Markdown
# Requirements Document
|
|
|
|
## Project Description (Input)
|
|
Bridge the WAL replicator (`repl.c`) and HA connection layer (`ha.c`/`config_repl.c`) so that WAL entries written on the primary are transmitted to all connected standbys, ACKed by a majority, and committed before the caller is acknowledged. Currently `repl_broadcast()` is a no-send stub, `repl_wait_majority_ack()` returns immediately because no standby endpoints are registered with the replicator, and job submits succeed locally but are invisible on standbys and not durable across failover.
|
|
|
|
## 1. Replicator Standby Registration
|
|
|
|
The primary shall maintain the replicator's standby list in sync with the HA layer's connected standby endpoints.
|
|
|
|
**1.1** When the primary establishes a standby control channel (promoted vote endpoint during election victory, or late-joiner outbound connect), the HA layer shall register the connected endpoint with the replicator via `repl_add_standby`.
|
|
|
|
**1.2** When a standby disconnects (RDHUP or connection error), the HA layer shall deregister the endpoint from the replicator via `repl_remove_standby`.
|
|
|
|
**1.3** If `repl_add_standby` is called with an endpoint already registered, the system shall treat it as idempotent (no-op or update-in-place).
|
|
|
|
## 2. WAL Broadcast
|
|
|
|
The primary shall transmit new WAL entries to all registered standbys.
|
|
|
|
**2.1** When `repl_broadcast` is called with a WAL entry header and payload, the system shall serialize the entry into a capnp message and enqueue it for transmission to every non-catch-up registered standby via `xce_request`.
|
|
|
|
**2.2** While a standby is in catch-up mode (`in_catch_up` flag set), the system shall skip that standby in `repl_broadcast` and not queue WAL entries for it.
|
|
|
|
**2.3** If a standby's pending ACK count exceeds `MAX_PENDING_ACKS`, the system shall trigger a catch-up reconnect for that standby and skip it in the current broadcast.
|
|
|
|
**2.4** If no standbys are registered, `repl_broadcast` shall return 0 immediately without sending any messages.
|
|
|
|
## 3. Standby WAL Receipt and ACK
|
|
|
|
The standby shall apply received WAL entries and acknowledge them to the primary.
|
|
|
|
**3.1** When the standby receives a WAL entry from the primary, it shall apply the entry to local persistent storage via `apply_wal_entry` and advance its `applied_lsn`.
|
|
|
|
**3.2** After successfully applying a WAL entry, the standby shall send an ACK reply to the primary carrying the applied LSN.
|
|
|
|
**3.3** If `apply_wal_entry` fails for a received WAL entry, the standby shall log the error, skip the entry, and not send an ACK for it.
|
|
|
|
**3.4** The standby shall remain in `CTRL_STANDBY` state throughout WAL receipt and application — it shall not accept job dispatch or client traffic while applying replicated entries.
|
|
|
|
## 4. Majority Commit
|
|
|
|
The primary shall advance `committed_lsn` only after a majority of standbys have acknowledged a WAL entry.
|
|
|
|
**4.1** When `repl_wait_majority_ack` is called for a WAL entry (ack_op == OPCode_walAck), the system shall count standbys whose `acked_lsn >= target_lsn` and return 0 when the count reaches quorum (ceil((n_active + 1) / 2) including the primary's self-ack).
|
|
|
|
**4.2** If `repl_wait_majority_ack` times out before quorum is reached, the system shall return -ETIMEDOUT and the caller shall not advance `committed_lsn`.
|
|
|
|
**4.3** When the primary receives a WAL ACK from a standby, it shall update that standby's `acked_lsn` in the replicator and signal any waiter on the ACK condition variable.
|
|
|
|
**4.4** In a single-node cluster (no registered standbys), `repl_wait_majority_ack` shall return 0 immediately and the primary shall advance `committed_lsn` to the written LSN.
|
|
|
|
## 5. Correctness and Observability
|
|
|
|
**5.1** The system shall apply WAL entries on the standby in LSN order — entries delivered out of order shall be buffered or rejected.
|
|
|
|
**5.2** When `repl_broadcast` transmits a WAL entry to a standby, the system shall log at DEBUG3 level the opcode and target LSN.
|
|
|
|
**5.3** When `repl_wait_majority_ack` reaches quorum, the system shall log at INFO level the opcode and quorum count.
|
|
|
|
**5.4** When `repl_wait_majority_ack` times out, the system shall log at WARN level with the opcode, timeout value, and per-standby ACK state.
|
|
|
|
**5.5** All new WAL replication paths shall pass the controller unit test suite under ASAN with zero new leaks or errors.
|