Rongsong Shen 06135954ff chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family

Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.

2026-06-11 13:21:50 +00:00

4.3 KiB

Raw Blame History

Requirements Document

Project Description (Input)

Bridge the WAL replicator (repl.c) and HA connection layer (ha.c/config_repl.c) so that WAL entries written on the primary are transmitted to all connected standbys, ACKed by a majority, and committed before the caller is acknowledged. Currently repl_broadcast() is a no-send stub, repl_wait_majority_ack() returns immediately because no standby endpoints are registered with the replicator, and job submits succeed locally but are invisible on standbys and not durable across failover.

1. Replicator Standby Registration

The primary shall maintain the replicator's standby list in sync with the HA layer's connected standby endpoints.

1.1 When the primary establishes a standby control channel (promoted vote endpoint during election victory, or late-joiner outbound connect), the HA layer shall register the connected endpoint with the replicator via repl_add_standby.

1.2 When a standby disconnects (RDHUP or connection error), the HA layer shall deregister the endpoint from the replicator via repl_remove_standby.

1.3 If repl_add_standby is called with an endpoint already registered, the system shall treat it as idempotent (no-op or update-in-place).

2. WAL Broadcast

The primary shall transmit new WAL entries to all registered standbys.

2.1 When repl_broadcast is called with a WAL entry header and payload, the system shall serialize the entry into a capnp message and enqueue it for transmission to every non-catch-up registered standby via xce_request.

2.2 While a standby is in catch-up mode (in_catch_up flag set), the system shall skip that standby in repl_broadcast and not queue WAL entries for it.

2.3 If a standby's pending ACK count exceeds MAX_PENDING_ACKS, the system shall trigger a catch-up reconnect for that standby and skip it in the current broadcast.

2.4 If no standbys are registered, repl_broadcast shall return 0 immediately without sending any messages.

3. Standby WAL Receipt and ACK

The standby shall apply received WAL entries and acknowledge them to the primary.

3.1 When the standby receives a WAL entry from the primary, it shall apply the entry to local persistent storage via apply_wal_entry and advance its applied_lsn.

3.2 After successfully applying a WAL entry, the standby shall send an ACK reply to the primary carrying the applied LSN.

3.3 If apply_wal_entry fails for a received WAL entry, the standby shall log the error, skip the entry, and not send an ACK for it.

3.4 The standby shall remain in CTRL_STANDBY state throughout WAL receipt and application — it shall not accept job dispatch or client traffic while applying replicated entries.

4. Majority Commit

The primary shall advance committed_lsn only after a majority of standbys have acknowledged a WAL entry.

4.1 When repl_wait_majority_ack is called for a WAL entry (ack_op == OPCode_walAck), the system shall count standbys whose acked_lsn >= target_lsn and return 0 when the count reaches quorum (ceil((n_active + 1) / 2) including the primary's self-ack).

4.2 If repl_wait_majority_ack times out before quorum is reached, the system shall return -ETIMEDOUT and the caller shall not advance committed_lsn.

4.3 When the primary receives a WAL ACK from a standby, it shall update that standby's acked_lsn in the replicator and signal any waiter on the ACK condition variable.

4.4 In a single-node cluster (no registered standbys), repl_wait_majority_ack shall return 0 immediately and the primary shall advance committed_lsn to the written LSN.

5. Correctness and Observability

5.1 The system shall apply WAL entries on the standby in LSN order — entries delivered out of order shall be buffered or rejected.

5.2 When repl_broadcast transmits a WAL entry to a standby, the system shall log at DEBUG3 level the opcode and target LSN.

5.3 When repl_wait_majority_ack reaches quorum, the system shall log at INFO level the opcode and quorum count.

5.4 When repl_wait_majority_ack times out, the system shall log at WARN level with the opcode, timeout value, and per-standby ACK state.

5.5 All new WAL replication paths shall pass the controller unit test suite under ASAN with zero new leaks or errors.

4.3 KiB Raw Blame History