Rongsong Shen
06135954ff
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
61 lines
3.2 KiB
Markdown
61 lines
3.2 KiB
Markdown
# Brief: ha-wal-replication
|
|
|
|
## Problem
|
|
WAL entries are written to disk on the primary but never transmitted to standbys.
|
|
`repl_broadcast()` is a no-send stub (casts payload to void, only increments counters).
|
|
`repl_wait_majority_ack()` returns immediately because the replicator's `standbys[]`
|
|
array is never populated from the HA layer's `standby_endpoints`. The result:
|
|
job submits succeed locally but are invisible on standbys and not durable across failover.
|
|
|
|
## Current State
|
|
- `repl.c`: `repl_add_standby()`, `repl_remove_standby()`, `repl_broadcast()`,
|
|
`repl_wait_majority_ack()` all exist with correct signatures. `repl_broadcast`
|
|
is a stub (line 196-197: `(void)payload; (void)payload_len`).
|
|
- `persist.c`: `persist_wal_sync_append()` calls `repl_broadcast` → `repl_wait_majority_ack`
|
|
→ advances `committed_lsn`. The call chain is wired; the send is what's missing.
|
|
- `ha.c` / `config_repl.c`: `ha->standby_endpoints` manages per-standby connections
|
|
(promoted vote eps, late-joiner connects). Nobody calls `repl_add_standby()` from here.
|
|
- `cluster.c`: `OPCode_walAck` handler exists (`ha_repl_ack_received`) but is unreachable
|
|
because standbys never receive WAL entries to ACK.
|
|
|
|
## Desired Outcome
|
|
1. WAL entries written on the primary are transmitted to all connected standbys.
|
|
2. Each standby applies received WAL entries and sends an ACK.
|
|
3. The primary advances `committed_lsn` only after a majority of standbys ACK.
|
|
4. Job submits, state transitions, and queue changes survive primary failover.
|
|
|
|
## Approach
|
|
Bridge the replicator and HA layer:
|
|
1. **Register**: When the HA layer connects to a standby (promoted vote ep or late-joiner),
|
|
call `repl_add_standby()` to register that endpoint with the replicator.
|
|
2. **Deregister**: When a standby disconnects (RDHUP / err_handler), call
|
|
`repl_remove_standby()`.
|
|
3. **Send**: Fill in `repl_broadcast()` to serialize the WAL header + payload into
|
|
an `OPCode_walAck` or similar message and enqueue via `xce_request()`.
|
|
4. **ACK**: On the standby, when a WAL entry is applied, send a reply carrying the
|
|
acked LSN. On the primary, `handle_wal_ack` calls `repl_on_ack()`.
|
|
|
|
## Scope
|
|
- **In**: repl_broadcast TCP send, HA→replicator standby registration/deregistration,
|
|
standby-side WAL ACK reply, integration with existing `repl_wait_majority_ack`.
|
|
- **Out**: Changes to the heartbeat-driven decision loop (WAL streaming via NOP replies
|
|
is already wired in ha-repl-hardening). Snapshot transfer path. Config replication path.
|
|
|
|
## Boundary Candidates
|
|
- `repl_broadcast` send implementation (repl.c)
|
|
- Standby registration bridge (ha.c / config_repl.c → repl_add_standby)
|
|
- Standby WAL ACK reply (cluster.c / ha.c)
|
|
|
|
## Upstream / Downstream
|
|
- **Upstream**: ha-config-bootstrap (repl_broadcast API, repl_wait_majority_ack),
|
|
ha-repl-hardening (WAL catch-up payload, standby-side apply_wal_entry)
|
|
- **Downstream**: None — this is the last missing link
|
|
|
|
## Existing Spec Touchpoints
|
|
- **Reopens**: ha-config-bootstrap task 7.6 (repl_broadcast TCP send deferred)
|
|
- **Adjacent**: ha-repl-hardening (WAL payload already wired to standby apply path)
|
|
|
|
## Constraints
|
|
- Must use the existing `repl_add_standby` / `repl_remove_standby` API
|
|
- Must reuse `OPCode_walAck` opcode for the ACK path
|
|
- Standby WAL apply must be idempotent (replay-safe)
|