468 lines
26 KiB
Markdown
468 lines
26 KiB
Markdown
# Design: HA Replication Hardening
|
||
|
||
## Overview
|
||
|
||
**Purpose**: This feature hardens the XCE controller replication path so that the primary maintains majority-consistent state as a continuous invariant throughout its lifecycle — from post-election bootstrap through every state-changing operation until failover.
|
||
|
||
**Users**: Cluster operators who require CP semantics (consistency over availability) — no acknowledged state change is ever lost on primary failover.
|
||
|
||
**Impact**: Replaces the current timeout-only COMMITTING path with a unified heartbeat-driven replication protocol. The standby is fully passive — it only reports its current `(config_tag, applied_lsn)` in heartbeat replies. The primary makes all replication decisions (config push, snapshot, WAL streaming) from that single data point. The same code path handles fresh standby bootstrap, catch-up after disconnection, and continuous state verification.
|
||
|
||
### Goals
|
||
|
||
- Unify replication into a single heartbeat-driven decision loop on the primary
|
||
- Standby remains fully passive — replies with state, receives what the primary sends
|
||
- Primary establishes majority consistency before serving first client request
|
||
- Primary continuously verifies majority consistency for every state-changing operation
|
||
- Lagging standbys receive snapshot + batch WAL catch-up to converge
|
||
- `handle_ctrlhello` reduces to connection setup only
|
||
|
||
### Non-Goals
|
||
|
||
- Config push comparison (already implemented — pushed only when tags differ)
|
||
- WAL segment retention policy changes
|
||
- Dynamic cluster membership changes
|
||
- Client-visible API changes
|
||
- Election protocol changes
|
||
|
||
## Majority Consistency Invariant
|
||
|
||
> **Any state change acknowledged to a client has been durably committed on a majority of controllers.**
|
||
|
||
Enforced at three stages:
|
||
|
||
| Stage | Mechanism | When |
|
||
|-------|-----------|------|
|
||
| **Bootstrap** | Heartbeat reply convergence check *(Note: superseded by `ha-single-leader-role`, closed 2026-06-11 — `CTRL_PRIMARY_COMMITTING` and the convergence-gated bootstrap were deleted; the elected node now enters `CTRL_PRIMARY` directly and write-confirmation is gated by a `WAL_NOOP` own-term commit through the commit engine, not by standby convergence tracking. See that spec's design.md.)* | After winning election, before serving client traffic |
|
||
| **Ongoing WAL** | `repl_wait_majority_ack` per entry (existing) | On every job submit, state transition, queue change |
|
||
| **Ongoing Config** | `push_config_to_standbys` + majority ack (existing) | On every config upgrade |
|
||
|
||
## Boundary Commitments
|
||
|
||
### This Spec Owns
|
||
|
||
- Heartbeat reply protocol: standby reports `(applied_lsn, config_tag)` to primary
|
||
- Unified replication decision loop: primary reads heartbeat state → decides config/snapshot/WAL
|
||
- Per-standby convergence tracking in `ha_state_t`
|
||
- WAL catch-up entry application on standby side
|
||
- Batch WAL entry encoding and transfer
|
||
- Snapshot creation and transfer
|
||
- COMMITTING→serving transition gated on majority convergence *(Note: superseded by `ha-single-leader-role` — the COMMITTING role and this gate were removed entirely; see note under Majority Consistency Invariant above)*
|
||
|
||
### Out of Boundary
|
||
|
||
- Election protocol (`election-iomgr-refactor`)
|
||
- WAL write path and `repl_wait_majority_ack` (existing in `ha-config-bootstrap`)
|
||
- Config push path and config tag comparison (existing — reused by decision loop)
|
||
- Heartbeat send scheduling (existing `ha_start_primary_heartbeat`)
|
||
- Connection lifecycle (existing iomgr; `handle_ctrlhello` reduces to setup only)
|
||
|
||
### Allowed Dependencies
|
||
|
||
- `primary_heartbeat_fn` — existing heartbeat send loop; extended with reply processing
|
||
- `apply_wal_entry()` — existing WAL entry replay
|
||
- `persist_checkpoint()` — existing snapshot creation
|
||
- `repl_wait_majority_ack()` — existing quorum ack for ongoing operations
|
||
- `push_config_to_standbys()` — existing config push (called from decision loop when tag differs)
|
||
|
||
### Revalidation Triggers
|
||
|
||
- Changes to `RequestReply` union schema in `controller.capnp`
|
||
- Changes to heartbeat interval or send mechanism
|
||
- Changes to WAL segment format or `apply_wal_entry` signature
|
||
- Changes to `standby_conn_t` or replicator tracking
|
||
|
||
## Architecture
|
||
|
||
### Unified Heartbeat-Driven Protocol
|
||
|
||
The standby is fully passive. The primary drives everything:
|
||
|
||
```
|
||
PRIMARY (heartbeat loop, fires every heartbeat_interval_ms)
|
||
|
||
For each standby in standby_endpoints:
|
||
|
||
1. Send OPCode_nop (or skip if recent traffic was sent)
|
||
2. Receive reply → (standby_tag, standby_lsn)
|
||
3. Decision:
|
||
|
||
if standby_tag < primary_committed_tag:
|
||
→ push_config_to_standby(standby)
|
||
return // re-evaluate next heartbeat
|
||
|
||
if standby_lsn < oldest_wal_lsn:
|
||
→ persist_checkpoint() → send OPCode_snapshotTransfer
|
||
return
|
||
|
||
if standby_lsn < primary_committed_lsn:
|
||
→ stream_wal_entries(from=standby_lsn+1, to=committed_lsn)
|
||
return // re-evaluate next heartbeat after catch-up
|
||
|
||
if standby_tag >= primary_tag AND standby_lsn >= committed_lsn:
|
||
→ mark converged ✓
|
||
|
||
4. After loop: if majority converged and role == COMMITTING:
|
||
→ transition to serving state
|
||
```
|
||
|
||
*(Note: step 4 and the COMMITTING role it references were removed by `ha-single-leader-role` (closed 2026-06-11) — see note under Majority Consistency Invariant above. Steps 1-3 (per-standby config/snapshot/WAL decision) remain accurate and are unaffected.)*
|
||
|
||
**Key insight**: The same decision tree handles all standby states — fresh bootstrap (tag=0, lsn=0), catch-up after disconnect (tag=current, lsn=stale), and converged (tag=current, lsn=current). No separate code paths needed.
|
||
|
||
### handle_ctrlhello — Reduced to Connection Setup
|
||
|
||
Before this change, `handle_ctrlhello` (then at config_repl.c:1359, now config_repl.c:2128 post-implementation) orchestrated replication:
|
||
- Registered standby in replication set
|
||
- Checked snapshot need
|
||
- Streamed WAL entries
|
||
- Pushed config
|
||
|
||
After this change, it only does connection bookkeeping:
|
||
1. Set `primary_ctrl_fd = ep->fd` (first ctrlhello from winner)
|
||
2. Call `ha_on_heartbeat_received` to reset liveness watchdog
|
||
3. Arm `ha_arm_heartbeat_timeout_vtask`
|
||
|
||
All replication decisions move to the heartbeat reply handler. The next heartbeat cycle will pick up the new standby and drive replication.
|
||
|
||
### Architecture Diagram
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant P as Primary
|
||
participant S as Standby
|
||
|
||
Note over P,S: --- Connection Setup ---
|
||
S->>P: OPCode_hello (applied_lsn=0, config_tag=0)
|
||
P-->>S: CtrlHelloReply (committed_lsn=10000, config_tag=T1)
|
||
P->>P: handle_ctrlhello: set primary_ctrl_fd, arm liveness
|
||
|
||
Note over P,S: --- First Heartbeat Cycle ---
|
||
P->>S: OPCode_nop (heartbeat)
|
||
S-->>P: reply {applied_lsn=0, config_tag=0}
|
||
P->>P: tag=0 < T1 → push config
|
||
P->>S: OPCode_replConfigPush {config blob, tag=T1}
|
||
|
||
Note over P,S: --- Second Heartbeat Cycle ---
|
||
P->>S: OPCode_nop
|
||
S-->>P: reply {applied_lsn=0, config_tag=T1}
|
||
P->>P: lsn=0 < 10000 → stream WAL [1..10000]
|
||
P->>S: OPCode_walCatchupEntry (batch: 1..256)
|
||
P->>S: OPCode_walCatchupEntry (batch: 257..512)
|
||
P->>S: ... (more batches)
|
||
|
||
Note over P,S: --- Third Heartbeat Cycle ---
|
||
P->>S: OPCode_nop
|
||
S-->>P: reply {applied_lsn=10000, config_tag=T1}
|
||
P->>P: tag=T1 ✓ AND lsn=10000 ✓ → converged!
|
||
P->>P: majority reached → CTRL_PRIMARY
|
||
```
|
||
|
||
### Key Decisions
|
||
|
||
| Decision | Rationale |
|
||
|----------|-----------|
|
||
| Primary drives all replication from heartbeat reply | Single decision point; no separate orchestrator code paths |
|
||
| Standby only reports state, never requests replication | Eliminates standby→primary request messages; standby is purely passive |
|
||
| Config/snapshot/WAL decided from `(tag, lsn)` alone | Two scalars encode all replication state needed |
|
||
| handle_ctrlhello reduces to connection setup | Clean separation: connection lifecycle vs replication logic |
|
||
| One action per heartbeat cycle per standby | Avoids flooding a standby with config + WAL simultaneously; natural pacing. `ha_stream_wal_to_standby()` (named `stream_wal_entries()` in this design; actual as-built name differs) enqueues all batch messages via `xce_request()` (non-blocking) in a single invocation — "one action" means no further decisions for this standby until the next heartbeat reply confirms the new LSN |
|
||
|
||
## File Structure Plan
|
||
|
||
### Modified Files
|
||
|
||
| File | Change | Responsibility |
|
||
|------|--------|---------------|
|
||
| `src/schema/controller.capnp` | Add `stateReport :group {appliedLsn, configTag}` to `RequestReply.data` | Wire protocol |
|
||
| `src/controller/ha.h` | Add `standby_reported_tag[N]`, `standby_reported_lsn[N]`, `standby_converged[N]` | Per-standby tracking |
|
||
| `src/controller/ha.c` | Add `ha_process_standby_state()` — the unified decision loop. Wire `OPCode_walCatchupEntry` handler. Process heartbeat replies. | Core replication logic |
|
||
| `src/controller/cluster.c` | Extend `OPCode_nop` handler: populate `stateReport` with `applied_lsn`/`config_tag` | Standby state reporting |
|
||
| `src/controller/config_repl.c` | Strip `handle_ctrlhello` to connection setup only. Add batch WAL streaming. Implement snapshot transfer. | Replication actions |
|
||
| `src/controller/repl.c` | Minor: add helper to read convergence state from heartbeat tracking | Quorum helper |
|
||
|
||
### No New Files
|
||
|
||
## Components and Interfaces
|
||
|
||
### Component: Unified Replication Decision Loop
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Intent | On each heartbeat reply, read standby state, decide next replication action |
|
||
| Requirements | 1.1–1.4, 2.1–2.5, 6.1–6.5 |
|
||
| Location | `ha.c` — called from heartbeat reply processing |
|
||
|
||
**Responsibilities & Constraints**:
|
||
- Read `(standby_tag, standby_lsn)` from heartbeat reply
|
||
- Compare against `primary_committed_tag` and `primary_committed_lsn`
|
||
- Decide: push config, send snapshot, stream WAL, or mark converged
|
||
- One action per heartbeat cycle per standby — config first, then WAL, then convergence
|
||
|
||
**Contract**:
|
||
```
|
||
Decision(standby_tag, standby_lsn) → Action
|
||
|
||
Action ∈ {
|
||
PUSH_CONFIG, // standby_tag < committed_tag
|
||
SEND_SNAPSHOT, // standby_lsn < oldest_wal_lsn
|
||
STREAM_WAL, // standby_lsn < committed_lsn
|
||
CONVERGED, // tag >= committed AND lsn >= committed
|
||
WAIT // action in progress, re-evaluate next cycle
|
||
}
|
||
```
|
||
|
||
**Invariants**:
|
||
- Config is pushed before WAL (tag must be current before LSN comparison is meaningful)
|
||
- Snapshot takes priority over WAL streaming (full state transfer needed)
|
||
- Convergence is terminal for this cycle — standby won't be re-evaluated until next heartbeat
|
||
|
||
### Component: Heartbeat State Reporter (Standby Side)
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Intent | Populate `stateReport` in every `OPCode_nop` reply |
|
||
| Requirements | 1.1, 1.3, 1.4 |
|
||
| Location | `cluster.c` OPCode_nop handler |
|
||
|
||
**Cap'n Proto Schema Change** (as built — a named struct referenced from the union, not an inline group as originally sketched here):
|
||
```capnp
|
||
# In RequestReply.data union:
|
||
stateReportMsg @15: StateReport;
|
||
|
||
struct StateReport {
|
||
appliedLsn @0 :UInt64; # standby's last applied WAL LSN
|
||
configTag @1 :UInt64; # standby's current config version tag
|
||
term @2 :UInt64; # standby's current raft term (added post-close by bug-1421,
|
||
# AE-reject term-discovery — not part of this design)
|
||
}
|
||
```
|
||
|
||
**Implementation**: In the `OPCode_nop` handler, `data_which` is set to `RequestReply_data_stateReportMsg`, populated from `ha->applied_lsn`, `myconf->config_version_tag`, and (post-close addition) `ha->term`. Function: `ha_nop_reply_state_report()` in `ha.c`.
|
||
|
||
### Component: Per-Standby State Tracking
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Intent | Store each standby's last reported state for convergence evaluation |
|
||
| Requirements | 2.1–2.5 |
|
||
| Location | `ha.h` ha_state_t |
|
||
|
||
**Fields**:
|
||
```c
|
||
/* MAX_HA_CANDIDATES is 4 (not 8), defined in src/include/config.h — not ha.h */
|
||
uint64_t standby_reported_tag[MAX_HA_CANDIDATES];
|
||
uint64_t standby_reported_lsn[MAX_HA_CANDIDATES];
|
||
bool standby_converged[MAX_HA_CANDIDATES];
|
||
```
|
||
|
||
- Indexed by peer index (0..N-1), matching `--peers` order
|
||
- Zero-initialized by `ha_init`
|
||
- Updated atomically from IO worker context on each heartbeat reply
|
||
- Read by convergence evaluator and committing timeout fn
|
||
|
||
### Component: WAL Catch-Up Handler (Standby Side)
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Intent | Apply WAL catch-up entries from primary |
|
||
| Requirements | 3.1–3.5 |
|
||
| Location | `ha.c` or `cluster.c` dispatch switch |
|
||
|
||
Wires the current stub (`OPCode_walCatchupEntry` returns 0) to `apply_wal_entry()`. For batch entries, applies each in order, advances `applied_lsn` after the full batch is durably written. Error entries are logged and skipped.
|
||
|
||
*(Note: `ha-term-safety` (closed 2026-07-17) materially extended the `WalCatchupEntry` wire message beyond this description — it now carries a full Raft AppendEntries-conformant envelope (`term`, `prevLogIndex`, `prevLogTerm`, `leaderCommit`, `leaderCurrentTerm`, and a real `entries: List(WalCatchupEntry)` for batching) so the standby intake runs through `raft_on_append_entries_request`'s logOk/conflict-truncation gate rather than blind sequential apply. See that spec's design.md and bug-1067/bug-1421.)*
|
||
|
||
**Durability**: `applied_lsn` is not persisted to disk. On controller restart the standby always reports `lsn=0`; the primary drives full re-bootstrap (config push → WAL streaming → converged) via the unified decision loop. This is correct and expected — the decision loop handles `lsn=0` identically to a fresh standby.
|
||
|
||
### Component: Snapshot Transfer
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Intent | Transfer full state snapshot when standby cannot catch up via WAL |
|
||
| Requirements | 4.1–4.5 |
|
||
| Location | `config_repl.c`, `persist.c` |
|
||
|
||
#### When Snapshot Is Needed
|
||
|
||
The unified decision loop checks two conditions on each heartbeat reply:
|
||
|
||
1. **WAL gap** (existing `repl_needs_snapshot`): `standby_lsn < oldest_wal_lsn` — the standby's position predates the oldest retained WAL segment. WAL entries needed for catch-up no longer exist.
|
||
|
||
2. **Excessive WAL backlog** (heuristic): `committed_lsn - standby_lsn > wal_streaming_threshold` (configurable, default 10M entries). Even within the WAL window, streaming millions of entries would take too long — a snapshot is faster.
|
||
|
||
*(Note: unverified against current code — `wal_streaming_threshold` and this backlog-size heuristic do not exist anywhere in the source tree. `ha_evaluate_repl_action()` (ha.c) implements only condition 1 above (`standby_lsn + 1 < oldest_wal_lsn`); condition 2 was never built. Not attributable to any later spec — appears to have simply not been implemented.)*
|
||
|
||
**Design principle**: Prefer WAL retention over snapshot. WAL entries are incremental (O(changes per operation)); a snapshot is O(total state size). Operators should configure `wal_retention_hours` large enough that snapshots are rare. Snapshot transfer is the fallback for long-disconnected standbys.
|
||
|
||
#### Transfer Strategy: Chunked + Resumable
|
||
|
||
A full snapshot may be hundreds of MB in a large cluster. Single-message transfer would block the control channel and be unrecoverable on failure.
|
||
|
||
```
|
||
Primary Standby
|
||
│ │
|
||
│ persist_checkpoint() → snap.bin │
|
||
│ OPCode_snapshotTransfer │
|
||
│ {chunk=0, offset=0, │
|
||
│ total=524288000, data=[64KB]} │
|
||
│ ─────────────────────────────────→ │ write to snap.part
|
||
│ │
|
||
│ OPCode_snapshotTransfer │
|
||
│ {chunk=1, offset=65536, ...} │
|
||
│ ─────────────────────────────────→ │ append to snap.part
|
||
│ ... (8000 chunks for 500MB) ... │
|
||
│ │
|
||
│ OPCode_snapshotTransfer │
|
||
│ {chunk=N, offset=..., │
|
||
│ data=[final], isLast=true} │
|
||
│ ─────────────────────────────────→ │ verify size, atomic rename
|
||
│ │ → load snapshot
|
||
│ │ → applied_lsn = checkpoint_lsn
|
||
│ ←── heartbeat reply ────────────── │ reports new lsn
|
||
```
|
||
|
||
**Chunk format** (per `OPCode_snapshotTransfer` message, as built — `struct SnapshotTransfer` in `controller.capnp`):
|
||
```
|
||
snapshot_lsn :UInt64 — LSN this snapshot corresponds to (design.md named it checkpoint_lsn)
|
||
payload :Data — chunk payload (64KB max) (design.md named it data)
|
||
chunk_index :UInt32 — 0-based chunk number
|
||
chunk_offset :UInt64 — byte offset in the full snapshot
|
||
total_size :UInt64 — total snapshot size in bytes
|
||
is_last :Bool — true for the final chunk
|
||
checkpoint_term :UInt64 — (Note: field added by `raft-ha-core-safety`, closed 2026-07-09,
|
||
not part of this design — the leader's term for snapshot_lsn,
|
||
reconciled into the receiver's per-LSN term index after install
|
||
so the snapshot boundary establishes a verifiable (term, lsn) pair)
|
||
```
|
||
|
||
**Failure handling**: Mid-transfer reconnect always restarts snapshot from chunk 0 — the standby discards its partial `snap.part` file on reconnect and the primary re-creates the checkpoint from scratch. Resumption is intentionally not implemented: snapshots are rare (WAL retention is the primary path), and the extra protocol state (`checkpoint_lsn`, `last_chunk_index`) in `stateReport` would complicate the common heartbeat path for an edge case. The existing reconnect model (standby re-sends ctrlhello, primary drives next heartbeat cycle) naturally handles this without schema changes. Req 4.4's "retry by reconnecting" is satisfied by this restart-on-reconnect approach.
|
||
|
||
**Completion**: When all chunks are received, the standby verifies `sum(chunk_sizes) == total_size`, atomically renames `snap.part` → `snap-{checkpoint_lsn}.bin`, and loads it. The next heartbeat reply reports `applied_lsn = checkpoint_lsn`. `applied_lsn` is in-memory only — it is not persisted to disk. On a subsequent restart the standby reports `lsn=0` and the primary re-runs the full bootstrap sequence.
|
||
|
||
#### Future: Block-Based Differential (Deferred)
|
||
|
||
For very large clusters where even chunked full transfer is slow, a block-based differential approach can be added later:
|
||
|
||
1. Primary divides snapshot into fixed-size blocks (e.g., 4KB), computes SHA256 of each
|
||
2. Standby computes SHA256 of its current state using same block alignment
|
||
3. Primary sends only blocks whose SHA256 differs from the standby's
|
||
4. Standby merges received blocks into its state
|
||
|
||
This is deferred — the chunked approach handles the common case, and WAL retention minimizes how often snapshots are needed.
|
||
|
||
### Component: Batch WAL Sender
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Intent | Transfer WAL entries in batches during catch-up |
|
||
| Requirements | 5.1–5.5 |
|
||
| Location | `config_repl.c` |
|
||
|
||
Accumulates entries up to batch limit (64KB/256 entries), sends as single message. Standby applies all entries in batch, advances `applied_lsn` after batch complete.
|
||
|
||
`ha_stream_wal_to_standby()` (this design's `stream_wal_entries()`) enqueues all batch `OPCode_walCatchupEntry` messages via `xce_request()` (non-blocking send-buffer enqueue) in a single handler invocation — it never blocks the vtaskmgr thread. "One action per heartbeat cycle" means the primary makes no further replication decisions for this standby until the next heartbeat reply arrives confirming the standby's updated `applied_lsn`.
|
||
|
||
*(Note: as built, each `WalCatchupEntry` message also carries the Raft AppendEntries envelope (`term`, `prevLogIndex`, `prevLogTerm`, `leaderCommit`, `leaderCurrentTerm`) added by `ha-term-safety` (closed 2026-07-17) — see the WAL Catch-Up Handler component above.)*
|
||
|
||
## System Flows
|
||
|
||
### Flow 0: Post-Election — Determine Safe Commit Point (Critical Safety)
|
||
|
||
**Optimization**: The winning candidate already has each voter's `peerLastLogLsn` and `peerConfigTag` from the `RequestVoteReply` messages (stored in `ha_elect_ctx_t.votes[]`). This eliminates the need to wait for the first heartbeat round — the safe commit point is computed immediately upon winning.
|
||
|
||
```
|
||
raft_cb_become_leader():
|
||
1. Collect (peerLastLogLsn, peerConfigTag) from votes[i] for all voters
|
||
2. safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
|
||
3. safe_config_tag = max(tag : count of voters with peerConfigTag >= tag) + 1 >= majority)
|
||
4. persist_truncate_wal_above(safe_committed_lsn)
|
||
5. committed_lsn = safe_committed_lsn
|
||
```
|
||
|
||
*(Note: step 4 is stale — bug-1060 (fixed 2026-06-24, closely tied to `raft-commit-safety`'s Leader Completeness work, closed 2026-06-25) proved a bare `safe_committed_lsn` truncation can cut a new leader's own durable committed entries when a slower peer sits in the granting majority. As built, `raft_cb_become_leader` (ha.c:1615) truncates to `ha_promotion_truncate_floor(safe_committed_lsn, persist_durable_floor) = max(safe_committed_lsn, own durable tip)` via `ha_compute_safe_committed_lsn()` (ha.c:1144) — a new leader never truncates its own durable log. See `src/controller/CLAUDE.md` Leader Completeness section.)*
|
||
|
||
**Raft TLA+ alignment**: Raft's `BecomeLeader` resets `matchIndex[j] = 0` for all followers and determines `commitIndex` via subsequent `AppendEntries` round-trips. Our optimization uses the vote replies (which carry the Raft-equivalent of each voter's lastLogIndex) to compute the safe commit point with zero additional round-trips. This is safe because any committed entry was on a majority and at least one voter in the winning quorum has it.
|
||
|
||
### Flow 1: Full Standby Bootstrap (config_tag=0, applied_lsn=0)
|
||
|
||
**Heartbeat Cycle 1**: standby reports `(tag=0, lsn=0)` → primary pushes config
|
||
**Heartbeat Cycle 2**: standby reports `(tag=T, lsn=0)` → primary streams WAL [1..N]
|
||
**Heartbeat Cycle 3+**: standby reports `(tag=T, lsn=N)` → converged
|
||
|
||
Each cycle is one heartbeat interval (3s). For a fresh cluster with empty WAL, bootstrap completes in 2 cycles (~6s). For a large cluster, WAL streaming spans multiple cycles.
|
||
|
||
### Flow 2: Standby Rejoin After Brief Disconnect
|
||
|
||
Standby reports `(tag=T, lsn=5000)`, primary has `committed_lsn=10000`. Standby is within WAL window → primary streams WAL [5001..10000]. Config already current → no config push. One or more heartbeat cycles depending on WAL volume.
|
||
|
||
### Flow 3: Standby Rejoin After Long Disconnect (Snapshot)
|
||
|
||
Standby reports `(tag=T, lsn=100)`, primary's `oldest_wal_lsn=5000`. Standby predates WAL window → primary creates checkpoint at lsn=10000, sends snapshot. Standby loads snapshot, reports `lsn=10000` next heartbeat → converged.
|
||
|
||
## Requirements Traceability
|
||
|
||
| Req | Design Element |
|
||
|-----|---------------|
|
||
| 1.1–1.4 | `cluster.c` OPCode_nop handler: populate `stateReport`; capnp schema change |
|
||
| 2.1–2.5 | `ha.c`: unified decision loop + per-standby tracking arrays |
|
||
| 3.1–3.5 | `ha.c`/`cluster.c`: wire `OPCode_walCatchupEntry` → `apply_wal_entry` |
|
||
| 4.1–4.5 | `config_repl.c`: snapshot creation + `OPCode_snapshotTransfer` |
|
||
| 5.1–5.5 | `config_repl.c`: batch accumulation in WAL streaming loop |
|
||
| 6.1–6.5 | `ha.c`: heartbeat-driven majority check; committing timeout as 60s fallback *(Note: COMMITTING and its 60s timeout were removed by `ha-single-leader-role` — see note under Majority Consistency Invariant)* |
|
||
| 7.1–7.5 | INFO/WARN log statements throughout new code paths |
|
||
|
||
## Data Models
|
||
|
||
### Cap'n Proto Schema Change
|
||
|
||
```capnp
|
||
# In RequestReply.data union, add:
|
||
stateReport :group {
|
||
appliedLsn @15 :UInt64;
|
||
configTag @16 :UInt64;
|
||
}
|
||
```
|
||
|
||
### In-Memory State
|
||
|
||
```
|
||
ha_state_t
|
||
├── standby_reported_tag[N] uint64_t per-standby last reported config tag
|
||
├── standby_reported_lsn[N] uint64_t per-standby last reported applied LSN
|
||
└── standby_converged[N] bool derived: tag>=committed && lsn>=committed
|
||
```
|
||
|
||
## Error Handling
|
||
|
||
| Error | Handling |
|
||
|-------|----------|
|
||
| WAL entry apply fails | Log ERROR, skip entry, continue; standby reports partial progress next heartbeat |
|
||
| Snapshot transfer fails | Standby logs ERROR, reconnects; primary retries next time standby appears in heartbeat loop |
|
||
| Heartbeat reply lost | Primary re-sends next cycle (3s); no single-reply dependency |
|
||
| Majority lost (standbys disconnect) | Primary continues serving; logs WARN; `repl_wait_majority_ack` returns timeout for new writes |
|
||
| Committing timeout fires (60s) | Primary enters degraded mode; logs WARN with per-standby convergence state *(Note: removed by `ha-single-leader-role` along with the COMMITTING role — see note under Majority Consistency Invariant)* |
|
||
| Config push fails mid-cycle | Standby reports old tag next heartbeat → primary retries config push |
|
||
|
||
## Testing Strategy
|
||
|
||
### Unit Tests
|
||
- Decision loop: `(tag=0, lsn=0)` → PUSH_CONFIG; `(tag=T, lsn=0)` → STREAM_WAL; `(tag<T, lsn=N)` → PUSH_CONFIG; `(tag=T, lsn=N)` → CONVERGED
|
||
- Heartbeat reply populates `stateReport` correctly
|
||
- Majority threshold: `n_peers / 2` (C integer division — primary + converged standbys form a cluster majority):
|
||
- n=3: 3/2 = 1 converged standby needed → primary + 1 = 2 of 3 = majority
|
||
- n=5: 5/2 = 2 converged standbys needed → primary + 2 = 3 of 5 = majority
|
||
- n=1: 1/2 = 0 → immediate promotion *(reconciled 2026-06-09)*
|
||
- WAL catch-up: apply single entry, batch, error entry skipped
|
||
- `handle_ctrlhello` no longer triggers replication — only connection setup
|
||
|
||
### Integration Tests
|
||
- 3-node cluster: primary enters serving via majority convergence (not timeout) *(Note: superseded by `ha-single-leader-role` — primary now enters CTRL_PRIMARY directly and gates confirmation on a WAL_NOOP own-term commit; see note under Majority Consistency Invariant)*
|
||
- Fresh standby (tag=0, lsn=0): bootstrap through config → WAL → converged in heartbeat cycles
|
||
- Standby rejoin within WAL window: WAL streaming only, no config push, no snapshot
|
||
- Standby rejoin outside WAL window: snapshot transfer → WAL streaming → converged
|
||
- Config upgrade: majority ack before committed; standby reports updated tag next heartbeat
|
||
|
||
### E2E Tests
|
||
- Full cluster startup via `/xce-deploy-ha`: verify single PRIMARY via convergence, no degraded mode
|
||
- Failover: kill primary, standby promotes, serves from committed state
|