myxce/.kiro/specs/ha-repl-hardening/design.md

Design: HA Replication Hardening

Overview

Purpose: This feature hardens the XCE controller replication path so that the primary maintains majority-consistent state as a continuous invariant throughout its lifecycle — from post-election bootstrap through every state-changing operation until failover.

Users: Cluster operators who require CP semantics (consistency over availability) — no acknowledged state change is ever lost on primary failover.

Impact: Replaces the current timeout-only COMMITTING path with a unified heartbeat-driven replication protocol. The standby is fully passive — it only reports its current (config_tag, applied_lsn) in heartbeat replies. The primary makes all replication decisions (config push, snapshot, WAL streaming) from that single data point. The same code path handles fresh standby bootstrap, catch-up after disconnection, and continuous state verification.

Goals

• Unify replication into a single heartbeat-driven decision loop on the primary
• Standby remains fully passive — replies with state, receives what the primary sends
• Primary establishes majority consistency before serving first client request
• Primary continuously verifies majority consistency for every state-changing operation
• Lagging standbys receive snapshot + batch WAL catch-up to converge
• handle_ctrlhello reduces to connection setup only

Non-Goals

• Config push comparison (already implemented — pushed only when tags differ)
• WAL segment retention policy changes
• Dynamic cluster membership changes
• Client-visible API changes
• Election protocol changes

Majority Consistency Invariant

Any state change acknowledged to a client has been durably committed on a majority of controllers.

Enforced at three stages:

Stage	Mechanism	When
Bootstrap	Heartbeat reply convergence check	After winning election, before serving client traffic
Ongoing WAL	repl_wait_majority_ack per entry (existing)	On every job submit, state transition, queue change
Ongoing Config	push_config_to_standbys + majority ack (existing)	On every config upgrade

Boundary Commitments

This Spec Owns

• Heartbeat reply protocol: standby reports (applied_lsn, config_tag) to primary
• Unified replication decision loop: primary reads heartbeat state → decides config/snapshot/WAL
• Per-standby convergence tracking in ha_state_t
• WAL catch-up entry application on standby side
• Batch WAL entry encoding and transfer
• Snapshot creation and transfer
• COMMITTING→serving transition gated on majority convergence

Out of Boundary

• Election protocol (election-iomgr-refactor)
• WAL write path and repl_wait_majority_ack (existing in ha-config-bootstrap)
• Config push path and config tag comparison (existing — reused by decision loop)
• Heartbeat send scheduling (existing ha_start_primary_heartbeat)
• Connection lifecycle (existing iomgr; handle_ctrlhello reduces to setup only)

Allowed Dependencies

• primary_heartbeat_fn — existing heartbeat send loop; extended with reply processing
• apply_wal_entry() — existing WAL entry replay
• persist_checkpoint() — existing snapshot creation
• repl_wait_majority_ack() — existing quorum ack for ongoing operations
• push_config_to_standbys() — existing config push (called from decision loop when tag differs)

Revalidation Triggers

• Changes to RequestReply union schema in controller.capnp
• Changes to heartbeat interval or send mechanism
• Changes to WAL segment format or apply_wal_entry signature
• Changes to standby_conn_t or replicator tracking

Architecture

Unified Heartbeat-Driven Protocol

The standby is fully passive. The primary drives everything:

PRIMARY (heartbeat loop, fires every heartbeat_interval_ms)

For each standby in standby_endpoints:

  1. Send OPCode_nop (or skip if recent traffic was sent)
  2. Receive reply → (standby_tag, standby_lsn)
  3. Decision:
     
     if standby_tag < primary_committed_tag:
         → push_config_to_standby(standby)
         return  // re-evaluate next heartbeat
     
     if standby_lsn < oldest_wal_lsn:
         → persist_checkpoint() → send OPCode_snapshotTransfer
         return
     
     if standby_lsn < primary_committed_lsn:
         → stream_wal_entries(from=standby_lsn+1, to=committed_lsn)
         return  // re-evaluate next heartbeat after catch-up
     
     if standby_tag >= primary_tag AND standby_lsn >= committed_lsn:
         → mark converged ✓
  
  4. After loop: if majority converged and role == COMMITTING:
         → transition to serving state

Key insight: The same decision tree handles all standby states — fresh bootstrap (tag=0, lsn=0), catch-up after disconnect (tag=current, lsn=stale), and converged (tag=current, lsn=current). No separate code paths needed.

handle_ctrlhello — Reduced to Connection Setup

Before this change, handle_ctrlhello (config_repl.c:1359) orchestrated replication:

• Registered standby in replication set
• Checked snapshot need
• Streamed WAL entries
• Pushed config

After this change, it only does connection bookkeeping:

1. Set primary_ctrl_fd = ep->fd (first ctrlhello from winner)
2. Call ha_on_heartbeat_received to reset liveness watchdog
3. Arm ha_arm_heartbeat_timeout_vtask

All replication decisions move to the heartbeat reply handler. The next heartbeat cycle will pick up the new standby and drive replication.

Architecture Diagram

sequenceDiagram
    participant P as Primary
    participant S as Standby

    Note over P,S: --- Connection Setup ---
    S->>P: OPCode_hello (applied_lsn=0, config_tag=0)
    P-->>S: CtrlHelloReply (committed_lsn=10000, config_tag=T1)
    P->>P: handle_ctrlhello: set primary_ctrl_fd, arm liveness

    Note over P,S: --- First Heartbeat Cycle ---
    P->>S: OPCode_nop (heartbeat)
    S-->>P: reply {applied_lsn=0, config_tag=0}
    P->>P: tag=0 < T1 → push config
    P->>S: OPCode_replConfigPush {config blob, tag=T1}

    Note over P,S: --- Second Heartbeat Cycle ---
    P->>S: OPCode_nop
    S-->>P: reply {applied_lsn=0, config_tag=T1}
    P->>P: lsn=0 < 10000 → stream WAL [1..10000]
    P->>S: OPCode_walCatchupEntry (batch: 1..256)
    P->>S: OPCode_walCatchupEntry (batch: 257..512)
    P->>S: ... (more batches)

    Note over P,S: --- Third Heartbeat Cycle ---
    P->>S: OPCode_nop
    S-->>P: reply {applied_lsn=10000, config_tag=T1}
    P->>P: tag=T1 ✓ AND lsn=10000 ✓ → converged!
    P->>P: majority reached → CTRL_PRIMARY

Key Decisions

Decision	Rationale
Primary drives all replication from heartbeat reply	Single decision point; no separate orchestrator code paths
Standby only reports state, never requests replication	Eliminates standby→primary request messages; standby is purely passive
Config/snapshot/WAL decided from (tag, lsn) alone	Two scalars encode all replication state needed
handle_ctrlhello reduces to connection setup	Clean separation: connection lifecycle vs replication logic
One action per heartbeat cycle per standby	Avoids flooding a standby with config + WAL simultaneously; natural pacing. stream_wal_entries() enqueues all batch messages via xce_request() (non-blocking) in a single invocation — "one action" means no further decisions for this standby until the next heartbeat reply confirms the new LSN

File Structure Plan

Modified Files

File	Change	Responsibility
src/schema/controller.capnp	Add stateReport :group {appliedLsn, configTag} to RequestReply.data	Wire protocol
src/controller/ha.h	Add standby_reported_tag[N], standby_reported_lsn[N], standby_converged[N]	Per-standby tracking
src/controller/ha.c	Add ha_process_standby_state() — the unified decision loop. Wire OPCode_walCatchupEntry handler. Process heartbeat replies.	Core replication logic
src/controller/cluster.c	Extend OPCode_nop handler: populate stateReport with applied_lsn/config_tag	Standby state reporting
src/controller/config_repl.c	Strip handle_ctrlhello to connection setup only. Add batch WAL streaming. Implement snapshot transfer.	Replication actions
src/controller/repl.c	Minor: add helper to read convergence state from heartbeat tracking	Quorum helper

No New Files

Components and Interfaces

Component: Unified Replication Decision Loop

Field	Detail
Intent	On each heartbeat reply, read standby state, decide next replication action
Requirements	1.1–1.4, 2.1–2.5, 6.1–6.5
Location	ha.c — called from heartbeat reply processing

Responsibilities & Constraints:

• Read (standby_tag, standby_lsn) from heartbeat reply
• Compare against primary_committed_tag and primary_committed_lsn
• Decide: push config, send snapshot, stream WAL, or mark converged
• One action per heartbeat cycle per standby — config first, then WAL, then convergence

Contract:

Decision(standby_tag, standby_lsn) → Action

Action ∈ {
    PUSH_CONFIG,      // standby_tag < committed_tag
    SEND_SNAPSHOT,    // standby_lsn < oldest_wal_lsn
    STREAM_WAL,       // standby_lsn < committed_lsn
    CONVERGED,        // tag >= committed AND lsn >= committed
    WAIT              // action in progress, re-evaluate next cycle
}

Invariants:

• Config is pushed before WAL (tag must be current before LSN comparison is meaningful)
• Snapshot takes priority over WAL streaming (full state transfer needed)
• Convergence is terminal for this cycle — standby won't be re-evaluated until next heartbeat

Component: Heartbeat State Reporter (Standby Side)

Field	Detail
Intent	Populate stateReport in every OPCode_nop reply
Requirements	1.1, 1.3, 1.4
Location	cluster.c OPCode_nop handler

Cap'n Proto Schema Change:

# In RequestReply.data union:
stateReport :group {
    appliedLsn @15 :UInt64;   # standby's last applied WAL LSN
    configTag  @16 :UInt64;   # standby's current config version tag
}

Implementation: In the OPCode_nop handler, replace data_which = nodata with data_which = stateReport, populate from ha->applied_lsn and myconf->config_version_tag.

Component: Per-Standby State Tracking

Field	Detail
Intent	Store each standby's last reported state for convergence evaluation
Requirements	2.1–2.5
Location	ha.h ha_state_t

Fields:

#define MAX_HA_CANDIDATES 8
uint64_t standby_reported_tag[MAX_HA_CANDIDATES];
uint64_t standby_reported_lsn[MAX_HA_CANDIDATES];
bool     standby_converged[MAX_HA_CANDIDATES];

• Indexed by peer index (0..N-1), matching --peers order
• Zero-initialized by ha_init
• Updated atomically from IO worker context on each heartbeat reply
• Read by convergence evaluator and committing timeout fn

Component: WAL Catch-Up Handler (Standby Side)

Field	Detail
Intent	Apply WAL catch-up entries from primary
Requirements	3.1–3.5
Location	ha.c or cluster.c dispatch switch

Wires the current stub (OPCode_walCatchupEntry returns 0) to apply_wal_entry(). For batch entries, applies each in order, advances applied_lsn after the full batch is durably written. Error entries are logged and skipped.

Durability: applied_lsn is not persisted to disk. On controller restart the standby always reports lsn=0; the primary drives full re-bootstrap (config push → WAL streaming → converged) via the unified decision loop. This is correct and expected — the decision loop handles lsn=0 identically to a fresh standby.

Component: Snapshot Transfer

Field	Detail
Intent	Transfer full state snapshot when standby cannot catch up via WAL
Requirements	4.1–4.5
Location	config_repl.c, persist.c

When Snapshot Is Needed

The unified decision loop checks two conditions on each heartbeat reply:

1. WAL gap (existing repl_needs_snapshot): standby_lsn < oldest_wal_lsn — the standby's position predates the oldest retained WAL segment. WAL entries needed for catch-up no longer exist.

2. Excessive WAL backlog (heuristic): committed_lsn - standby_lsn > wal_streaming_threshold (configurable, default 10M entries). Even within the WAL window, streaming millions of entries would take too long — a snapshot is faster.

Design principle: Prefer WAL retention over snapshot. WAL entries are incremental (O(changes per operation)); a snapshot is O(total state size). Operators should configure wal_retention_hours large enough that snapshots are rare. Snapshot transfer is the fallback for long-disconnected standbys.

Transfer Strategy: Chunked + Resumable

A full snapshot may be hundreds of MB in a large cluster. Single-message transfer would block the control channel and be unrecoverable on failure.

Primary                              Standby
  │                                    │
  │ persist_checkpoint() → snap.bin   │
  │ OPCode_snapshotTransfer             │
  │   {chunk=0, offset=0,              │
  │    total=524288000, data=[64KB]}   │
  │ ─────────────────────────────────→ │ write to snap.part
  │                                    │
  │ OPCode_snapshotTransfer             │
  │   {chunk=1, offset=65536, ...}     │
  │ ─────────────────────────────────→ │ append to snap.part
  │ ... (8000 chunks for 500MB) ...    │
  │                                    │
  │ OPCode_snapshotTransfer             │
  │   {chunk=N, offset=...,            │
  │    data=[final], isLast=true}      │
  │ ─────────────────────────────────→ │ verify size, atomic rename
  │                                    │ → load snapshot
  │                                    │ → applied_lsn = checkpoint_lsn
  │ ←── heartbeat reply ────────────── │ reports new lsn

Chunk format (per OPCode_snapshotTransfer message):

checkpoint_lsn :UInt64    — LSN this snapshot corresponds to
chunk_index    :UInt32    — 0-based chunk number
chunk_offset   :UInt64    — byte offset in the full snapshot
total_size     :UInt64    — total snapshot size in bytes
is_last        :Bool      — true for the final chunk
data           :Data      — chunk payload (64KB max)

Failure handling: Mid-transfer reconnect always restarts snapshot from chunk 0 — the standby discards its partial snap.part file on reconnect and the primary re-creates the checkpoint from scratch. Resumption is intentionally not implemented: snapshots are rare (WAL retention is the primary path), and the extra protocol state (checkpoint_lsn, last_chunk_index) in stateReport would complicate the common heartbeat path for an edge case. The existing reconnect model (standby re-sends ctrlhello, primary drives next heartbeat cycle) naturally handles this without schema changes. Req 4.4's "retry by reconnecting" is satisfied by this restart-on-reconnect approach.

Completion: When all chunks are received, the standby verifies sum(chunk_sizes) == total_size, atomically renames snap.part → snap-{checkpoint_lsn}.bin, and loads it. The next heartbeat reply reports applied_lsn = checkpoint_lsn. applied_lsn is in-memory only — it is not persisted to disk. On a subsequent restart the standby reports lsn=0 and the primary re-runs the full bootstrap sequence.

Future: Block-Based Differential (Deferred)

For very large clusters where even chunked full transfer is slow, a block-based differential approach can be added later:

1. Primary divides snapshot into fixed-size blocks (e.g., 4KB), computes SHA256 of each
2. Standby computes SHA256 of its current state using same block alignment
3. Primary sends only blocks whose SHA256 differs from the standby's
4. Standby merges received blocks into its state

This is deferred — the chunked approach handles the common case, and WAL retention minimizes how often snapshots are needed.

Component: Batch WAL Sender

Field	Detail
Intent	Transfer WAL entries in batches during catch-up
Requirements	5.1–5.5
Location	config_repl.c

Accumulates entries up to batch limit (64KB/256 entries), sends as single message. Standby applies all entries in batch, advances applied_lsn after batch complete.

stream_wal_entries() enqueues all batch OPCode_walCatchupEntry messages via xce_request() (non-blocking send-buffer enqueue) in a single handler invocation — it never blocks the vtaskmgr thread. "One action per heartbeat cycle" means the primary makes no further replication decisions for this standby until the next heartbeat reply arrives confirming the standby's updated applied_lsn.

System Flows

Flow 0: Post-Election — Determine Safe Commit Point (Critical Safety)

Optimization: The winning candidate already has each voter's peerLastLogLsn and peerConfigTag from the RequestVoteReply messages (stored in ha_elect_ctx_t.votes[]). This eliminates the need to wait for the first heartbeat round — the safe commit point is computed immediately upon winning.

raft_cb_become_leader():
  1. Collect (peerLastLogLsn, peerConfigTag) from votes[i] for all voters
  2. safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
  3. safe_config_tag = max(tag : count of voters with peerConfigTag >= tag) + 1 >= majority)
  4. persist_truncate_wal_above(safe_committed_lsn)
  5. committed_lsn = safe_committed_lsn

Raft TLA+ alignment: Raft's BecomeLeader resets matchIndex[j] = 0 for all followers and determines commitIndex via subsequent AppendEntries round-trips. Our optimization uses the vote replies (which carry the Raft-equivalent of each voter's lastLogIndex) to compute the safe commit point with zero additional round-trips. This is safe because any committed entry was on a majority and at least one voter in the winning quorum has it.

Flow 1: Full Standby Bootstrap (config_tag=0, applied_lsn=0)

Heartbeat Cycle 1: standby reports (tag=0, lsn=0) → primary pushes config Heartbeat Cycle 2: standby reports (tag=T, lsn=0) → primary streams WAL [1..N] Heartbeat Cycle 3+: standby reports (tag=T, lsn=N) → converged

Each cycle is one heartbeat interval (3s). For a fresh cluster with empty WAL, bootstrap completes in 2 cycles (~6s). For a large cluster, WAL streaming spans multiple cycles.

Flow 2: Standby Rejoin After Brief Disconnect

Standby reports (tag=T, lsn=5000), primary has committed_lsn=10000. Standby is within WAL window → primary streams WAL [5001..10000]. Config already current → no config push. One or more heartbeat cycles depending on WAL volume.

Flow 3: Standby Rejoin After Long Disconnect (Snapshot)

Standby reports (tag=T, lsn=100), primary's oldest_wal_lsn=5000. Standby predates WAL window → primary creates checkpoint at lsn=10000, sends snapshot. Standby loads snapshot, reports lsn=10000 next heartbeat → converged.

Requirements Traceability

Req	Design Element
1.1–1.4	cluster.c OPCode_nop handler: populate stateReport; capnp schema change
2.1–2.5	ha.c: unified decision loop + per-standby tracking arrays
3.1–3.5	ha.c/cluster.c: wire OPCode_walCatchupEntry → apply_wal_entry
4.1–4.5	config_repl.c: snapshot creation + OPCode_snapshotTransfer
5.1–5.5	config_repl.c: batch accumulation in WAL streaming loop
6.1–6.5	ha.c: heartbeat-driven majority check; committing timeout as 60s fallback
7.1–7.5	INFO/WARN log statements throughout new code paths

Data Models

Cap'n Proto Schema Change

# In RequestReply.data union, add:
stateReport :group {
    appliedLsn @15 :UInt64;
    configTag  @16 :UInt64;
}

In-Memory State

ha_state_t
├── standby_reported_tag[N]     uint64_t   per-standby last reported config tag
├── standby_reported_lsn[N]     uint64_t   per-standby last reported applied LSN
└── standby_converged[N]        bool       derived: tag>=committed && lsn>=committed

Error Handling

Error	Handling
WAL entry apply fails	Log ERROR, skip entry, continue; standby reports partial progress next heartbeat
Snapshot transfer fails	Standby logs ERROR, reconnects; primary retries next time standby appears in heartbeat loop
Heartbeat reply lost	Primary re-sends next cycle (3s); no single-reply dependency
Majority lost (standbys disconnect)	Primary continues serving; logs WARN; repl_wait_majority_ack returns timeout for new writes
Committing timeout fires (60s)	Primary enters degraded mode; logs WARN with per-standby convergence state
Config push fails mid-cycle	Standby reports old tag next heartbeat → primary retries config push

Testing Strategy

Unit Tests

• Decision loop: (tag=0, lsn=0) → PUSH_CONFIG; (tag=T, lsn=0) → STREAM_WAL; (tag<T, lsn=N) → PUSH_CONFIG; (tag=T, lsn=N) → CONVERGED
• Heartbeat reply populates stateReport correctly
• Majority threshold: n_peers / 2 (C integer division — primary + converged standbys form a cluster majority):
  • n=3: 3/2 = 1 converged standby needed → primary + 1 = 2 of 3 = majority
  • n=5: 5/2 = 2 converged standbys needed → primary + 2 = 3 of 5 = majority
  • n=1: 1/2 = 0 → immediate promotion (reconciled 2026-06-09)
• WAL catch-up: apply single entry, batch, error entry skipped
• handle_ctrlhello no longer triggers replication — only connection setup

Integration Tests

• 3-node cluster: primary enters serving via majority convergence (not timeout)
• Fresh standby (tag=0, lsn=0): bootstrap through config → WAL → converged in heartbeat cycles
• Standby rejoin within WAL window: WAL streaming only, no config push, no snapshot
• Standby rejoin outside WAL window: snapshot transfer → WAL streaming → converged
• Config upgrade: majority ack before committed; standby reports updated tag next heartbeat

E2E Tests

• Full cluster startup via /xce-deploy-ha: verify single PRIMARY via convergence, no degraded mode
• Failover: kill primary, standby promotes, serves from committed state