docs(ha-repl-hardening): add spec with reviewed design and approved tasks

- requirements.md: 7 requirements covering heartbeat state reporting, majority convergence tracking, WAL catch-up, snapshot transfer, batch WAL, COMMITTING gate, and observability - design.md: unified heartbeat-driven replication design; reviewed and fixed three issues (snapshot resumption→restart-on-reconnect, WAL streaming non-blocking contract, applied_lsn durability) - tasks.md: 21 sub-tasks across 6 groups; sanity reviewed and fixed (removed resumption language from tasks 3.5/6.4, added Req 3.4 coverage, fixed 2.1/2.2 dependency, log coverage trace in 5.4) - spec.json: design and tasks approved, ready for implementation
2026-06-04 12:58:39 +00:00 · 2026-06-04 12:58:39 +00:00 · 15bd291bdd
commit 15bd291bdd
parent 374f3bfd0e
6 changed files with 1028 additions and 0 deletions
--- a/.kiro/specs/ha-repl-hardening/design.md
+++ b/.kiro/specs/ha-repl-hardening/design.md
@ -0,0 +1,447 @@
+# Design: HA Replication Hardening
+
+## Overview
+
+**Purpose**: This feature hardens the XCE controller replication path so that the primary maintains majority-consistent state as a continuous invariant throughout its lifecycle — from post-election bootstrap through every state-changing operation until failover.
+
+**Users**: Cluster operators who require CP semantics (consistency over availability) — no acknowledged state change is ever lost on primary failover.
+
+**Impact**: Replaces the current timeout-only COMMITTING path with a unified heartbeat-driven replication protocol. The standby is fully passive — it only reports its current `(config_tag, applied_lsn)` in heartbeat replies. The primary makes all replication decisions (config push, snapshot, WAL streaming) from that single data point. The same code path handles fresh standby bootstrap, catch-up after disconnection, and continuous state verification.
+
+### Goals
+
+- Unify replication into a single heartbeat-driven decision loop on the primary
+- Standby remains fully passive — replies with state, receives what the primary sends
+- Primary establishes majority consistency before serving first client request
+- Primary continuously verifies majority consistency for every state-changing operation
+- Lagging standbys receive snapshot + batch WAL catch-up to converge
+- `handle_ctrlhello` reduces to connection setup only
+
+### Non-Goals
+
+- Config push comparison (already implemented — pushed only when tags differ)
+- WAL segment retention policy changes
+- Dynamic cluster membership changes
+- Client-visible API changes
+- Election protocol changes
+
+## Majority Consistency Invariant
+
+> **Any state change acknowledged to a client has been durably committed on a majority of controllers.**
+
+Enforced at three stages:
+
+| Stage | Mechanism | When |
+|-------|-----------|------|
+| **Bootstrap** | Heartbeat reply convergence check | After winning election, before serving client traffic |
+| **Ongoing WAL** | `repl_wait_majority_ack` per entry (existing) | On every job submit, state transition, queue change |
+| **Ongoing Config** | `push_config_to_standbys` + majority ack (existing) | On every config upgrade |
+
+## Boundary Commitments
+
+### This Spec Owns
+
+- Heartbeat reply protocol: standby reports `(applied_lsn, config_tag)` to primary
+- Unified replication decision loop: primary reads heartbeat state → decides config/snapshot/WAL
+- Per-standby convergence tracking in `ha_state_t`
+- WAL catch-up entry application on standby side
+- Batch WAL entry encoding and transfer
+- Snapshot creation and transfer
+- COMMITTING→serving transition gated on majority convergence
+
+### Out of Boundary
+
+- Election protocol (`election-iomgr-refactor`)
+- WAL write path and `repl_wait_majority_ack` (existing in `ha-config-bootstrap`)
+- Config push path and config tag comparison (existing — reused by decision loop)
+- Heartbeat send scheduling (existing `ha_start_primary_heartbeat`)
+- Connection lifecycle (existing iomgr; `handle_ctrlhello` reduces to setup only)
+
+### Allowed Dependencies
+
+- `primary_heartbeat_fn` — existing heartbeat send loop; extended with reply processing
+- `apply_wal_entry()` — existing WAL entry replay
+- `persist_checkpoint()` — existing snapshot creation
+- `repl_wait_majority_ack()` — existing quorum ack for ongoing operations
+- `push_config_to_standbys()` — existing config push (called from decision loop when tag differs)
+
+### Revalidation Triggers
+
+- Changes to `RequestReply` union schema in `controller.capnp`
+- Changes to heartbeat interval or send mechanism
+- Changes to WAL segment format or `apply_wal_entry` signature
+- Changes to `standby_conn_t` or replicator tracking
+
+## Architecture
+
+### Unified Heartbeat-Driven Protocol
+
+The standby is fully passive. The primary drives everything:
+
+```
+PRIMARY (heartbeat loop, fires every heartbeat_interval_ms)
+
+For each standby in standby_endpoints:
+
+  1. Send OPCode_nop (or skip if recent traffic was sent)
+  2. Receive reply → (standby_tag, standby_lsn)
+  3. Decision:
+     
+     if standby_tag < primary_committed_tag:
+         → push_config_to_standby(standby)
+         return  // re-evaluate next heartbeat
+     
+     if standby_lsn < oldest_wal_lsn:
+         → persist_checkpoint() → send OPCode_snapshotTransfer
+         return
+     
+     if standby_lsn < primary_committed_lsn:
+         → stream_wal_entries(from=standby_lsn+1, to=committed_lsn)
+         return  // re-evaluate next heartbeat after catch-up
+     
+     if standby_tag >= primary_tag AND standby_lsn >= committed_lsn:
+         → mark converged ✓
+  
+  4. After loop: if majority converged and role == COMMITTING:
+         → transition to serving state
+```
+
+**Key insight**: The same decision tree handles all standby states — fresh bootstrap (tag=0, lsn=0), catch-up after disconnect (tag=current, lsn=stale), and converged (tag=current, lsn=current). No separate code paths needed.
+
+### handle_ctrlhello — Reduced to Connection Setup
+
+Before this change, `handle_ctrlhello` (config_repl.c:1359) orchestrated replication:
+- Registered standby in replication set
+- Checked snapshot need
+- Streamed WAL entries
+- Pushed config
+
+After this change, it only does connection bookkeeping:
+1. Set `primary_ctrl_fd = ep->fd` (first ctrlhello from winner)
+2. Call `ha_on_heartbeat_received` to reset liveness watchdog
+3. Arm `ha_arm_heartbeat_timeout_vtask`
+
+All replication decisions move to the heartbeat reply handler. The next heartbeat cycle will pick up the new standby and drive replication.
+
+### Architecture Diagram
+
+```mermaid
+sequenceDiagram
+    participant P as Primary
+    participant S as Standby
+
+    Note over P,S: --- Connection Setup ---
+    S->>P: OPCode_hello (applied_lsn=0, config_tag=0)
+    P-->>S: CtrlHelloReply (committed_lsn=10000, config_tag=T1)
+    P->>P: handle_ctrlhello: set primary_ctrl_fd, arm liveness
+
+    Note over P,S: --- First Heartbeat Cycle ---
+    P->>S: OPCode_nop (heartbeat)
+    S-->>P: reply {applied_lsn=0, config_tag=0}
+    P->>P: tag=0 < T1 → push config
+    P->>S: OPCode_replConfigPush {config blob, tag=T1}
+
+    Note over P,S: --- Second Heartbeat Cycle ---
+    P->>S: OPCode_nop
+    S-->>P: reply {applied_lsn=0, config_tag=T1}
+    P->>P: lsn=0 < 10000 → stream WAL [1..10000]
+    P->>S: OPCode_walCatchupEntry (batch: 1..256)
+    P->>S: OPCode_walCatchupEntry (batch: 257..512)
+    P->>S: ... (more batches)
+
+    Note over P,S: --- Third Heartbeat Cycle ---
+    P->>S: OPCode_nop
+    S-->>P: reply {applied_lsn=10000, config_tag=T1}
+    P->>P: tag=T1 ✓ AND lsn=10000 ✓ → converged!
+    P->>P: majority reached → CTRL_PRIMARY
+```
+
+### Key Decisions
+
+| Decision | Rationale |
+|----------|-----------|
+| Primary drives all replication from heartbeat reply | Single decision point; no separate orchestrator code paths |
+| Standby only reports state, never requests replication | Eliminates standby→primary request messages; standby is purely passive |
+| Config/snapshot/WAL decided from `(tag, lsn)` alone | Two scalars encode all replication state needed |
+| handle_ctrlhello reduces to connection setup | Clean separation: connection lifecycle vs replication logic |
+| One action per heartbeat cycle per standby | Avoids flooding a standby with config + WAL simultaneously; natural pacing. `stream_wal_entries()` enqueues all batch messages via `xce_request()` (non-blocking) in a single invocation — "one action" means no further decisions for this standby until the next heartbeat reply confirms the new LSN |
+
+## File Structure Plan
+
+### Modified Files
+
+| File | Change | Responsibility |
+|------|--------|---------------|
+| `src/schema/controller.capnp` | Add `stateReport :group {appliedLsn, configTag}` to `RequestReply.data` | Wire protocol |
+| `src/controller/ha.h` | Add `standby_reported_tag[N]`, `standby_reported_lsn[N]`, `standby_converged[N]` | Per-standby tracking |
+| `src/controller/ha.c` | Add `ha_process_standby_state()` — the unified decision loop. Wire `OPCode_walCatchupEntry` handler. Process heartbeat replies. | Core replication logic |
+| `src/controller/cluster.c` | Extend `OPCode_nop` handler: populate `stateReport` with `applied_lsn`/`config_tag` | Standby state reporting |
+| `src/controller/config_repl.c` | Strip `handle_ctrlhello` to connection setup only. Add batch WAL streaming. Implement snapshot transfer. | Replication actions |
+| `src/controller/repl.c` | Minor: add helper to read convergence state from heartbeat tracking | Quorum helper |
+
+### No New Files
+
+## Components and Interfaces
+
+### Component: Unified Replication Decision Loop
+
+| Field | Detail |
+|-------|--------|
+| Intent | On each heartbeat reply, read standby state, decide next replication action |
+| Requirements | 1.1–1.4, 2.1–2.5, 6.1–6.5 |
+| Location | `ha.c` — called from heartbeat reply processing |
+
+**Responsibilities & Constraints**:
+- Read `(standby_tag, standby_lsn)` from heartbeat reply
+- Compare against `primary_committed_tag` and `primary_committed_lsn`
+- Decide: push config, send snapshot, stream WAL, or mark converged
+- One action per heartbeat cycle per standby — config first, then WAL, then convergence
+
+**Contract**:
+```
+Decision(standby_tag, standby_lsn) → Action
+
+Action ∈ {
+    PUSH_CONFIG,      // standby_tag < committed_tag
+    SEND_SNAPSHOT,    // standby_lsn < oldest_wal_lsn
+    STREAM_WAL,       // standby_lsn < committed_lsn
+    CONVERGED,        // tag >= committed AND lsn >= committed
+    WAIT              // action in progress, re-evaluate next cycle
+}
+```
+
+**Invariants**:
+- Config is pushed before WAL (tag must be current before LSN comparison is meaningful)
+- Snapshot takes priority over WAL streaming (full state transfer needed)
+- Convergence is terminal for this cycle — standby won't be re-evaluated until next heartbeat
+
+### Component: Heartbeat State Reporter (Standby Side)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Populate `stateReport` in every `OPCode_nop` reply |
+| Requirements | 1.1, 1.3, 1.4 |
+| Location | `cluster.c` OPCode_nop handler |
+
+**Cap'n Proto Schema Change**:
+```capnp
+# In RequestReply.data union:
+stateReport :group {
+    appliedLsn @15 :UInt64;   # standby's last applied WAL LSN
+    configTag  @16 :UInt64;   # standby's current config version tag
+}
+```
+
+**Implementation**: In the `OPCode_nop` handler, replace `data_which = nodata` with `data_which = stateReport`, populate from `ha->applied_lsn` and `myconf->config_version_tag`.
+
+### Component: Per-Standby State Tracking
+
+| Field | Detail |
+|-------|--------|
+| Intent | Store each standby's last reported state for convergence evaluation |
+| Requirements | 2.1–2.5 |
+| Location | `ha.h` ha_state_t |
+
+**Fields**:
+```c
+#define MAX_HA_CANDIDATES 8
+uint64_t standby_reported_tag[MAX_HA_CANDIDATES];
+uint64_t standby_reported_lsn[MAX_HA_CANDIDATES];
+bool     standby_converged[MAX_HA_CANDIDATES];
+```
+
+- Indexed by peer index (0..N-1), matching `--peers` order
+- Zero-initialized by `ha_init`
+- Updated atomically from IO worker context on each heartbeat reply
+- Read by convergence evaluator and committing timeout fn
+
+### Component: WAL Catch-Up Handler (Standby Side)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Apply WAL catch-up entries from primary |
+| Requirements | 3.1–3.5 |
+| Location | `ha.c` or `cluster.c` dispatch switch |
+
+Wires the current stub (`OPCode_walCatchupEntry` returns 0) to `apply_wal_entry()`. For batch entries, applies each in order, advances `applied_lsn` after the full batch is durably written. Error entries are logged and skipped.
+
+**Durability**: `applied_lsn` is not persisted to disk. On controller restart the standby always reports `lsn=0`; the primary drives full re-bootstrap (config push → WAL streaming → converged) via the unified decision loop. This is correct and expected — the decision loop handles `lsn=0` identically to a fresh standby.
+
+### Component: Snapshot Transfer
+
+| Field | Detail |
+|-------|--------|
+| Intent | Transfer full state snapshot when standby cannot catch up via WAL |
+| Requirements | 4.1–4.5 |
+| Location | `config_repl.c`, `persist.c` |
+
+#### When Snapshot Is Needed
+
+The unified decision loop checks two conditions on each heartbeat reply:
+
+1. **WAL gap** (existing `repl_needs_snapshot`): `standby_lsn < oldest_wal_lsn` — the standby's position predates the oldest retained WAL segment. WAL entries needed for catch-up no longer exist.
+
+2. **Excessive WAL backlog** (heuristic): `committed_lsn - standby_lsn > wal_streaming_threshold` (configurable, default 10M entries). Even within the WAL window, streaming millions of entries would take too long — a snapshot is faster.
+
+**Design principle**: Prefer WAL retention over snapshot. WAL entries are incremental (O(changes per operation)); a snapshot is O(total state size). Operators should configure `wal_retention_hours` large enough that snapshots are rare. Snapshot transfer is the fallback for long-disconnected standbys.
+
+#### Transfer Strategy: Chunked + Resumable
+
+A full snapshot may be hundreds of MB in a large cluster. Single-message transfer would block the control channel and be unrecoverable on failure.
+
+```
+Primary                              Standby
+  │                                    │
+  │ persist_checkpoint() → snap.bin   │
+  │ OPCode_snapshotTransfer             │
+  │   {chunk=0, offset=0,              │
+  │    total=524288000, data=[64KB]}   │
+  │ ─────────────────────────────────→ │ write to snap.part
+  │                                    │
+  │ OPCode_snapshotTransfer             │
+  │   {chunk=1, offset=65536, ...}     │
+  │ ─────────────────────────────────→ │ append to snap.part
+  │ ... (8000 chunks for 500MB) ...    │
+  │                                    │
+  │ OPCode_snapshotTransfer             │
+  │   {chunk=N, offset=...,            │
+  │    data=[final], isLast=true}      │
+  │ ─────────────────────────────────→ │ verify size, atomic rename
+  │                                    │ → load snapshot
+  │                                    │ → applied_lsn = checkpoint_lsn
+  │ ←── heartbeat reply ────────────── │ reports new lsn
+```
+
+**Chunk format** (per `OPCode_snapshotTransfer` message):
+```
+checkpoint_lsn :UInt64    — LSN this snapshot corresponds to
+chunk_index    :UInt32    — 0-based chunk number
+chunk_offset   :UInt64    — byte offset in the full snapshot
+total_size     :UInt64    — total snapshot size in bytes
+is_last        :Bool      — true for the final chunk
+data           :Data      — chunk payload (64KB max)
+```
+
+**Failure handling**: Mid-transfer reconnect always restarts snapshot from chunk 0 — the standby discards its partial `snap.part` file on reconnect and the primary re-creates the checkpoint from scratch. Resumption is intentionally not implemented: snapshots are rare (WAL retention is the primary path), and the extra protocol state (`checkpoint_lsn`, `last_chunk_index`) in `stateReport` would complicate the common heartbeat path for an edge case. The existing reconnect model (standby re-sends ctrlhello, primary drives next heartbeat cycle) naturally handles this without schema changes. Req 4.4's "retry by reconnecting" is satisfied by this restart-on-reconnect approach.
+
+**Completion**: When all chunks are received, the standby verifies `sum(chunk_sizes) == total_size`, atomically renames `snap.part` → `snap-{checkpoint_lsn}.bin`, and loads it. The next heartbeat reply reports `applied_lsn = checkpoint_lsn`. `applied_lsn` is in-memory only — it is not persisted to disk. On a subsequent restart the standby reports `lsn=0` and the primary re-runs the full bootstrap sequence.
+
+#### Future: Block-Based Differential (Deferred)
+
+For very large clusters where even chunked full transfer is slow, a block-based differential approach can be added later:
+
+1. Primary divides snapshot into fixed-size blocks (e.g., 4KB), computes SHA256 of each
+2. Standby computes SHA256 of its current state using same block alignment
+3. Primary sends only blocks whose SHA256 differs from the standby's
+4. Standby merges received blocks into its state
+
+This is deferred — the chunked approach handles the common case, and WAL retention minimizes how often snapshots are needed.
+
+### Component: Batch WAL Sender
+
+| Field | Detail |
+|-------|--------|
+| Intent | Transfer WAL entries in batches during catch-up |
+| Requirements | 5.1–5.5 |
+| Location | `config_repl.c` |
+
+Accumulates entries up to batch limit (64KB/256 entries), sends as single message. Standby applies all entries in batch, advances `applied_lsn` after batch complete.
+
+`stream_wal_entries()` enqueues all batch `OPCode_walCatchupEntry` messages via `xce_request()` (non-blocking send-buffer enqueue) in a single handler invocation — it never blocks the vtaskmgr thread. "One action per heartbeat cycle" means the primary makes no further replication decisions for this standby until the next heartbeat reply arrives confirming the standby's updated `applied_lsn`.
+
+## System Flows
+
+### Flow 0: Post-Election — Determine Safe Commit Point (Critical Safety)
+
+**Optimization**: The winning candidate already has each voter's `peerLastLogLsn` and `peerConfigTag` from the `RequestVoteReply` messages (stored in `ha_elect_ctx_t.votes[]`). This eliminates the need to wait for the first heartbeat round — the safe commit point is computed immediately upon winning.
+
+```
+raft_cb_become_leader():
+  1. Collect (peerLastLogLsn, peerConfigTag) from votes[i] for all voters
+  2. safe_committed_lsn = max(lsn : count of voters with peerLastLogLsn >= lsn) + 1 >= majority)
+  3. safe_config_tag = max(tag : count of voters with peerConfigTag >= tag) + 1 >= majority)
+  4. persist_truncate_wal_above(safe_committed_lsn)
+  5. committed_lsn = safe_committed_lsn
+```
+
+**Raft TLA+ alignment**: Raft's `BecomeLeader` resets `matchIndex[j] = 0` for all followers and determines `commitIndex` via subsequent `AppendEntries` round-trips. Our optimization uses the vote replies (which carry the Raft-equivalent of each voter's lastLogIndex) to compute the safe commit point with zero additional round-trips. This is safe because any committed entry was on a majority and at least one voter in the winning quorum has it.
+
+### Flow 1: Full Standby Bootstrap (config_tag=0, applied_lsn=0)
+
+**Heartbeat Cycle 1**: standby reports `(tag=0, lsn=0)` → primary pushes config
+**Heartbeat Cycle 2**: standby reports `(tag=T, lsn=0)` → primary streams WAL [1..N]
+**Heartbeat Cycle 3+**: standby reports `(tag=T, lsn=N)` → converged
+
+Each cycle is one heartbeat interval (3s). For a fresh cluster with empty WAL, bootstrap completes in 2 cycles (~6s). For a large cluster, WAL streaming spans multiple cycles.
+
+### Flow 2: Standby Rejoin After Brief Disconnect
+
+Standby reports `(tag=T, lsn=5000)`, primary has `committed_lsn=10000`. Standby is within WAL window → primary streams WAL [5001..10000]. Config already current → no config push. One or more heartbeat cycles depending on WAL volume.
+
+### Flow 3: Standby Rejoin After Long Disconnect (Snapshot)
+
+Standby reports `(tag=T, lsn=100)`, primary's `oldest_wal_lsn=5000`. Standby predates WAL window → primary creates checkpoint at lsn=10000, sends snapshot. Standby loads snapshot, reports `lsn=10000` next heartbeat → converged.
+
+## Requirements Traceability
+
+| Req | Design Element |
+|-----|---------------|
+| 1.1–1.4 | `cluster.c` OPCode_nop handler: populate `stateReport`; capnp schema change |
+| 2.1–2.5 | `ha.c`: unified decision loop + per-standby tracking arrays |
+| 3.1–3.5 | `ha.c`/`cluster.c`: wire `OPCode_walCatchupEntry` → `apply_wal_entry` |
+| 4.1–4.5 | `config_repl.c`: snapshot creation + `OPCode_snapshotTransfer` |
+| 5.1–5.5 | `config_repl.c`: batch accumulation in WAL streaming loop |
+| 6.1–6.5 | `ha.c`: heartbeat-driven majority check; committing timeout as 60s fallback |
+| 7.1–7.5 | INFO/WARN log statements throughout new code paths |
+
+## Data Models
+
+### Cap'n Proto Schema Change
+
+```capnp
+# In RequestReply.data union, add:
+stateReport :group {
+    appliedLsn @15 :UInt64;
+    configTag  @16 :UInt64;
+}
+```
+
+### In-Memory State
+
+```
+ha_state_t
+├── standby_reported_tag[N]     uint64_t   per-standby last reported config tag
+├── standby_reported_lsn[N]     uint64_t   per-standby last reported applied LSN
+└── standby_converged[N]        bool       derived: tag>=committed && lsn>=committed
+```
+
+## Error Handling
+
+| Error | Handling |
+|-------|----------|
+| WAL entry apply fails | Log ERROR, skip entry, continue; standby reports partial progress next heartbeat |
+| Snapshot transfer fails | Standby logs ERROR, reconnects; primary retries next time standby appears in heartbeat loop |
+| Heartbeat reply lost | Primary re-sends next cycle (3s); no single-reply dependency |
+| Majority lost (standbys disconnect) | Primary continues serving; logs WARN; `repl_wait_majority_ack` returns timeout for new writes |
+| Committing timeout fires (60s) | Primary enters degraded mode; logs WARN with per-standby convergence state |
+| Config push fails mid-cycle | Standby reports old tag next heartbeat → primary retries config push |
+
+## Testing Strategy
+
+### Unit Tests
+- Decision loop: `(tag=0, lsn=0)` → PUSH_CONFIG; `(tag=T, lsn=0)` → STREAM_WAL; `(tag<T, lsn=N)` → PUSH_CONFIG; `(tag=T, lsn=N)` → CONVERGED
+- Heartbeat reply populates `stateReport` correctly
+- Majority detection: 1/2 converged → not majority; 2/2 → majority for 3-node
+- WAL catch-up: apply single entry, batch, error entry skipped
+- `handle_ctrlhello` no longer triggers replication — only connection setup
+
+### Integration Tests
+- 3-node cluster: primary enters serving via majority convergence (not timeout)
+- Fresh standby (tag=0, lsn=0): bootstrap through config → WAL → converged in heartbeat cycles
+- Standby rejoin within WAL window: WAL streaming only, no config push, no snapshot
+- Standby rejoin outside WAL window: snapshot transfer → WAL streaming → converged
+- Config upgrade: majority ack before committed; standby reports updated tag next heartbeat
+
+### E2E Tests
+- Full cluster startup via `/xce-deploy-ha`: verify single PRIMARY via convergence, no degraded mode
+- Failover: kill primary, standby promotes, serves from committed state
--- a/.kiro/specs/ha-repl-hardening/raft-tla-audit.md
+++ b/.kiro/specs/ha-repl-hardening/raft-tla-audit.md
@ -0,0 +1,91 @@
+# Raft TLA+ Specification Audit
+
+Comparing XCE HA implementation against the formal Raft TLA+ specification (ongardie/raft.tla).
+
+## State Variable Mapping
+
+| TLA+ Variable | XCE Implementation | Location | Status |
+|--------------|-------------------|----------|--------|
+| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | ha.h:139 | ✅ |
+| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | ha.h:161 | ✅ |
+| `log[i]` | WAL segments in `persist->wal_path` | persist.c | ✅ |
+| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | persist.h | ✅ |
+| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | ha.h:141 | ✅ |
+| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,COMMITTING,PRIMARY,STANDBY} | ha.h:139 | ✅ |
+| `leaderId[i]` | `ha->primary_host` / `primary_port` | ha.h:142-143 | ✅ |
+| `nextIndex[i, j]` | `standby.join_applied_lsn` (where to start sending) | repl.h | ⚠️ Implicit |
+| `matchIndex[i, j]` | `standby.acked_lsn` (highest replicated) | repl.h | ⚠️ Implicit |
+
+**Gap**: `nextIndex`/`matchIndex` are implicit in the replicator, not exposed as first-class per-standby state. The heartbeat-driven design (`standby_reported_lsn`) makes `matchIndex` explicit and visible to the convergence check.
+
+## Action Mapping
+
+| TLA+ Action | XCE Implementation | Location | Status |
+|------------|-------------------|----------|--------|
+| `RequestVote(i, j)` | `raft_on_vote_request()` | raft.c:106 | ✅ |
+| `RequestVoteReply(i, j)` | `raft_on_vote_reply()` | raft.c:169 | ✅ |
+| `AppendEntries(i, j)` | `OPCode_nop` heartbeat + WAL streaming | ha.c, config_repl.c | ⚠️ No follower state in reply |
+| `AppendEntriesReply(i, j)` | Heartbeat reply (currently empty `nodata`) | cluster.c:225 | ⚠️ Missing `success` + `matchIndex` |
+| `BecomeLeader(i)` | `raft_cb_become_leader()` | ha.c:347 | ✅ |
+| `AdvanceCommitIndex(i)` | `committed_lsn` advancement after `repl_wait_majority_ack` | repl.c:408 | ✅ |
+| `ClientRequest(i, v)` | Job submit → WAL append → `repl_wait_majority_ack` | persist.c, repl.c | ✅ |
+| `Timeout(i)` | `raft_cb_schedule_timeout` via delaymgr | ha.c | ✅ |
+
+**Key gap**: `AppendEntries` in Raft is bidirectional — leader sends `(term, prevLogIndex, prevLogTerm, entries[], leaderCommit)`, follower replies `(term, success, matchIndex)`. Our heartbeat (`OPCode_nop`) is unidirectional. This spec fixes exactly this gap by adding `stateReport` to the heartbeat reply.
+
+## Safety Property Audit
+
+### Election Safety (at most one leader per term)
+
+**TLA+**: `∀ i,j ∈ servers : state[i] = Leader ∧ state[j] = Leader ⇒ currentTerm[i] ≠ currentTerm[j]`
+
+**XCE**: ✅ Enforced through `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). To become leader, a candidate must receive votes from a majority, and each server votes at most once per term. Therefore at most one leader per term.
+
+### Log Matching (logs identical up to matching index+term)
+
+**TLA+**: `∀ i,j, index : log[i][index].term = log[j][index].term ⇒ ∀ k ≤ index : log[i][k] = log[j][k]`
+
+**XCE**: ✅ Enforced through election freshness gate. A candidate must have `config_tag ≥ voter's config_tag` and `applied_lsn ≥ voter's applied_lsn` to receive a vote. This guarantees the winner has the most complete log. `persist_truncate_wal_above(committed_lsn)` on promotion removes any divergent uncommitted entries.
+
+### Leader Completeness (committed entries survive leader changes)
+
+**TLA+**: `∀ i, index, term : committed(log[i][index], term) ⇒ ∀ j, futureTerm : leader(j, futureTerm) ⇒ log[j][index].term = term`
+
+**XCE**: ✅ A newly elected leader truncates WAL entries above `last_known_committed_lsn` (from CtrlHelloReply). The freshness gate ensures the winner has at least the committed entries of any voter. Since the winner needs a majority, and any committed entry was on a majority, the winner must have all committed entries.
+
+### State Machine Safety (same index → same entry)
+
+**TLA+**: `∀ i, j, index : applied(log[i][index]) ∧ applied(log[j][index]) ⇒ log[i][index] = log[j][index]`
+
+**XCE**: ✅ `committed_lsn` advances monotonically and WAL entries are applied in order. The truncation on leader change removes divergent entries above committed_lsn, and the freshness gate ensures the new leader has all entries at or below committed_lsn.
+
+### Commit Rule (current-term entry required)
+
+**TLA+**: A leader can commit `log[i][index]` when:
+- `index > commitIndex[i]`
+- `log[i][index].term = currentTerm[i]`
+- A majority of servers have `matchIndex[j] ≥ index`
+
+**XCE**: ✅ Equivalent achieved through per-entry majority ack. Each WAL entry is committed individually via `repl_wait_majority_ack()` — the commit only succeeds after a majority have durably written the entry AND replied with an ACK. Since entries are written by the current leader (single-writer), every committed entry is implicitly from the current leader's term. The "entry from current term" requirement is satisfied because `persist_truncate_wal_above` removes any uncommitted entries from previous terms before the new leader writes its own.
+
+## Findings Summary
+
+### Correctly Implemented
+1. Term management and monotonicity ✅
+2. One-vote-per-term enforcement ✅
+3. Election freshness gate (equivalent to Raft's log completeness check) ✅
+4. Committed entry durability through majority ack ✅
+5. WAL truncation on leader change ✅
+6. Leader completeness through election gate ✅
+
+### Gaps Being Addressed by This Spec
+7. **AppendEntries bidirectional state**: Missing follower state in heartbeat reply → fixed by `stateReport` in this spec
+8. **`matchIndex` visibility**: Implicit in replicator → made explicit via `standby_reported_lsn[N]` in this spec
+
+### Not Applicable (Intentional Differences)
+9. **`nextIndex` decrement on inconsistency**: Not needed — we truncate uncommitted entries on leader change, so there are no divergent entries for the consistency check to catch
+10. **No-op commit of current-term entry**: Not needed — we use per-entry synchronous majority ack instead of batched commit
+
+## Recommendation
+
+No safety gaps found. The two items this spec addresses (heartbeat state reporting, matchIndex visibility) close the remaining protocol gaps vs the formal Raft specification. The per-entry majority ack model is functionally equivalent to Raft's batched commit rule for our append-only single-writer WAL.
--- a/.kiro/specs/ha-repl-hardening/requirements.md
+++ b/.kiro/specs/ha-repl-hardening/requirements.md
@ -0,0 +1,103 @@
+# Requirements Document
+
+## Introduction
+
+XCE controller operators running HA clusters face three replication gaps that prevent the system from reaching a safe majority-consistent state after election. The majority-ack fast path (`ha_repl_ack_received`) exists in code but is unreachable because no standby reports its catch-up status back to the primary. The committing timeout (10s degraded-mode fallback) is the only path to PRIMARY — every startup enters degraded mode. Snapshot transfer for standbys that fall behind the WAL retention window is not implemented, leaving those standbys permanently dead. WAL catch-up streams entries one `xce_request` per entry, blocking a vtaskmgr thread for the entire duration.
+
+This feature implements a heartbeat-driven state synchronization protocol: the primary pushes config and WAL entries to standbys, each standby acknowledges with its current `(config_tag, applied_lsn)` in heartbeat replies, and the primary transitions to `CTRL_PRIMARY` when a majority of standbys report state converged with the primary. Separate catch-up-done and per-entry ack opcodes are eliminated in favor of piggybacked state reporting.
+
+## Boundary Context
+
+- **In scope**: standby-side handling of `OPCode_walCatchupEntry` with local WAL application, heartbeat reply carrying standby state `(config_tag, applied_lsn)`, primary tracking per-standby state and determining majority convergence, snapshot creation and transfer when a standby predates the WAL retention window, batch transfer of multiple WAL entries per message, and gating the `CTRL_PRIMARY` transition on majority state convergence rather than a timeout
+- **Out of scope**: config push optimization (already implemented — config is pushed only when tags differ), WAL segment retention policy changes, dynamic cluster membership, mTLS between controllers, the election protocol itself, new wire protocol opcodes (the feature reuses existing heartbeat and ctrlhello messages extended with state fields)
+- **Adjacent expectations**: the primary already sends heartbeats to standbys via `ha_start_primary_heartbeat` during `CTRL_PRIMARY_COMMITTING`; `CtrlHelloReply` already carries `committed_lsn` and `config_tag` fields in the capnp schema; `apply_wal_entry` already exists and can replay individual WAL entries; `persist_checkpoint` already exists and can create a consistent snapshot
+
+## Requirements
+
+### Requirement 1: Heartbeat Reply Carries Standby State
+
+**Objective:** As a cluster operator, I want each standby to report its current replication state in every heartbeat reply, so that the primary can determine when the cluster has reached majority consistency without a separate catch-up-complete signal.
+
+#### Acceptance Criteria
+1. When a standby receives a heartbeat message from the primary, the system shall reply with a heartbeat acknowledgement containing the standby's current config version tag and applied WAL log sequence number.
+2. When the primary receives a heartbeat reply from a standby, the system shall record that standby's `(config_tag, applied_lsn)` and evaluate whether the standby has converged with the primary's committed state.
+3. A standby is considered converged when its reported config tag is greater than or equal to the primary's committed config tag and its reported applied log sequence number is greater than or equal to the primary's committed log sequence number.
+4. If a standby has not yet received the current configuration, the system shall report its config tag as zero in heartbeat replies until the config push completes.
+
+---
+
+### Requirement 2: Primary Tracks Per-Standby Convergence State
+
+**Objective:** As a cluster operator, I want the primary to continuously track each standby's reported state, so that it can determine when a majority of standbys have converged and the cluster is safe to enter `CTRL_PRIMARY`.
+
+#### Acceptance Criteria
+1. When the primary enters `CTRL_PRIMARY_COMMITTING` state, the system shall initialize per-standby tracking with each standby marked as not-converged.
+2. When a heartbeat reply arrives from a standby and the standby's reported state satisfies the convergence criteria, the system shall mark that standby as converged and re-evaluate the majority threshold.
+3. When the number of converged standbys reaches the majority threshold (≥ ceil(n_peers/2) − 1 other nodes), the system shall transition the primary to `CTRL_PRIMARY` immediately.
+4. When the primary transitions to `CTRL_PRIMARY` via majority convergence, the system shall log at INFO level the number of converged standbys and the quorum size.
+5. If a converged standby later reports a config tag or applied log sequence number that no longer satisfies the convergence criteria (e.g., after a restart), the system shall mark that standby as not-converged and re-evaluate the majority threshold.
+
+---
+
+### Requirement 3: Standby Applies WAL Catch-Up Entries to Local State
+
+**Objective:** As a cluster operator, I want a standby that receives WAL catch-up entries from the primary to apply them to its local durable state, so that the standby's state converges with the primary's and the standby is eligible to become primary on failover.
+
+#### Acceptance Criteria
+1. When a standby controller receives an `OPCode_walCatchupEntry` message, the system shall decode the WAL entry and apply it to local persistent storage.
+2. The system shall update the standby's applied log sequence number after each successfully applied WAL entry, so that the next heartbeat reply reflects the latest state.
+3. If a WAL entry fails to apply (corrupt payload, disk error), the system shall log an ERROR and continue processing subsequent entries without crashing.
+4. While the standby is processing WAL catch-up entries, the system shall remain in `CTRL_STANDBY` state and shall not accept job dispatch or queue management requests.
+5. The system shall process WAL catch-up entries in log sequence number order, preserving the same ordering the primary committed them in.
+
+---
+
+### Requirement 4: Snapshot Transfer for Standbys Outside WAL Retention Window
+
+**Objective:** As a cluster operator, I want a standby whose applied log sequence number predates the oldest retained WAL segment to receive a full state snapshot, so that the standby can catch up and rejoin the replication set rather than being permanently excluded.
+
+#### Acceptance Criteria
+1. When the primary determines that a standby's applied log sequence number predates the oldest retained WAL segment, the system shall create a consistent checkpoint of its current in-memory state and transfer the resulting snapshot data to the standby.
+2. When a standby receives a snapshot transfer, the system shall load the snapshot to replace its local state, set its applied log sequence number to the snapshot's log sequence number, and then report the updated log sequence number in subsequent heartbeat replies.
+3. When a standby receives a snapshot transfer, the system shall not accept job dispatch or queue management requests until the snapshot is fully loaded and all subsequent WAL catch-up entries are applied.
+4. If snapshot transfer fails (connection drops, disk full), the standby shall log an ERROR and retry by reconnecting to the primary and initiating a new ctrlhello handshake.
+5. When the cluster is a single node, the system shall not initiate snapshot transfer (no standbys exist).
+
+---
+
+### Requirement 5: Batch WAL Catch-Up Entry Transfer
+
+**Objective:** As a cluster operator, I want the primary to transfer multiple WAL entries in a single message during standby catch-up, so that a fresh standby joining a long-running cluster can catch up in reasonable time rather than being blocked by per-entry overhead.
+
+#### Acceptance Criteria
+1. When the primary streams WAL entries to a catching-up standby, the system shall pack multiple entries into each `OPCode_walCatchupEntry` message rather than sending one message per entry.
+2. The system shall accumulate entries up to a configurable maximum batch size (in bytes or entry count) before sending.
+3. When a standby receives a batch of WAL catch-up entries, the system shall apply each entry in order and update the applied log sequence number after the entire batch is durably written, so that the next heartbeat reply reflects the post-batch state.
+4. If the primary has fewer entries remaining than the batch size, the system shall send a partial batch without waiting for more entries.
+5. The batch transfer mechanism shall not change the ordering or durability guarantees of WAL entries — entries within a batch are applied in log sequence number order, and all entries in the batch are durably written before the applied log sequence number is advanced.
+
+---
+
+### Requirement 6: COMMITTING Phase Gated on Majority Convergence
+
+**Objective:** As a cluster operator, I want the elected primary to remain in `CTRL_PRIMARY_COMMITTING` state until a majority of standbys report converged state via heartbeat replies, so that the cluster is guaranteed to be in a majority-consistent state before accepting client traffic.
+
+#### Acceptance Criteria
+1. While in `CTRL_PRIMARY_COMMITTING` state, the system shall continue sending heartbeats to all standbys and processing heartbeat replies to track convergence.
+2. While in `CTRL_PRIMARY_COMMITTING` state, the system shall accept only replication-related opcodes (WAL ack, config push ack, ctrlhello) and shall reply with a not-ready error to job dispatch, queue management, and resource management opcodes.
+3. When a majority of standbys have reported converged state via heartbeat replies, the system shall transition to `CTRL_PRIMARY` immediately without waiting for any timeout.
+4. If the committing timeout expires before majority convergence is reached, the system shall log a WARN and may transition to `CTRL_PRIMARY` in degraded mode, allowing the cluster to make progress when standbys are genuinely unavailable.
+5. When the cluster is a single node, the system shall consider the convergence criterion satisfied immediately (no standbys needed).
+
+---
+
+### Requirement 7: Observability for Replication Progress
+
+**Objective:** As a cluster operator, I want structured log events for replication state transitions, so that catch-up progress, snapshot transfers, and majority-convergence status can be monitored without external tooling.
+
+#### Acceptance Criteria
+1. When a standby begins WAL catch-up, the system shall log at INFO level the starting and target log sequence numbers.
+2. When the primary detects that a standby has converged via heartbeat reply, the system shall log at INFO level the standby's identity and its reported config tag and applied log sequence number.
+3. When a snapshot transfer begins and completes, the system shall log at INFO level the snapshot log sequence number and the transfer size.
+4. When the primary reaches majority convergence and transitions to `CTRL_PRIMARY`, the system shall log at INFO level the number of standbys that have converged and the quorum size.
+5. When the committing timeout fires before majority convergence is reached, the system shall log at WARN level indicating the cluster is entering degraded mode, including the number of standbys that have converged and the number still pending.
--- a/.kiro/specs/ha-repl-hardening/research.md
+++ b/.kiro/specs/ha-repl-hardening/research.md
@ -0,0 +1,135 @@
+# Gap Analysis: HA Replication Hardening
+
+> Analysis date: 2026-06-04. Phase: `requirements-generated`.
+> Purpose: identify what exists, what's missing, and viable implementation approaches.
+
+## Analysis Summary
+
+- **Scope**: 7 requirements — heartbeat-driven state convergence, WAL catch-up wiring, per-standby tracking, snapshot transfer, batch WAL transfer, COMMITTING gate, observability
+- **Key finding**: The primary already sends heartbeats (`OPCode_nop`) and receives replies; the `CtrlHelloReply` already carries `committed_lsn` and `config_tag`; `persist_checkpoint` already exists. The gaps are in wiring, not new infrastructure.
+- **Largest gaps**: (1) heartbeat reply doesn't carry standby's `applied_lsn`/`config_tag`; (2) WAL catch-up handler is a stub; (3) snapshot transfer path is a stub; (4) no per-standby convergence tracking in the primary.
+- **Recommendation**: Extend existing heartbeat reply with state fields; wire the WAL catch-up stub to `apply_wal_entry()`; implement snapshot transfer using existing `persist_checkpoint()`; add per-standby convergence tracking to `ha_state_t` or extend `standby_conn_t`.
+
+## 1. Current State Investigation
+
+### 1.1 Heartbeat Protocol
+
+**Current state**: Asymmetric, unidirectional state push.
+- Primary sends `OPCode_nop` via `primary_heartbeat_fn` (ha.c:2880) every `heartbeat_interval_ms` (3s default)
+- Standby receives nop in `myreq_handler` (cluster.c:225), sends empty `RequestReply`, calls `ha_on_heartbeat_received` (ha.c:2655) to reset liveness watchdog
+- `CtrlHelloRequest`/`CtrlHelloReply` (controller.capnp:835-846) carry `appliedLsn`, `configTag`, `committedLsn` — but only exchanged once on connect, not periodically
+
+**Gap**: Heartbeat reply is empty. Standby's current `(applied_lsn, config_tag)` is not reported back to the primary on each heartbeat cycle.
+
+### 1.2 Per-Standby State Tracking
+
+**Current state**: `standby_conn_t` (repl.h:22-33) tracks per-standby `acked_lsn` and `acked_config_lsn` (for WAL and config quorum), plus `in_catch_up` and `pending_ack_count`. `ha_state_t` has `standby_endpoints` (zarray of endpoints) and `n_peers`, but no per-standby convergence tracking.
+
+**Gap**: No field for the standby's self-reported `applied_lsn` or `config_tag` from heartbeat replies. The primary has no way to know whether a standby has converged without receiving explicit WAL/config ACKs.
+
+### 1.3 WAL Catch-Up Handler
+
+**Current state**: Stub at ha.c dispatch gate (OPCode_walCatchupEntry returns 0). The primary-side streaming loop in config_repl.c:1489-1520 reads WAL segments and sends `OPCode_walCatchupEntry` messages, but the standby does nothing with them.
+
+**Gap**: Standby never applies WAL catch-up entries to local state. `apply_wal_entry()` (persist.c:685) exists and is used during startup WAL replay — can be reused.
+
+### 1.4 Snapshot Transfer
+
+**Current state**: `handle_ctrlhello` (config_repl.c:1359) checks `repl_needs_snapshot()` (repl.c:491) and if true, logs WARN at line 1418 ("snapshot transfer not yet implemented") and skips. `persist_checkpoint()` (persist.c:610) exists and creates a consistent checkpoint.
+
+**Gap**: The snapshot creation and transfer path is entirely unimplemented. Without it, any standby whose LSN predates the WAL retention window is permanently dead.
+
+### 1.5 WAL Entry Batching
+
+**Current state**: The WAL streaming loop in config_repl.c:1481-1521 sends one `xce_request` per WAL entry. `xce_request` enqueues to the endpoint's send buffer (non-blocking), but the loop processes entries sequentially with `pread()` per entry + `malloc`/`free` per payload.
+
+**Gap**: No batching. For a standby catching up on millions of entries, this is CPU-intensive and slow. The wire protocol needs a batch message format.
+
+### 1.6 COMMITTING Gate
+
+**Current state**: `ha_repl_ack_received()` (ha.c:2197) exists with correct quorum math, but is only called from `repl_ack_decode_and_dispatch()` (ha.c:1859) which processes `OPCode_replCatchUpDone`. No standby sends this opcode. The committing timeout (10s) is the only path to PRIMARY.
+
+**Gap**: The majority-convergence check is unreachable. The committing timeout degrades the cluster on every startup.
+
+## 2. Requirement-to-Asset Map
+
+| Req | Key Gap | Key Reusable Asset | Gap Tag |
+|-----|---------|-------------------|---------|
+| 1 — Heartbeat carries standby state | Reply body is empty; no state fields in nop reply | `OPCode_nop` send/receive loop, `CtrlHelloReply` fields in capnp schema, `xce_reply` pattern in cluster.c:225-228 | **Missing** |
+| 2 — Per-standby convergence tracking | No `applied_lsn`/`config_tag` reported per standby | `standby_conn_t` with `acked_lsn`, `ha->standby_endpoints`, `repl_wait_majority_ack` quorum math | **Missing** |
+| 3 — WAL catch-up handler | Stub returns 0 (ha.c:882) | `apply_wal_entry()` (persist.c:685), primary WAL streaming loop (config_repl.c:1489) | **Stub** |
+| 4 — Snapshot transfer | Stub logs WARN, skips (config_repl.c:1417-1425) | `persist_checkpoint()` (persist.c:610), `repl_needs_snapshot()` (repl.c:491), `CtrlHelloReply.needsSnapshot` field | **Missing** |
+| 5 — Batch WAL transfer | One entry per `xce_request` | Existing WAL streaming loop, capnp message format for walCatchupEntry | **Missing** |
+| 6 — COMMITTING gate | `ha_repl_ack_received` unreachable; timeout-only path | `ha_repl_ack_received()` (ha.c:2197) with correct quorum math, `ha_promote_to_primary()` | **Missing** |
+| 7 — Observability | Logs exist but need new convergence/snapshot events | `INFO`/`WARN` macros in xcelog.h | **Missing (alongside)** |
+
+## 3. Implementation Approach Options
+
+### Option A: Extend heartbeat reply with state fields (Recommended)
+
+Extend the existing `OPCode_nop` reply or create a lightweight heartbeat-ack to carry `(applied_lsn, config_tag)`:
+- **Standby side**: In the nop handler (cluster.c:225), populate reply with current `applied_lsn` and `config_tag` before `xce_reply`
+- **Primary side**: In the heartbeat send path or a reply handler, read the standby's reported state and update per-standby tracking
+- **Why**: Minimal new code — reuses existing heartbeat loop; no new opcodes needed if we extend the reply body; follows Raft AppendEntries pattern
+
+**Trade-offs**:
+- ✅ Reuses existing infrastructure; minimal new opcodes
+- ✅ Heartbeat already fires every 3s — convergence detected promptly
+- ❌ Need capnp schema change if RequestReply needs new fields
+- ❌ Primary must process heartbeat replies (currently fire-and-forget)
+
+### Option B: Use CtrlHelloReply periodically
+
+Send `CtrlHelloRequest` periodically (not just on connect) and use the existing `CtrlHelloReply` fields:
+- Already carries `committedLsn`, `configTag` in reply
+- Already carries `appliedLsn`, `configTag` in request
+- **Why**: No capnp schema changes; ctrlhello already carries all needed fields
+
+**Trade-offs**:
+- ✅ No schema changes
+- ❌ CtrlHelloRequest triggers WAL streaming (heavy) — not suitable for periodic use
+- ❌ Semantic mismatch: ctrlhello is a connection-setup message, not a state-reporting message
+
+### Option C: New lightweight opcode
+
+Add `OPCode_stateReport` / `OPCode_stateAck` to capnp schema:
+- Dedicated message type with `(applied_lsn, config_tag, in_catch_up)` fields
+- Sent by standby in response to heartbeat, or periodically
+- **Why**: Clean separation of concerns
+
+**Trade-offs**:
+- ✅ Clean protocol design
+- ❌ New opcode, new capnp struct, new handler on both sides
+
+## 4. Effort and Risk Summary
+
+| Req | Effort | Risk | Justification |
+|-----|--------|------|---------------|
+| 1 — Heartbeat state report | S (1-2d) | Low | Extend existing reply; capnp schema change if needed |
+| 2 — Per-standby tracking | S (1d) | Low | New fields in ha_state_t or extend standby_conn_t |
+| 3 — WAL catch-up wiring | S (1d) | Low | Wire stub to existing apply_wal_entry() |
+| 4 — Snapshot transfer | M (3-5d) | Medium | persist_checkpoint exists; transfer protocol is new |
+| 5 — Batch WAL transfer | S (1-2d) | Low | Accumulate entries in loop; capnp schema change |
+| 6 — COMMITTING gate | S (1d) | Low | Replace ha_repl_ack_received with heartbeat-driven check |
+| 7 — Observability | S (alongside) | Low | INFO/WARN log statements |
+
+**Total estimated effort**: 1-2 weeks. Risk: Low-Medium (snapshot transfer is the main risk).
+
+## 5. Key Reusable Assets
+
+| Asset | Location | Reuse in |
+|-------|----------|----------|
+| `OPCode_nop` send/reply loop | cluster.c:225, ha.c:2880 | Req 1 (heartbeat state report) |
+| `CtrlHelloReply` fields | controller.capnp:835-846 | Req 1 (capnp schema template) |
+| `standby_conn_t` + `repl_wait_majority_ack` | repl.h:22, repl.c:338 | Req 2 (convergence tracking) |
+| `apply_wal_entry()` | persist.c:685 | Req 3 (WAL catch-up) |
+| `persist_checkpoint()` | persist.c:610 | Req 4 (snapshot creation) |
+| WAL streaming loop | config_repl.c:1481 | Req 5 (batch send) |
+| `ha_repl_ack_received()` | ha.c:2197 | Req 6 (quorum math reuse) |
+| `INFO`/`WARN` macros | xcelog.h | Req 7 (observability) |
+
+## 6. Research Needed (Design Phase)
+
+- **Capnp schema design**: Whether to extend `RequestReply` with state fields, add a new `stateReport` struct to existing heartbeat messages, or create a new opcode. Evaluate against existing capnp codegen patterns.
+- **Heartbeat reply processing**: Currently fire-and-forget; need to add a reply handler on the primary side. Determine best insertion point in the dispatch path.
+- **Snapshot wire format**: How to serialize `persist_checkpoint` output for transfer. COW zarray vs raw file transfer.
--- a/.kiro/specs/ha-repl-hardening/spec.json
+++ b/.kiro/specs/ha-repl-hardening/spec.json
@ -0,0 +1,22 @@
+{
+  "feature_name": "ha-repl-hardening",
+  "created_at": "2026-06-04T03:30:00Z",
+  "updated_at": "2026-06-04T10:35:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}
--- a/.kiro/specs/ha-repl-hardening/tasks.md
+++ b/.kiro/specs/ha-repl-hardening/tasks.md
@ -0,0 +1,230 @@
+# Implementation Plan: HA Replication Hardening
+
+---
+
+## Task Overview
+
+7 requirements, 6 design components. Progression: schema first → standby-side handlers → primary-side decision loop → post-election safety → integration cleanup → validation.
+
+---
+
+- [ ] 1. Foundation: Schema changes and state tracking fields
+- [ ] 1.1 Add `stateReport` group to `RequestReply.data` union in controller.capnp
+  - Add `appliedLsn :UInt64 @15` and `configTag :UInt64 @16` fields inside a `stateReport :group` variant
+  - Regenerate capnp bindings; verify compilation succeeds
+  - Existing decode paths and opcode dispatch unaffected
+  - Compilation succeeds and existing controller tests remain green
+  - _Requirements: 1.1, 1.3_
+  - _Boundary: Wire Protocol (controller.capnp)_
+
+- [ ] 1.2 Extend `WalCatchupEntry` for batch support in controller.capnp
+  - Add `entries :List(WalCatchupEntry)` field for multiple entries per message
+  - Regenerate capnp bindings; verify compilation
+  - Compilation succeeds; single-entry decode path still works
+  - _Requirements: 5.1_
+  - _Boundary: Wire Protocol (controller.capnp)_
+
+- [ ] 1.3 Add snapshot chunk fields to capnp schema
+  - Add `checkpointLsn`, `chunkIndex`, `chunkOffset`, `totalSize`, `isLast` fields to the snapshot transfer message
+  - No resumption fields — restart-on-reconnect policy means no snapshot progress state in `stateReport`
+  - Regenerate capnp bindings; verify compilation
+  - Compilation succeeds
+  - _Requirements: 4.1, 4.2_
+  - _Boundary: Wire Protocol (controller.capnp)_
+
+- [ ] 1.4 Add per-standby convergence tracking fields to `ha_state_t`
+  - Add `standby_reported_tag[MAX_HA_CANDIDATES]`, `standby_reported_lsn[MAX_HA_CANDIDATES]`, `standby_converged[MAX_HA_CANDIDATES]` arrays
+  - Zero-initialize in `ha_init()`; these fields are read by the convergence evaluator in task 3.1
+  - Unit test: verify fields are zero after `ha_init()` and writable from test context
+  - _Requirements: 2.1_
+  - _Boundary: Controller FSM (ha.h)_
+
+---
+
+- [ ] 2. Standby-side: Wire WAL catch-up and heartbeat state reporting
+- [ ] 2.1 (P) Wire `OPCode_walCatchupEntry` single-entry handler on standby
+  - Replace the stub in the dispatch switch (returns 0) with a call to `apply_wal_entry()` for single entries
+  - On success, advance `applied_lsn` atomically; on error, log ERROR and skip — do not crash
+  - Standby remains in `CTRL_STANDBY` throughout; does not accept job dispatch during catch-up
+  - `applied_lsn` is in-memory only; on restart standby reports `lsn=0` and re-bootstraps via heartbeat loop
+  - Unit test: apply a single WAL entry, verify `applied_lsn` advances; error entry is skipped, not crashed
+  - _Requirements: 3.1, 3.3, 3.4, 3.5_
+  - _Boundary: Standby Dispatch (cluster.c / ha.c)_
+
+- [ ] 2.2 Handle batch WAL catch-up entries on standby
+  - Extend the `OPCode_walCatchupEntry` handler from task 2.1 to decode the `entries` list for batch messages
+  - Apply each entry in order via `apply_wal_entry()`; advance `applied_lsn` only after the full batch is durably written
+  - Standby remains in `CTRL_STANDBY` during batch apply; `applied_lsn` not persisted to disk
+  - Unit test: apply a batch of 3 entries, verify `applied_lsn` advances by 3; partial batch (< max) applied correctly
+  - _Requirements: 5.3, 5.5_
+  - _Boundary: Standby Dispatch (cluster.c / ha.c)_
+  - _Depends: 1.2, 2.1_
+
+- [ ] 2.3 (P) Populate `stateReport` in heartbeat reply on standby
+  - In the `OPCode_nop` handler (`cluster.c`), replace `data_which = nodata` with `data_which = stateReport`
+  - Populate `appliedLsn` from `ha->applied_lsn` and `configTag` from `myconf->config_version_tag`
+  - Unit test: verify reply carries correct `applied_lsn` and `config_tag` after WAL catch-up
+  - _Requirements: 1.1, 1.3, 1.4_
+  - _Boundary: Standby Heartbeat (cluster.c)_
+  - _Depends: 1.1_
+
+---
+
+- [ ] 3. Primary-side: Unified heartbeat-driven replication decision loop
+- [ ] 3.1 Implement per-standby convergence tracking update
+  - On each heartbeat reply from a standby, extract `appliedLsn` and `configTag` from `stateReport`
+  - Store in `standby_reported_tag[idx]` and `standby_reported_lsn[idx]`
+  - Evaluate convergence: `reported_tag >= committed_tag AND reported_lsn >= committed_lsn`
+  - Set `standby_converged[idx] = true/false`; log INFO when a standby first transitions to converged
+  - Unit test: send heartbeat reply with `lsn=N`, verify `standby_reported_lsn[idx]=N` and convergence flag updated correctly
+  - _Requirements: 1.2, 2.2, 2.5_
+  - _Boundary: Primary Heartbeat Handler (ha.c)_
+  - _Depends: 1.1, 1.4_
+
+- [ ] 3.2 Implement unified decision loop in heartbeat reply processing
+  - After updating per-standby tracking, evaluate: tag stale → push config; `lsn < oldest_wal_lsn` → send snapshot; `lsn < committed_lsn` → stream WAL; both current → converged
+  - One action per heartbeat cycle per standby; `stream_wal_entries()` enqueues all batches non-blocking via `xce_request()` in a single invocation — never blocks vtaskmgr thread
+  - Log decision at INFO: which standby, which action triggered
+  - Unit test: decision function returns `PUSH_CONFIG` / `SEND_SNAPSHOT` / `STREAM_WAL` / `CONVERGED` for each representative `(tag, lsn)` input
+  - _Requirements: 2.2, 2.3, 2.4_
+  - _Boundary: Primary Heartbeat Handler (ha.c)_
+  - _Depends: 3.1_
+
+- [ ] 3.3 Trigger per-standby config push from decision loop
+  - When `standby_tag < committed_tag`, call `push_config_to_standbys()` targeting the specific standby
+  - Reuse existing config push path; trigger is now heartbeat decision loop instead of `handle_ctrlhello`
+  - After config push, primary skips further decisions for this standby until the next heartbeat reply confirms the updated tag
+  - Unit test: decision loop with stale tag triggers config push; subsequent heartbeat with updated tag does not re-push
+  - _Requirements: 1.4_
+  - _Boundary: Config Replication (config_repl.c)_
+  - _Depends: 3.2_
+
+- [ ] 3.4 Implement batch WAL streaming from decision loop
+  - When `standby_lsn < committed_lsn AND standby_lsn >= oldest_wal_lsn`, stream entries `[standby_lsn+1, committed_lsn]`
+  - Accumulate entries up to 64KB/256 entries per batch; enqueue all batches via `xce_request()` in a single non-blocking invocation — do not block vtaskmgr thread
+  - Send each batch as `OPCode_walCatchupEntry` with the `entries` list; partial final batch sent immediately (no wait for fill)
+  - Unit test: stream 500 entries, verify 2 batches sent; all entries applied in LSN order on standby
+  - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
+  - _Boundary: WAL Streaming (config_repl.c)_
+  - _Depends: 1.2, 3.2_
+
+- [ ] 3.5 Implement chunked snapshot transfer
+  - When `standby_lsn < oldest_wal_lsn`, call `persist_checkpoint()` then split snapshot into 64KB chunks
+  - Send each chunk via `OPCode_snapshotTransfer` with `{checkpointLsn, chunkIndex, chunkOffset, totalSize, isLast, data}`
+  - Standby writes chunks to `snap.part`; on `isLast=true`, verify `sum(chunk_sizes) == total_size`, atomic rename `snap.part → snap-{lsn}.bin`, load snapshot; `applied_lsn = checkpoint_lsn` (in-memory)
+  - Mid-transfer reconnect: standby discards `snap.part` on reconnect; primary restarts snapshot from chunk 0 on next heartbeat cycle
+  - Unit test: transfer 128KB snapshot in 2 chunks, verify standby loads and `applied_lsn = checkpoint_lsn`
+  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
+  - _Boundary: Snapshot Transfer (config_repl.c, persist.c)_
+  - _Depends: 1.3, 3.2_
+
+---
+
+- [ ] 4. Post-election safety: Safe commit point from vote replies
+- [ ] 4.1 Determine `safe_committed_lsn` from vote replies in `raft_cb_become_leader`
+  - `ha_elect_ctx_t.votes[]` is already populated by the vote reply handler (`peerLastLogLsn`, `peerConfigTag` per voter); this task reads those fields after winning election
+  - Compute `safe_committed_lsn` = max LSN held by a majority of voters (iterate all voters with `granted=true`)
+  - Compute `safe_config_tag` from `peerConfigTag` using the same majority logic
+  - Unit test: voters at LSNs `[99, 100]`, self at `100` → `safe_committed_lsn=100`; voters at `[50, 51]` → `safe_committed_lsn=51`
+  - _Requirements: 2.3_
+  - _Boundary: Controller FSM (ha.c)_
+
+- [ ] 4.2 Replace stale-ctrlhello truncation with vote-derived safe commit point
+  - Remove `persist_truncate_wal_above(last_known_committed_lsn)` from the become-leader path
+  - Replace with `persist_truncate_wal_above(safe_committed_lsn)` using the value from task 4.1
+  - Set `committed_lsn = safe_committed_lsn` after truncation
+  - Unit test: verify entry at `safe_committed_lsn` is NOT truncated; entry above it IS
+  - _Requirements: 2.3_
+  - _Boundary: Controller FSM (ha.c)_
+  - _Depends: 4.1_
+
+---
+
+- [ ] 5. Integration: Simplify ctrlhello and harden COMMITTING gate
+- [ ] 5.1 Reduce `handle_ctrlhello` to connection setup only
+  - Remove config push, snapshot check, and WAL streaming from `handle_ctrlhello`
+  - Keep: set `primary_ctrl_fd`, call `ha_on_heartbeat_received`, arm liveness watchdog (`ha_arm_heartbeat_timeout_vtask`)
+  - All replication decisions handled by the heartbeat decision loop (tasks 3.2–3.5)
+  - Unit test: call `handle_ctrlhello`, verify no `OPCode_replConfigPush` or `OPCode_walCatchupEntry` messages enqueued
+  - _Requirements: 1.1, 1.2_
+  - _Boundary: Connection Setup (config_repl.c)_
+
+- [ ] 5.2 Wire majority-convergence → PRIMARY transition
+  - In the heartbeat reply handler, after updating convergence tracking, count `standby_converged` and compare to majority threshold
+  - If `count >= ceil(n_peers/2)` and role is `CTRL_PRIMARY_COMMITTING`, promote to `CTRL_PRIMARY`
+  - Single-node: `n_peers=0`, majority threshold = 0 → satisfied immediately
+  - Log INFO: "majority converged (N/M standbys) → CTRL_PRIMARY"
+  - Unit test: 3-node cluster — 1/2 standbys converged → stays `COMMITTING`; 2/2 → `CTRL_PRIMARY`; single-node → immediate
+  - _Requirements: 2.3, 2.4, 6.1, 6.3, 6.5_
+  - _Boundary: Controller FSM (ha.c)_
+  - _Depends: 3.1_
+
+- [ ] 5.3 Extend committing timeout to degraded-mode safety net
+  - Change `committing_timeout_ms` default from `10000` to `60000`
+  - Timeout handler logs WARN with per-standby convergence state, promotes in degraded mode
+  - Majority-convergence path (task 5.2) is the PRIMARY promotion mechanism; timeout is the fallback
+  - WARN log includes: number of standbys converged and number still pending
+  - _Requirements: 6.4_
+  - _Boundary: Controller FSM (ha.c)_
+
+- [ ] 5.4 Add observability log statements
+  - WAL catch-up start: INFO with start/target LSN — covered by unit test in task 6.2
+  - Standby converged: INFO with identity, config_tag, applied_lsn — covered by unit test in task 6.1
+  - Snapshot transfer start/complete: INFO with checkpoint_lsn, size — covered by unit test in task 6.4
+  - Majority reached → CTRL_PRIMARY: INFO with converged count and quorum size — covered by unit test in task 5.2
+  - Committing timeout: WARN with per-standby state — covered by unit test in task 5.3
+  - All 5 log statements visible in test output; `meson test --suite controller` green
+  - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5_
+  - _Boundary: Logging (ha.c, config_repl.c)_
+
+---
+
+- [ ] 6. Validation: Tests and integration verification
+- [ ] 6.1 Unit tests for decision loop and convergence tracking
+  - Test: `(tag=0, lsn=0)` → PUSH_CONFIG; `(tag=T, lsn=0)` → STREAM_WAL; `(tag=T, lsn=N)` → CONVERGED
+  - Test: majority detection (1/2 converged → not majority; 2/2 → majority for 3-node)
+  - Test: heartbeat reply populates `stateReport` with correct values
+  - All tests pass under ASAN
+  - _Requirements: 1.1, 1.2, 2.2, 2.3, 2.4_
+
+- [ ] 6.2 Unit tests for WAL catch-up and batch handling
+  - Test: apply single entry, apply batch of 256, error entry skipped, `applied_lsn` advances correctly
+  - Test: batch size limit enforced; `applied_lsn` only advances after full batch written
+  - All tests pass under ASAN
+  - _Requirements: 3.1, 3.3, 3.4, 3.5, 5.3, 5.5_
+
+- [ ] 6.3 Unit tests for safe commit point from vote replies
+  - Test: voters at LSNs `[99, 100]`, self at `100` → `safe_committed_lsn=100`
+  - Test: voters at LSNs `[50, 51]`, self at `50` → `safe_committed_lsn=51` (majority has 51)
+  - Test: single node → `safe_committed_lsn = self.applied_lsn`
+  - All tests pass under ASAN
+  - _Requirements: 2.3_
+
+- [ ] 6.4 Unit tests for snapshot chunk transfer
+  - Test: 128KB snapshot in 2 chunks → standby receives, verifies total size, atomically renames, loads; `applied_lsn = checkpoint_lsn`
+  - Test: mid-transfer reconnect → standby discards `snap.part`, primary restarts from chunk 0 on reconnect
+  - Test: snapshot transfer only triggered when `standby_lsn < oldest_wal_lsn`
+  - All tests pass under ASAN
+  - _Requirements: 4.1, 4.2, 4.3, 4.4_
+
+- [ ] 6.5 Integration test: 3-node startup via majority convergence
+  - Requires task 5.3 (`committing_timeout_ms=60000`) deployed so timeout does not fire during test
+  - Deploy via `/xce-deploy-ha`; all 3 nodes start within 15s
+  - Verify primary enters `CTRL_PRIMARY` via majority-convergence INFO log (not committing timeout WARN)
+  - Verify standby with `config_tag=0` receives config push, then WAL, then reports converged
+  - Verify no degraded-mode WARN in logs
+  - _Requirements: 6.1, 6.3, 6.5, 7.4_
+
+- [ ] 6.6 Integration test: snapshot transfer for lagging standby
+  - Start cluster, let WAL accumulate, kill one standby
+  - Trim WAL on primary so `oldest_wal_lsn > standby's last applied_lsn`
+  - Restart standby; verify snapshot transfer triggered (INFO log: checkpoint_lsn, size) and completes
+  - Verify standby converges and enters `CTRL_STANDBY`
+  - _Requirements: 4.1, 4.2, 4.5_
+
+- [ ] 6.7 E2E test: full cluster lifecycle
+  - Deploy 3 nodes via `/xce-deploy-ha`; verify single PRIMARY, zero degraded mode
+  - Kill primary; verify standby promotes via RDHUP → election → majority convergence
+  - Verify promoted primary serves from committed state (no data loss)
+  - Full ASAN build: `meson test --suite controller` green
+  - _Requirements: 6.2, 6.3, 6.4_