myxce/.kiro/specs/ha-repl-hardening/tasks.md

# Implementation Plan: HA Replication Hardening

---

## Task Overview

7 requirements, 6 design components. Progression: schema first → standby-side handlers → primary-side decision loop → post-election safety → integration cleanup → validation.

---

- [x] 1. Foundation: Schema changes and state tracking fields
- [x] 1.1 Add `stateReport` group to `RequestReply.data` union in controller.capnp
  - Add `appliedLsn :UInt64 @15` and `configTag :UInt64 @16` fields inside a `stateReport :group` variant
  - Regenerate capnp bindings; verify compilation succeeds
  - Existing decode paths and opcode dispatch unaffected
  - Compilation succeeds and existing controller tests remain green
  - _Requirements: 1.1, 1.3_
  - _Boundary: Wire Protocol (controller.capnp)_

- [x] 1.2 Extend `WalCatchupEntry` for batch support in controller.capnp
  - Add `entries :List(WalCatchupEntry)` field for multiple entries per message
  - Regenerate capnp bindings; verify compilation
  - Compilation succeeds; single-entry decode path still works
  - _Requirements: 5.1_
  - _Boundary: Wire Protocol (controller.capnp)_

- [x] 1.3 Add snapshot chunk fields to capnp schema
  - Add `checkpointLsn`, `chunkIndex`, `chunkOffset`, `totalSize`, `isLast` fields to the snapshot transfer message
  - No resumption fields — restart-on-reconnect policy means no snapshot progress state in `stateReport`
  - Regenerate capnp bindings; verify compilation
  - Compilation succeeds
  - _Requirements: 4.1, 4.2_
  - _Boundary: Wire Protocol (controller.capnp)_

- [x] 1.4 Add per-standby convergence tracking fields to `ha_state_t`
  - Add `standby_reported_tag[MAX_HA_CANDIDATES]`, `standby_reported_lsn[MAX_HA_CANDIDATES]`, `standby_converged[MAX_HA_CANDIDATES]` arrays
  - Zero-initialize in `ha_init()`; these fields are read by the convergence evaluator in task 3.1
  - Unit test: verify fields are zero after `ha_init()` and writable from test context
  - _Requirements: 2.1_
  - _Boundary: Controller FSM (ha.h)_

---

- [x] 2. Standby-side: Wire WAL catch-up and heartbeat state reporting
- [x] 2.1 (P) Wire `OPCode_walCatchupEntry` single-entry handler on standby
  - Replace the stub in the dispatch switch (returns 0) with a call to `apply_wal_entry()` for single entries
  - On success, advance `applied_lsn` atomically; on error, log ERROR and skip — do not crash
  - Standby remains in `CTRL_STANDBY` throughout; does not accept job dispatch during catch-up
  - `applied_lsn` is in-memory only; on restart standby reports `lsn=0` and re-bootstraps via heartbeat loop
  - Unit test: apply a single WAL entry, verify `applied_lsn` advances; error entry is skipped, not crashed
  - _Requirements: 3.1, 3.3, 3.4, 3.5_
  - _Boundary: Standby Dispatch (cluster.c / ha.c)_

- [x] 2.2 Handle batch WAL catch-up entries on standby
  - Extend the `OPCode_walCatchupEntry` handler from task 2.1 to decode the `entries` list for batch messages
  - Apply each entry in order via `apply_wal_entry()`; advance `applied_lsn` only after the full batch is durably written
  - Standby remains in `CTRL_STANDBY` during batch apply; `applied_lsn` not persisted to disk
  - Unit test: apply a batch of 3 entries, verify `applied_lsn` advances by 3; partial batch (< max) applied correctly
  - _Requirements: 5.3, 5.5_
  - _Boundary: Standby Dispatch (cluster.c / ha.c)_
  - _Depends: 1.2, 2.1_

- [x] 2.3 (P) Populate `stateReport` in heartbeat reply on standby
  - In the `OPCode_nop` handler (`cluster.c`), replace `data_which = nodata` with `data_which = stateReport`
  - Populate `appliedLsn` from `ha->applied_lsn` and `configTag` from `myconf->config_version_tag`
  - Unit test: verify reply carries correct `applied_lsn` and `config_tag` after WAL catch-up
  - _Requirements: 1.1, 1.3, 1.4_
  - _Boundary: Standby Heartbeat (cluster.c)_
  - _Depends: 1.1_

---

- [x] 3. Primary-side: Unified heartbeat-driven replication decision loop
- [x] 3.1 Implement per-standby convergence tracking update
  - On each heartbeat reply from a standby, extract `appliedLsn` and `configTag` from `stateReport`
  - Store in `standby_reported_tag[idx]` and `standby_reported_lsn[idx]`
  - Evaluate convergence: `reported_tag >= committed_tag AND reported_lsn >= committed_lsn`
  - Set `standby_converged[idx] = true/false`; log INFO when a standby first transitions to converged
  - Unit test: send heartbeat reply with `lsn=N`, verify `standby_reported_lsn[idx]=N` and convergence flag updated correctly
  - _Requirements: 1.2, 2.2, 2.5_
  - _Boundary: Primary Heartbeat Handler (ha.c)_
  - _Depends: 1.1, 1.4_

- [x] 3.2 Implement unified decision loop in heartbeat reply processing
  - After updating per-standby tracking, evaluate: tag stale → push config; `lsn < oldest_wal_lsn` → send snapshot; `lsn < committed_lsn` → stream WAL; both current → converged
  - One action per heartbeat cycle per standby; `stream_wal_entries()` enqueues all batches non-blocking via `xce_request()` in a single invocation — never blocks vtaskmgr thread
  - Log decision at INFO: which standby, which action triggered
  - Unit test: decision function returns `PUSH_CONFIG` / `SEND_SNAPSHOT` / `STREAM_WAL` / `CONVERGED` for each representative `(tag, lsn)` input
  - _Requirements: 2.2, 2.3, 2.4_
  - _Boundary: Primary Heartbeat Handler (ha.c)_
  - _Depends: 3.1_

- [x] 3.3 Trigger per-standby config push from decision loop
  - When `standby_tag < committed_tag`, call `push_config_to_standbys()` targeting the specific standby
  - Reuse existing config push path; trigger is now heartbeat decision loop instead of `handle_ctrlhello`
  - After config push, primary skips further decisions for this standby until the next heartbeat reply confirms the updated tag
  - Unit test: decision loop with stale tag triggers config push; subsequent heartbeat with updated tag does not re-push
  - _Requirements: 1.4_
  - _Boundary: Config Replication (config_repl.c)_
  - _Depends: 3.2_

- [x] 3.4 Implement batch WAL streaming from decision loop
  - When `standby_lsn < committed_lsn AND standby_lsn >= oldest_wal_lsn`, stream entries `[standby_lsn+1, committed_lsn]`
  - Accumulate entries up to 64KB/256 entries per batch; enqueue all batches via `xce_request()` in a single non-blocking invocation — do not block vtaskmgr thread
  - Send each batch as `OPCode_walCatchupEntry` with the `entries` list; partial final batch sent immediately (no wait for fill)
  - Unit test: stream 500 entries, verify 2 batches sent; all entries applied in LSN order on standby
  - _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5_
  - _Boundary: WAL Streaming (config_repl.c)_
  - _Depends: 1.2, 3.2_

- [x] 3.5a Primary-side snapshot chunk sender (ha_send_snapshot_to_standby)
  - Read snapshot file, split into 64KB chunks, enqueue each via xce_request() with correct metadata
  - Chunk fields: snapshotLsn, chunkIndex, chunkOffset, totalSize, isLast; payload=raw_chunk_bytes (wired via xce_snapshot_chunk_req_raw, task 7.2)
  - Unit test: 128KB file → 2 chunks with correct offsets; small file → 1 chunk
  - _Requirements: 4.1, 4.2_
  - _Boundary: Snapshot Transfer (config_repl.c)_
  - _Depends: 1.3_

- [x] 3.5b Fix SEND_SNAPSHOT production wiring and oldest_wal_lsn
  - Use persist.checkpoint_lsn (not committed_lsn) for snapshot file lookup
  - Stat the snapshot file to get actual total_size (not hardcoded 131072)
  - Wire persist.oldest_wal_lsn into ha_evaluate_repl_action call site (remove the hardcoded 0)
  - Unit test: verify decision loop triggers SEND_SNAPSHOT when standby_lsn < oldest_wal_lsn
  - _Requirements: 4.1, 4.3_
  - _Boundary: Controller FSM (ha.c)_
  - _Depends: 3.5a, 3.2_

- [x] 3.5c Standby-side snapshot receive handler (OPCode_snapshotTransfer)
  - Add case OPCode_snapshotTransfer in myreq_handler dispatch (cluster.c)
  - Write each chunk payload to snap.part temp file; on isLast=true, verify total size, atomic rename to snap-{lsn}.bin, load via load_snapshot()
  - Set applied_lsn = checkpoint_lsn (in-memory) after successful load
  - On mid-transfer reconnect: discard snap.part (restart from chunk 0 on next reconnect)
  - Unit test: receive 2 chunks, verify file assembled and renamed correctly
  - _Requirements: 4.2, 4.3, 4.4_
  - _Boundary: Standby Dispatch (cluster.c, ha.c)_
  - _Depends: 1.3, 3.5a_

- [x] 3.5d Snapshot transfer integration and dispatch-path tests
  - Test: SEND_SNAPSHOT dispatch path triggers hook (like PUSH_CONFIG/STREAM_WAL tests in test_ha_role.c)
  - Test: standby receive → snap.part assembled → isLast rename → applied_lsn set
  - Test: reconnected standby discards snap.part (restart from chunk 0)
  - All tests pass under ASAN
  - _Requirements: 4.3, 4.4, 4.5_
  - _Boundary: Integration (ha.c, cluster.c, test_ha_role.c, test_config_repl.c)_
  - _Depends: 3.5b, 3.5c_

---

- [x] 4. Post-election safety: Safe commit point from vote replies
- [x] 4.1 Determine `safe_committed_lsn` from vote replies in `raft_cb_become_leader`
  - `ha_elect_ctx_t.votes[]` is already populated by the vote reply handler (`peerLastLogLsn`, `peerConfigTag` per voter); this task reads those fields after winning election
  - Compute `safe_committed_lsn` = max LSN held by a majority of voters (iterate all voters with `granted=true`)
  - Compute `safe_config_tag` from `peerConfigTag` using the same majority logic
  - Unit test: voters at LSNs `[99, 100]`, self at `100` → `safe_committed_lsn=100`; voters at `[50, 51]` → `safe_committed_lsn=51`
  - _Requirements: 2.3_
  - _Boundary: Controller FSM (ha.c)_

- [x] 4.2 Replace stale-ctrlhello truncation with vote-derived safe commit point
  - Remove `persist_truncate_wal_above(last_known_committed_lsn)` from the become-leader path
  - Replace with `persist_truncate_wal_above(safe_committed_lsn)` using the value from task 4.1
  - Set `committed_lsn = safe_committed_lsn` after truncation
  - Unit test: verify entry at `safe_committed_lsn` is NOT truncated; entry above it IS
  - _Requirements: 2.3_
  - _Boundary: Controller FSM (ha.c)_
  - _Depends: 4.1_

---

- [x] 5. Integration: Simplify ctrlhello and harden COMMITTING gate
- [x] 5.1 Reduce `handle_ctrlhello` to connection setup only
  - Remove config push, snapshot check, and WAL streaming from `handle_ctrlhello`
  - Keep: set `primary_ctrl_fd`, call `ha_on_heartbeat_received`, arm liveness watchdog (`ha_arm_heartbeat_timeout_vtask`)
  - All replication decisions handled by the heartbeat decision loop (tasks 3.2–3.5)
  - Unit test: call `handle_ctrlhello`, verify no `OPCode_replConfigPush` or `OPCode_walCatchupEntry` messages enqueued
  - _Requirements: 1.1, 1.2_
  - _Boundary: Connection Setup (config_repl.c)_

- [x] 5.2 Wire majority-convergence → PRIMARY transition
  - In the heartbeat reply handler, after updating convergence tracking, count `standby_converged` and compare to majority threshold
  - Threshold: `count >= n_peers / 2` (C integer division — for odd n the primary +
    converged standbys form a cluster majority; n=3→1, n=5→2, n=1→0).
    Raft rejects even n_peers > 1. *(reconciled 2026-06-09, task 7.5)*
    promote to `CTRL_PRIMARY`  *(formula reconciled 2026-06-09, task 7.5)*
  - Single-node: `n_peers <= 1`, threshold = 0 — satisfied immediately
  - Log INFO: "majority converged (N/M standbys, threshold T) — CTRL_PRIMARY"
  - Unit test: 3-node — 0/2 → COMMITTING; 1/2 → PRIMARY (majority); 5-node — 1/4 → COMMITTING; 2/4 → PRIMARY; single-node → immediate
  - _Requirements: 2.3, 2.4, 6.1, 6.3, 6.5_
  - _Boundary: Controller FSM (ha.c)_
  - _Depends: 3.1_

- [x] 5.3 Extend committing timeout to degraded-mode safety net
  - Change `committing_timeout_ms` default from `10000` to `60000`
  - Timeout handler logs WARN with per-standby convergence state, promotes in degraded mode
  - Majority-convergence path (task 5.2) is the PRIMARY promotion mechanism; timeout is the fallback
  - WARN log includes: number of standbys converged and number still pending
  - _Requirements: 6.4_
  - _Boundary: Controller FSM (ha.c)_

- [x] 5.4 Add observability log statements
  - WAL catch-up start: INFO with start/target LSN — covered by unit test in task 6.2
  - Standby converged: INFO with identity, config_tag, applied_lsn — covered by unit test in task 6.1
  - Snapshot transfer start/complete: INFO with checkpoint_lsn, size — covered by unit test in task 6.4
  - Majority reached → CTRL_PRIMARY: INFO with converged count and quorum size — covered by unit test in task 5.2
  - Committing timeout: WARN with per-standby state — covered by unit test in task 5.3
  - All 5 log statements visible in test output; `meson test --suite controller` green
  - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5_
  - _Boundary: Logging (ha.c, config_repl.c)_

---

- [x] 6. Validation: Tests and integration verification
- [x] 6.1 Unit tests for decision loop and convergence tracking
  - Test: `(tag=0, lsn=0)` → PUSH_CONFIG; `(tag=T, lsn=0)` → STREAM_WAL; `(tag=T, lsn=N)` → CONVERGED
  - Test: majority detection (0/2 → not majority; 1/2 → majority for 3-node; per n_peers/2 formula)
  - Test: heartbeat reply populates `stateReport` with correct values
  - All tests pass under ASAN
  - _Requirements: 1.1, 1.2, 2.2, 2.3, 2.4_

- [x] 6.2 Unit tests for WAL catch-up and batch handling
  - Test: apply single entry, apply batch of 256, error entry skipped, `applied_lsn` advances correctly
  - Test: batch size limit enforced; `applied_lsn` only advances after full batch written
  - All tests pass under ASAN
  - _Requirements: 3.1, 3.3, 3.4, 3.5, 5.3, 5.5_

- [x] 6.3 Unit tests for safe commit point from vote replies
  - Test: voters at LSNs `[99, 100]`, self at `100` → `safe_committed_lsn=100`
  - Test: voters at LSNs `[50, 51]`, self at `50` → `safe_committed_lsn=51` (majority has 51)
  - Test: single node → `safe_committed_lsn = self.applied_lsn`
  - All tests pass under ASAN
  - _Requirements: 2.3_

- [x] 6.4 Unit tests for snapshot chunk transfer
  - Test: 128KB snapshot in 2 chunks → standby receives, verifies total size, atomically renames, loads; `applied_lsn = checkpoint_lsn`
  - Test: mid-transfer reconnect → standby discards `snap.part`, primary restarts from chunk 0 on reconnect
  - Test: snapshot transfer only triggered when `standby_lsn < oldest_wal_lsn`
  - All tests pass under ASAN
  - _Requirements: 4.1, 4.2, 4.3, 4.4_

- [x] 6.5 Integration test: 3-node startup via majority convergence
  - Requires task 5.3 (`committing_timeout_ms=60000`) deployed so timeout does not fire during test
  - Deploy via `/xce-deploy-ha`; all 3 nodes start within 15s
  - Verify primary enters `CTRL_PRIMARY` via majority-convergence INFO log (not committing timeout WARN)
  - Verify standby with `config_tag=0` receives config push, then WAL, then reports converged
  - Verify no degraded-mode WARN in logs
  - _Requirements: 6.1, 6.3, 6.5, 7.4_

- [x] 6.6 Integration test: snapshot transfer for lagging standby
  - Start cluster, let WAL accumulate, kill one standby
  - Trim WAL on primary so `oldest_wal_lsn > standby's last applied_lsn`
  - Restart standby; verify snapshot transfer triggered (INFO log: checkpoint_lsn, size) and completes
  - Verify standby converges and enters `CTRL_STANDBY`
  - _Requirements: 4.1, 4.2, 4.5_

- [x] 6.7 E2E test: full cluster lifecycle
  - Deploy 3 nodes via `/xce-deploy-ha`; verify single PRIMARY, zero degraded mode
  - Kill primary; verify standby promotes via RDHUP → election → majority convergence
  - Verify promoted primary serves from committed state (no data loss)
  - Full ASAN build: `meson test --suite controller` green
  - _Requirements: 6.2, 6.3, 6.4_

---

## Reopened 2026-06-09 — Gap Analysis Validation Pass

The 2026-06-09 gap analysis (research.md) found the replication **data plane missing**:
snapshot chunks ship with `payload = NULL` (config_repl.c "encoding deferred"; cluster.c)
and WAL catch-up applies entries header-only (`payload_len=0`), so `apply_wal_entry`
advances LSN without restoring entities. Standbys report "converged" without holding the
state. Tests 6.5–6.7 pass because they assert LSN convergence only. The spec MUST NOT be
marked `implemented` until this series lands.

- [x] 7. Replication payload data plane
- [x] 7.1 Define the payload wire format: capnp `Data` field + typemap codegen for WAL
  entries and snapshot chunks. Coordinate with ha-config-bootstrap task 7.6 — live
  `repl_broadcast` is also a no-send stub and must share the same format.
  - _Requirements: 3.1, 4.2, 5.3_
  - _Boundary: Wire format (controller.capnp, config_repl.c)_
- [x] 7.2 Snapshot chunk payload transmission (primary) + chunk write to `snap.part`
  (standby); verify total size before atomic rename (phased option: hand-wire raw bytes
  first if typemap codegen is blocked)
  - _Requirements: 4.1, 4.2, 4.3_
  - _Boundary: Snapshot transfer (config_repl.c, cluster.c)_
- [x] 7.3 WAL catch-up entry payload transmission + entity restoration in
  `apply_wal_entry` (e.g. `WAL_JOB_SUBMIT` recreates the job); `applied_lsn` advances
  only after the entity is applied
  - _Requirements: 3.1, 3.4, 5.3_
  - _Boundary: WAL catch-up (ha.c, config_repl.c, persist.c)_
- [x] 7.4 Re-zero convergence tracking arrays (`standby_reported_*`,
  `standby_converged`) on entry to CTRL_PRIMARY_COMMITTING — stale convergence from a
  prior term must not satisfy the gate after re-election
  - _Requirements: 2.1_
  - _Boundary: Convergence tracking (ha.c)_
- [x] 7.5 Reconcile the convergence threshold: code requires ALL standbys (`n_peers−1`),
  requirements say `ceil(n/2)−1`, tasks say `ceil(n/2)`, code was `n_peers−1` — reconciled to `n_peers / 2` (C integer division), recorded in
  Decision Log 2026-06-09
  - _Requirements: 2.4, 6.1_
  - _Boundary: Quorum policy (ha.c, design.md)_
- [x] 7.6 Production `unlink(snap.part)` on mid-transfer reconnect (Req 4.4) and fresh
  checkpoint creation at snapshot start instead of reusing the old checkpoint (Req 4.1)
  - _Requirements: 4.1, 4.4_
  - _Boundary: Snapshot lifecycle (config_repl.c)_
  - _Requirements: 6.2, 6.4_
  - _Boundary: Integration test (deploy-ha)_