myxce/.kiro/specs/ha-repl-hardening/tasks.md

Implementation Plan: HA Replication Hardening

---

Task Overview

7 requirements, 6 design components. Progression: schema first → standby-side handlers → primary-side decision loop → post-election safety → integration cleanup → validation.

---

• 1. Foundation: Schema changes and state tracking fields

• 1.1 Add stateReport group to RequestReply.data union in controller.capnp

  • Add appliedLsn :UInt64 @15 and configTag :UInt64 @16 fields inside a stateReport :group variant
  • Regenerate capnp bindings; verify compilation succeeds
  • Existing decode paths and opcode dispatch unaffected
  • Compilation succeeds and existing controller tests remain green
  • Requirements: 1.1, 1.3
  • Boundary: Wire Protocol (controller.capnp)

• 1.2 Extend WalCatchupEntry for batch support in controller.capnp

  • Add entries :List(WalCatchupEntry) field for multiple entries per message
  • Regenerate capnp bindings; verify compilation
  • Compilation succeeds; single-entry decode path still works
  • Requirements: 5.1
  • Boundary: Wire Protocol (controller.capnp)

• 1.3 Add snapshot chunk fields to capnp schema

  • Add checkpointLsn, chunkIndex, chunkOffset, totalSize, isLast fields to the snapshot transfer message
  • No resumption fields — restart-on-reconnect policy means no snapshot progress state in stateReport
  • Regenerate capnp bindings; verify compilation
  • Compilation succeeds
  • Requirements: 4.1, 4.2
  • Boundary: Wire Protocol (controller.capnp)

• 1.4 Add per-standby convergence tracking fields to ha_state_t

  • Add standby_reported_tag[MAX_HA_CANDIDATES], standby_reported_lsn[MAX_HA_CANDIDATES], standby_converged[MAX_HA_CANDIDATES] arrays
  • Zero-initialize in ha_init(); these fields are read by the convergence evaluator in task 3.1
  • Unit test: verify fields are zero after ha_init() and writable from test context
  • Requirements: 2.1
  • Boundary: Controller FSM (ha.h)

---

• 2. Standby-side: Wire WAL catch-up and heartbeat state reporting

• 2.1 (P) Wire OPCode_walCatchupEntry single-entry handler on standby

  • Replace the stub in the dispatch switch (returns 0) with a call to apply_wal_entry() for single entries
  • On success, advance applied_lsn atomically; on error, log ERROR and skip — do not crash
  • Standby remains in CTRL_STANDBY throughout; does not accept job dispatch during catch-up
  • applied_lsn is in-memory only; on restart standby reports lsn=0 and re-bootstraps via heartbeat loop
  • Unit test: apply a single WAL entry, verify applied_lsn advances; error entry is skipped, not crashed
  • Requirements: 3.1, 3.3, 3.4, 3.5
  • Boundary: Standby Dispatch (cluster.c / ha.c)

• 2.2 Handle batch WAL catch-up entries on standby

  • Extend the OPCode_walCatchupEntry handler from task 2.1 to decode the entries list for batch messages
  • Apply each entry in order via apply_wal_entry(); advance applied_lsn only after the full batch is durably written
  • Standby remains in CTRL_STANDBY during batch apply; applied_lsn not persisted to disk
  • Unit test: apply a batch of 3 entries, verify applied_lsn advances by 3; partial batch (< max) applied correctly
  • Requirements: 5.3, 5.5
  • Boundary: Standby Dispatch (cluster.c / ha.c)
  • Depends: 1.2, 2.1

• 2.3 (P) Populate stateReport in heartbeat reply on standby

  • In the OPCode_nop handler (cluster.c), replace data_which = nodata with data_which = stateReport
  • Populate appliedLsn from ha->applied_lsn and configTag from myconf->config_version_tag
  • Unit test: verify reply carries correct applied_lsn and config_tag after WAL catch-up
  • Requirements: 1.1, 1.3, 1.4
  • Boundary: Standby Heartbeat (cluster.c)
  • Depends: 1.1

---

• 3. Primary-side: Unified heartbeat-driven replication decision loop

• 3.1 Implement per-standby convergence tracking update

  • On each heartbeat reply from a standby, extract appliedLsn and configTag from stateReport
  • Store in standby_reported_tag[idx] and standby_reported_lsn[idx]
  • Evaluate convergence: reported_tag >= committed_tag AND reported_lsn >= committed_lsn
  • Set standby_converged[idx] = true/false; log INFO when a standby first transitions to converged
  • Unit test: send heartbeat reply with lsn=N, verify standby_reported_lsn[idx]=N and convergence flag updated correctly
  • Requirements: 1.2, 2.2, 2.5
  • Boundary: Primary Heartbeat Handler (ha.c)
  • Depends: 1.1, 1.4

• 3.2 Implement unified decision loop in heartbeat reply processing

  • After updating per-standby tracking, evaluate: tag stale → push config; lsn < oldest_wal_lsn → send snapshot; lsn < committed_lsn → stream WAL; both current → converged
  • One action per heartbeat cycle per standby; stream_wal_entries() enqueues all batches non-blocking via xce_request() in a single invocation — never blocks vtaskmgr thread
  • Log decision at INFO: which standby, which action triggered
  • Unit test: decision function returns PUSH_CONFIG / SEND_SNAPSHOT / STREAM_WAL / CONVERGED for each representative (tag, lsn) input
  • Requirements: 2.2, 2.3, 2.4
  • Boundary: Primary Heartbeat Handler (ha.c)
  • Depends: 3.1

• 3.3 Trigger per-standby config push from decision loop

  • When standby_tag < committed_tag, call push_config_to_standbys() targeting the specific standby
  • Reuse existing config push path; trigger is now heartbeat decision loop instead of handle_ctrlhello
  • After config push, primary skips further decisions for this standby until the next heartbeat reply confirms the updated tag
  • Unit test: decision loop with stale tag triggers config push; subsequent heartbeat with updated tag does not re-push
  • Requirements: 1.4
  • Boundary: Config Replication (config_repl.c)
  • Depends: 3.2

• 3.4 Implement batch WAL streaming from decision loop

  • When standby_lsn < committed_lsn AND standby_lsn >= oldest_wal_lsn, stream entries [standby_lsn+1, committed_lsn]
  • Accumulate entries up to 64KB/256 entries per batch; enqueue all batches via xce_request() in a single non-blocking invocation — do not block vtaskmgr thread
  • Send each batch as OPCode_walCatchupEntry with the entries list; partial final batch sent immediately (no wait for fill)
  • Unit test: stream 500 entries, verify 2 batches sent; all entries applied in LSN order on standby
  • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5
  • Boundary: WAL Streaming (config_repl.c)
  • Depends: 1.2, 3.2

• 3.5a Primary-side snapshot chunk sender (ha_send_snapshot_to_standby)

  • Read snapshot file, split into 64KB chunks, enqueue each via xce_request() with correct metadata
  • Chunk fields: snapshotLsn, chunkIndex, chunkOffset, totalSize, isLast; payload=NULL (encoding deferred)
  • Unit test: 128KB file → 2 chunks with correct offsets; small file → 1 chunk
  • Requirements: 4.1, 4.2
  • Boundary: Snapshot Transfer (config_repl.c)
  • Depends: 1.3

• 3.5b Fix SEND_SNAPSHOT production wiring and oldest_wal_lsn

  • Use persist.checkpoint_lsn (not committed_lsn) for snapshot file lookup
  • Stat the snapshot file to get actual total_size (not hardcoded 131072)
  • Wire persist.oldest_wal_lsn into ha_evaluate_repl_action call site (remove the hardcoded 0)
  • Unit test: verify decision loop triggers SEND_SNAPSHOT when standby_lsn < oldest_wal_lsn
  • Requirements: 4.1, 4.3
  • Boundary: Controller FSM (ha.c)
  • Depends: 3.5a, 3.2

• 3.5c Standby-side snapshot receive handler (OPCode_snapshotTransfer)

  • Add case OPCode_snapshotTransfer in myreq_handler dispatch (cluster.c)
  • Write each chunk payload to snap.part temp file; on isLast=true, verify total size, atomic rename to snap-{lsn}.bin, load via load_snapshot()
  • Set applied_lsn = checkpoint_lsn (in-memory) after successful load
  • On mid-transfer reconnect: discard snap.part (restart from chunk 0 on next reconnect)
  • Unit test: receive 2 chunks, verify file assembled and renamed correctly
  • Requirements: 4.2, 4.3, 4.4
  • Boundary: Standby Dispatch (cluster.c, ha.c)
  • Depends: 1.3, 3.5a

• 3.5d Snapshot transfer integration and dispatch-path tests

  • Test: SEND_SNAPSHOT dispatch path triggers hook (like PUSH_CONFIG/STREAM_WAL tests in test_ha_role.c)
  • Test: standby receive → snap.part assembled → isLast rename → applied_lsn set
  • Test: reconnected standby discards snap.part (restart from chunk 0)
  • All tests pass under ASAN
  • Requirements: 4.3, 4.4, 4.5
  • Boundary: Integration (ha.c, cluster.c, test_ha_role.c, test_config_repl.c)
  • Depends: 3.5b, 3.5c

---

• 4. Post-election safety: Safe commit point from vote replies

• 4.1 Determine safe_committed_lsn from vote replies in raft_cb_become_leader

  • ha_elect_ctx_t.votes[] is already populated by the vote reply handler (peerLastLogLsn, peerConfigTag per voter); this task reads those fields after winning election
  • Compute safe_committed_lsn = max LSN held by a majority of voters (iterate all voters with granted=true)
  • Compute safe_config_tag from peerConfigTag using the same majority logic
  • Unit test: voters at LSNs [99, 100], self at 100 → safe_committed_lsn=100; voters at [50, 51] → safe_committed_lsn=51
  • Requirements: 2.3
  • Boundary: Controller FSM (ha.c)

• 4.2 Replace stale-ctrlhello truncation with vote-derived safe commit point

  • Remove persist_truncate_wal_above(last_known_committed_lsn) from the become-leader path
  • Replace with persist_truncate_wal_above(safe_committed_lsn) using the value from task 4.1
  • Set committed_lsn = safe_committed_lsn after truncation
  • Unit test: verify entry at safe_committed_lsn is NOT truncated; entry above it IS
  • Requirements: 2.3
  • Boundary: Controller FSM (ha.c)
  • Depends: 4.1

---

• 5. Integration: Simplify ctrlhello and harden COMMITTING gate

• 5.1 Reduce handle_ctrlhello to connection setup only

  • Remove config push, snapshot check, and WAL streaming from handle_ctrlhello
  • Keep: set primary_ctrl_fd, call ha_on_heartbeat_received, arm liveness watchdog (ha_arm_heartbeat_timeout_vtask)
  • All replication decisions handled by the heartbeat decision loop (tasks 3.2–3.5)
  • Unit test: call handle_ctrlhello, verify no OPCode_replConfigPush or OPCode_walCatchupEntry messages enqueued
  • Requirements: 1.1, 1.2
  • Boundary: Connection Setup (config_repl.c)

• 5.2 Wire majority-convergence → PRIMARY transition

  • In the heartbeat reply handler, after updating convergence tracking, count standby_converged and compare to majority threshold
  • If count >= ceil(n_peers/2) and role is CTRL_PRIMARY_COMMITTING, promote to CTRL_PRIMARY
  • Single-node: n_peers=0, majority threshold = 0 → satisfied immediately
  • Log INFO: "majority converged (N/M standbys) → CTRL_PRIMARY"
  • Unit test: 3-node cluster — 1/2 standbys converged → stays COMMITTING; 2/2 → CTRL_PRIMARY; single-node → immediate
  • Requirements: 2.3, 2.4, 6.1, 6.3, 6.5
  • Boundary: Controller FSM (ha.c)
  • Depends: 3.1

• 5.3 Extend committing timeout to degraded-mode safety net

  • Change committing_timeout_ms default from 10000 to 60000
  • Timeout handler logs WARN with per-standby convergence state, promotes in degraded mode
  • Majority-convergence path (task 5.2) is the PRIMARY promotion mechanism; timeout is the fallback
  • WARN log includes: number of standbys converged and number still pending
  • Requirements: 6.4
  • Boundary: Controller FSM (ha.c)

• 5.4 Add observability log statements

  • WAL catch-up start: INFO with start/target LSN — covered by unit test in task 6.2
  • Standby converged: INFO with identity, config_tag, applied_lsn — covered by unit test in task 6.1
  • Snapshot transfer start/complete: INFO with checkpoint_lsn, size — covered by unit test in task 6.4
  • Majority reached → CTRL_PRIMARY: INFO with converged count and quorum size — covered by unit test in task 5.2
  • Committing timeout: WARN with per-standby state — covered by unit test in task 5.3
  • All 5 log statements visible in test output; meson test --suite controller green
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5
  • Boundary: Logging (ha.c, config_repl.c)

---

• 6. Validation: Tests and integration verification

• 6.1 Unit tests for decision loop and convergence tracking

  • Test: (tag=0, lsn=0) → PUSH_CONFIG; (tag=T, lsn=0) → STREAM_WAL; (tag=T, lsn=N) → CONVERGED
  • Test: majority detection (1/2 converged → not majority; 2/2 → majority for 3-node)
  • Test: heartbeat reply populates stateReport with correct values
  • All tests pass under ASAN
  • Requirements: 1.1, 1.2, 2.2, 2.3, 2.4

• 6.2 Unit tests for WAL catch-up and batch handling

  • Test: apply single entry, apply batch of 256, error entry skipped, applied_lsn advances correctly
  • Test: batch size limit enforced; applied_lsn only advances after full batch written
  • All tests pass under ASAN
  • Requirements: 3.1, 3.3, 3.4, 3.5, 5.3, 5.5

• 6.3 Unit tests for safe commit point from vote replies

  • Test: voters at LSNs [99, 100], self at 100 → safe_committed_lsn=100
  • Test: voters at LSNs [50, 51], self at 50 → safe_committed_lsn=51 (majority has 51)
  • Test: single node → safe_committed_lsn = self.applied_lsn
  • All tests pass under ASAN
  • Requirements: 2.3

• 6.4 Unit tests for snapshot chunk transfer

  • Test: 128KB snapshot in 2 chunks → standby receives, verifies total size, atomically renames, loads; applied_lsn = checkpoint_lsn
  • Test: mid-transfer reconnect → standby discards snap.part, primary restarts from chunk 0 on reconnect
  • Test: snapshot transfer only triggered when standby_lsn < oldest_wal_lsn
  • All tests pass under ASAN
  • Requirements: 4.1, 4.2, 4.3, 4.4

• 6.5 Integration test: 3-node startup via majority convergence

  • Requires task 5.3 (committing_timeout_ms=60000) deployed so timeout does not fire during test
  • Deploy via /xce-deploy-ha; all 3 nodes start within 15s
  • Verify primary enters CTRL_PRIMARY via majority-convergence INFO log (not committing timeout WARN)
  • Verify standby with config_tag=0 receives config push, then WAL, then reports converged
  • Verify no degraded-mode WARN in logs
  • Requirements: 6.1, 6.3, 6.5, 7.4

• 6.6 Integration test: snapshot transfer for lagging standby

  • Start cluster, let WAL accumulate, kill one standby
  • Trim WAL on primary so oldest_wal_lsn > standby's last applied_lsn
  • Restart standby; verify snapshot transfer triggered (INFO log: checkpoint_lsn, size) and completes
  • Verify standby converges and enters CTRL_STANDBY
  • Requirements: 4.1, 4.2, 4.5

• 6.7 E2E test: full cluster lifecycle

  • Deploy 3 nodes via /xce-deploy-ha; verify single PRIMARY, zero degraded mode
  • Kill primary; verify standby promotes via RDHUP → election → majority convergence
  • Verify promoted primary serves from committed state (no data loss)
  • Full ASAN build: meson test --suite controller green
  • Requirements: 6.2, 6.3, 6.4