myxce/.kiro/specs/ha-repl-hardening/tasks.md

Implementation Plan: HA Replication Hardening

---

Task Overview

7 requirements, 6 design components. Progression: schema first → standby-side handlers → primary-side decision loop → post-election safety → integration cleanup → validation.

---

• 1. Foundation: Schema changes and state tracking fields

• 1.1 Add stateReport group to RequestReply.data union in controller.capnp

  • Add appliedLsn :UInt64 @15 and configTag :UInt64 @16 fields inside a stateReport :group variant
  • Regenerate capnp bindings; verify compilation succeeds
  • Existing decode paths and opcode dispatch unaffected
  • Compilation succeeds and existing controller tests remain green
  • Requirements: 1.1, 1.3
  • Boundary: Wire Protocol (controller.capnp)

• 1.2 Extend WalCatchupEntry for batch support in controller.capnp

  • Add entries :List(WalCatchupEntry) field for multiple entries per message
  • Regenerate capnp bindings; verify compilation
  • Compilation succeeds; single-entry decode path still works
  • Requirements: 5.1
  • Boundary: Wire Protocol (controller.capnp)

• 1.3 Add snapshot chunk fields to capnp schema

  • Add checkpointLsn, chunkIndex, chunkOffset, totalSize, isLast fields to the snapshot transfer message
  • No resumption fields — restart-on-reconnect policy means no snapshot progress state in stateReport
  • Regenerate capnp bindings; verify compilation
  • Compilation succeeds
  • Requirements: 4.1, 4.2
  • Boundary: Wire Protocol (controller.capnp)

• 1.4 Add per-standby convergence tracking fields to ha_state_t

  • Add standby_reported_tag[MAX_HA_CANDIDATES], standby_reported_lsn[MAX_HA_CANDIDATES], standby_converged[MAX_HA_CANDIDATES] arrays
  • Zero-initialize in ha_init(); these fields are read by the convergence evaluator in task 3.1
  • Unit test: verify fields are zero after ha_init() and writable from test context
  • Requirements: 2.1
  • Boundary: Controller FSM (ha.h)

---

• 2. Standby-side: Wire WAL catch-up and heartbeat state reporting

• 2.1 (P) Wire OPCode_walCatchupEntry single-entry handler on standby

  • Replace the stub in the dispatch switch (returns 0) with a call to apply_wal_entry() for single entries
  • On success, advance applied_lsn atomically; on error, log ERROR and skip — do not crash
  • Standby remains in CTRL_STANDBY throughout; does not accept job dispatch during catch-up
  • applied_lsn is in-memory only; on restart standby reports lsn=0 and re-bootstraps via heartbeat loop
  • Unit test: apply a single WAL entry, verify applied_lsn advances; error entry is skipped, not crashed
  • Requirements: 3.1, 3.3, 3.4, 3.5
  • Boundary: Standby Dispatch (cluster.c / ha.c)

• 2.2 Handle batch WAL catch-up entries on standby

  • Extend the OPCode_walCatchupEntry handler from task 2.1 to decode the entries list for batch messages
  • Apply each entry in order via apply_wal_entry(); advance applied_lsn only after the full batch is durably written
  • Standby remains in CTRL_STANDBY during batch apply; applied_lsn not persisted to disk
  • Unit test: apply a batch of 3 entries, verify applied_lsn advances by 3; partial batch (< max) applied correctly
  • Requirements: 5.3, 5.5
  • Boundary: Standby Dispatch (cluster.c / ha.c)
  • Depends: 1.2, 2.1

• 2.3 (P) Populate stateReport in heartbeat reply on standby

  • In the OPCode_nop handler (cluster.c), replace data_which = nodata with data_which = stateReport
  • Populate appliedLsn from ha->applied_lsn and configTag from myconf->config_version_tag
  • Unit test: verify reply carries correct applied_lsn and config_tag after WAL catch-up
  • Requirements: 1.1, 1.3, 1.4
  • Boundary: Standby Heartbeat (cluster.c)
  • Depends: 1.1

---

• 3. Primary-side: Unified heartbeat-driven replication decision loop

• 3.1 Implement per-standby convergence tracking update

  • On each heartbeat reply from a standby, extract appliedLsn and configTag from stateReport
  • Store in standby_reported_tag[idx] and standby_reported_lsn[idx]
  • Evaluate convergence: reported_tag >= committed_tag AND reported_lsn >= committed_lsn
  • Set standby_converged[idx] = true/false; log INFO when a standby first transitions to converged
  • Unit test: send heartbeat reply with lsn=N, verify standby_reported_lsn[idx]=N and convergence flag updated correctly
  • Requirements: 1.2, 2.2, 2.5
  • Boundary: Primary Heartbeat Handler (ha.c)
  • Depends: 1.1, 1.4

• 3.2 Implement unified decision loop in heartbeat reply processing

  • After updating per-standby tracking, evaluate: tag stale → push config; lsn < oldest_wal_lsn → send snapshot; lsn < committed_lsn → stream WAL; both current → converged
  • One action per heartbeat cycle per standby; stream_wal_entries() enqueues all batches non-blocking via xce_request() in a single invocation — never blocks vtaskmgr thread
  • Log decision at INFO: which standby, which action triggered
  • Unit test: decision function returns PUSH_CONFIG / SEND_SNAPSHOT / STREAM_WAL / CONVERGED for each representative (tag, lsn) input
  • Requirements: 2.2, 2.3, 2.4
  • Boundary: Primary Heartbeat Handler (ha.c)
  • Depends: 3.1

• 3.3 Trigger per-standby config push from decision loop

  • When standby_tag < committed_tag, call push_config_to_standbys() targeting the specific standby
  • Reuse existing config push path; trigger is now heartbeat decision loop instead of handle_ctrlhello
  • After config push, primary skips further decisions for this standby until the next heartbeat reply confirms the updated tag
  • Unit test: decision loop with stale tag triggers config push; subsequent heartbeat with updated tag does not re-push
  • Requirements: 1.4
  • Boundary: Config Replication (config_repl.c)
  • Depends: 3.2

• 3.4 Implement batch WAL streaming from decision loop

  • When standby_lsn < committed_lsn AND standby_lsn >= oldest_wal_lsn, stream entries [standby_lsn+1, committed_lsn]
  • Accumulate entries up to 64KB/256 entries per batch; enqueue all batches via xce_request() in a single non-blocking invocation — do not block vtaskmgr thread
  • Send each batch as OPCode_walCatchupEntry with the entries list; partial final batch sent immediately (no wait for fill)
  • Unit test: stream 500 entries, verify 2 batches sent; all entries applied in LSN order on standby
  • Requirements: 5.1, 5.2, 5.3, 5.4, 5.5
  • Boundary: WAL Streaming (config_repl.c)
  • Depends: 1.2, 3.2

• 3.5a Primary-side snapshot chunk sender (ha_send_snapshot_to_standby)

  • Read snapshot file, split into 64KB chunks, enqueue each via xce_request() with correct metadata
  • Chunk fields: snapshotLsn, chunkIndex, chunkOffset, totalSize, isLast; payload=raw_chunk_bytes (wired via xce_snapshot_chunk_req_raw, task 7.2)
  • Unit test: 128KB file → 2 chunks with correct offsets; small file → 1 chunk
  • Requirements: 4.1, 4.2
  • Boundary: Snapshot Transfer (config_repl.c)
  • Depends: 1.3

• 3.5b Fix SEND_SNAPSHOT production wiring and oldest_wal_lsn

  • Use persist.checkpoint_lsn (not committed_lsn) for snapshot file lookup
  • Stat the snapshot file to get actual total_size (not hardcoded 131072)
  • Wire persist.oldest_wal_lsn into ha_evaluate_repl_action call site (remove the hardcoded 0)
  • Unit test: verify decision loop triggers SEND_SNAPSHOT when standby_lsn < oldest_wal_lsn
  • Requirements: 4.1, 4.3
  • Boundary: Controller FSM (ha.c)
  • Depends: 3.5a, 3.2

• 3.5c Standby-side snapshot receive handler (OPCode_snapshotTransfer)

  • Add case OPCode_snapshotTransfer in myreq_handler dispatch (cluster.c)
  • Write each chunk payload to snap.part temp file; on isLast=true, verify total size, atomic rename to snap-{lsn}.bin, load via load_snapshot()
  • Set applied_lsn = checkpoint_lsn (in-memory) after successful load
  • On mid-transfer reconnect: discard snap.part (restart from chunk 0 on next reconnect)
  • Unit test: receive 2 chunks, verify file assembled and renamed correctly
  • Requirements: 4.2, 4.3, 4.4
  • Boundary: Standby Dispatch (cluster.c, ha.c)
  • Depends: 1.3, 3.5a

• 3.5d Snapshot transfer integration and dispatch-path tests

  • Test: SEND_SNAPSHOT dispatch path triggers hook (like PUSH_CONFIG/STREAM_WAL tests in test_ha_role.c)
  • Test: standby receive → snap.part assembled → isLast rename → applied_lsn set
  • Test: reconnected standby discards snap.part (restart from chunk 0)
  • All tests pass under ASAN
  • Requirements: 4.3, 4.4, 4.5
  • Boundary: Integration (ha.c, cluster.c, test_ha_role.c, test_config_repl.c)
  • Depends: 3.5b, 3.5c

---

• 4. Post-election safety: Safe commit point from vote replies

• 4.1 Determine safe_committed_lsn from vote replies in raft_cb_become_leader

  • ha_elect_ctx_t.votes[] is already populated by the vote reply handler (peerLastLogLsn, peerConfigTag per voter); this task reads those fields after winning election
  • Compute safe_committed_lsn = max LSN held by a majority of voters (iterate all voters with granted=true)
  • Compute safe_config_tag from peerConfigTag using the same majority logic
  • Unit test: voters at LSNs [99, 100], self at 100 → safe_committed_lsn=100; voters at [50, 51] → safe_committed_lsn=51
  • Requirements: 2.3
  • Boundary: Controller FSM (ha.c)

• 4.2 Replace stale-ctrlhello truncation with vote-derived safe commit point

  • Remove persist_truncate_wal_above(last_known_committed_lsn) from the become-leader path
  • Replace with persist_truncate_wal_above(safe_committed_lsn) using the value from task 4.1
  • Set committed_lsn = safe_committed_lsn after truncation
  • Unit test: verify entry at safe_committed_lsn is NOT truncated; entry above it IS
  • Requirements: 2.3
  • Boundary: Controller FSM (ha.c)
  • Depends: 4.1

---

• 5. Integration: Simplify ctrlhello and harden COMMITTING gate

• 5.1 Reduce handle_ctrlhello to connection setup only

  • Remove config push, snapshot check, and WAL streaming from handle_ctrlhello
  • Keep: set primary_ctrl_fd, call ha_on_heartbeat_received, arm liveness watchdog (ha_arm_heartbeat_timeout_vtask)
  • All replication decisions handled by the heartbeat decision loop (tasks 3.2–3.5)
  • Unit test: call handle_ctrlhello, verify no OPCode_replConfigPush or OPCode_walCatchupEntry messages enqueued
  • Requirements: 1.1, 1.2
  • Boundary: Connection Setup (config_repl.c)

• 5.2 Wire majority-convergence → PRIMARY transition

  • In the heartbeat reply handler, after updating convergence tracking, count standby_converged and compare to majority threshold
  • Threshold: count >= n_peers / 2 (C integer division — for odd n the primary + converged standbys form a cluster majority; n=3→1, n=5→2, n=1→0). Raft rejects even n_peers > 1. (reconciled 2026-06-09, task 7.5) promote to CTRL_PRIMARY (formula reconciled 2026-06-09, task 7.5)
  • Single-node: n_peers <= 1, threshold = 0 — satisfied immediately
  • Log INFO: "majority converged (N/M standbys, threshold T) — CTRL_PRIMARY"
  • Unit test: 3-node — 0/2 → COMMITTING; 1/2 → PRIMARY (majority); 5-node — 1/4 → COMMITTING; 2/4 → PRIMARY; single-node → immediate
  • Requirements: 2.3, 2.4, 6.1, 6.3, 6.5
  • Boundary: Controller FSM (ha.c)
  • Depends: 3.1

• 5.3 Extend committing timeout to degraded-mode safety net

  • Change committing_timeout_ms default from 10000 to 60000
  • Timeout handler logs WARN with per-standby convergence state, promotes in degraded mode
  • Majority-convergence path (task 5.2) is the PRIMARY promotion mechanism; timeout is the fallback
  • WARN log includes: number of standbys converged and number still pending
  • Requirements: 6.4
  • Boundary: Controller FSM (ha.c)

• 5.4 Add observability log statements

  • WAL catch-up start: INFO with start/target LSN — covered by unit test in task 6.2
  • Standby converged: INFO with identity, config_tag, applied_lsn — covered by unit test in task 6.1
  • Snapshot transfer start/complete: INFO with checkpoint_lsn, size — covered by unit test in task 6.4
  • Majority reached → CTRL_PRIMARY: INFO with converged count and quorum size — covered by unit test in task 5.2
  • Committing timeout: WARN with per-standby state — covered by unit test in task 5.3
  • All 5 log statements visible in test output; meson test --suite controller green
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5
  • Boundary: Logging (ha.c, config_repl.c)

---

• 6. Validation: Tests and integration verification

• 6.1 Unit tests for decision loop and convergence tracking

  • Test: (tag=0, lsn=0) → PUSH_CONFIG; (tag=T, lsn=0) → STREAM_WAL; (tag=T, lsn=N) → CONVERGED
  • Test: majority detection (0/2 → not majority; 1/2 → majority for 3-node; per n_peers/2 formula)
  • Test: heartbeat reply populates stateReport with correct values
  • All tests pass under ASAN
  • Requirements: 1.1, 1.2, 2.2, 2.3, 2.4

• 6.2 Unit tests for WAL catch-up and batch handling

  • Test: apply single entry, apply batch of 256, error entry skipped, applied_lsn advances correctly
  • Test: batch size limit enforced; applied_lsn only advances after full batch written
  • All tests pass under ASAN
  • Requirements: 3.1, 3.3, 3.4, 3.5, 5.3, 5.5

• 6.3 Unit tests for safe commit point from vote replies

  • Test: voters at LSNs [99, 100], self at 100 → safe_committed_lsn=100
  • Test: voters at LSNs [50, 51], self at 50 → safe_committed_lsn=51 (majority has 51)
  • Test: single node → safe_committed_lsn = self.applied_lsn
  • All tests pass under ASAN
  • Requirements: 2.3

• 6.4 Unit tests for snapshot chunk transfer

  • Test: 128KB snapshot in 2 chunks → standby receives, verifies total size, atomically renames, loads; applied_lsn = checkpoint_lsn
  • Test: mid-transfer reconnect → standby discards snap.part, primary restarts from chunk 0 on reconnect
  • Test: snapshot transfer only triggered when standby_lsn < oldest_wal_lsn
  • All tests pass under ASAN
  • Requirements: 4.1, 4.2, 4.3, 4.4

• 6.5 Integration test: 3-node startup via majority convergence

  • Requires task 5.3 (committing_timeout_ms=60000) deployed so timeout does not fire during test
  • Deploy via /xce-deploy-ha; all 3 nodes start within 15s
  • Verify primary enters CTRL_PRIMARY via majority-convergence INFO log (not committing timeout WARN)
  • Verify standby with config_tag=0 receives config push, then WAL, then reports converged
  • Verify no degraded-mode WARN in logs
  • Requirements: 6.1, 6.3, 6.5, 7.4

• 6.6 Integration test: snapshot transfer for lagging standby

  • Start cluster, let WAL accumulate, kill one standby
  • Trim WAL on primary so oldest_wal_lsn > standby's last applied_lsn
  • Restart standby; verify snapshot transfer triggered (INFO log: checkpoint_lsn, size) and completes
  • Verify standby converges and enters CTRL_STANDBY
  • Requirements: 4.1, 4.2, 4.5

• 6.7 E2E test: full cluster lifecycle

  • Deploy 3 nodes via /xce-deploy-ha; verify single PRIMARY, zero degraded mode
  • Kill primary; verify standby promotes via RDHUP → election → majority convergence
  • Verify promoted primary serves from committed state (no data loss)
  • Full ASAN build: meson test --suite controller green
  • Requirements: 6.2, 6.3, 6.4

---

Reopened 2026-06-09 — Gap Analysis Validation Pass

The 2026-06-09 gap analysis (research.md) found the replication data plane missing: snapshot chunks ship with payload = NULL (config_repl.c "encoding deferred"; cluster.c) and WAL catch-up applies entries header-only (payload_len=0), so apply_wal_entry advances LSN without restoring entities. Standbys report "converged" without holding the state. Tests 6.5–6.7 pass because they assert LSN convergence only. The spec MUST NOT be marked implemented until this series lands.

• 7. Replication payload data plane
• 7.1 Define the payload wire format: capnp Data field + typemap codegen for WAL entries and snapshot chunks. Coordinate with ha-config-bootstrap task 7.6 — live repl_broadcast is also a no-send stub and must share the same format.
  • Requirements: 3.1, 4.2, 5.3
  • Boundary: Wire format (controller.capnp, config_repl.c)
• 7.2 Snapshot chunk payload transmission (primary) + chunk write to snap.part (standby); verify total size before atomic rename (phased option: hand-wire raw bytes first if typemap codegen is blocked)
  • Requirements: 4.1, 4.2, 4.3
  • Boundary: Snapshot transfer (config_repl.c, cluster.c)
• 7.3 WAL catch-up entry payload transmission + entity restoration in apply_wal_entry (e.g. WAL_JOB_SUBMIT recreates the job); applied_lsn advances only after the entity is applied
  • Requirements: 3.1, 3.4, 5.3
  • Boundary: WAL catch-up (ha.c, config_repl.c, persist.c)
• 7.4 Re-zero convergence tracking arrays (standby_reported_*, standby_converged) on entry to CTRL_PRIMARY_COMMITTING — stale convergence from a prior term must not satisfy the gate after re-election
  • Requirements: 2.1
  • Boundary: Convergence tracking (ha.c)
• 7.5 Reconcile the convergence threshold: code requires ALL standbys (n_peers−1), requirements say ceil(n/2)−1, tasks say ceil(n/2), code was n_peers−1 — reconciled to n_peers / 2 (C integer division), recorded in Decision Log 2026-06-09
  • Requirements: 2.4, 6.1
  • Boundary: Quorum policy (ha.c, design.md)
• 7.6 Production unlink(snap.part) on mid-transfer reconnect (Req 4.4) and fresh checkpoint creation at snapshot start instead of reusing the old checkpoint (Req 4.1)
  • Requirements: 4.1, 4.4
  • Boundary: Snapshot lifecycle (config_repl.c)
  • Requirements: 6.2, 6.4
  • Boundary: Integration test (deploy-ha)