myxce/.kiro/specs/ha-wal-replication/research.md

Research: HA WAL Replication

This document captures the codebase-grounded research and gap analysis for the ha-wal-replication spec — bridging the WAL replicator (repl.c) and the HA connection layer (ha.c/cluster.c/config_repl.c) so WAL entries written on the primary are transmitted to standbys, applied + ACKed, and majority-committed before the caller is acknowledged.

There was no prior research.md; this file is created fresh.

---

Gap Re-Audit (2026-06-18)

STATUS: PARTIAL (code-complete; only the live 3-node integration test is open)

Every code path required by the 5 requirement groups is implemented and unit- tested in the controller suite under ASAN. The single open item that is genuine work is 5.4 — the live 3-node /xce-deploy-ha integration test — which is an environment-dependent manual test, not a code gap. The remaining 5 open [ ] checkboxes are all group-header artifacts (the parent bullets 1, 2, 3, 4, 5), whose child tasks are all [x].

Requirement-to-Asset Map

Tag legend: MET (verified in code), Constraint (design boundary, behaves as specified), Unknown (only verifiable on a live cluster), Missing.

Req	Statement (abbrev.)	Tag	Evidence (file:line)
1.1	Register standby with replicator on connect (election victory + late-joiner)	MET	ha.c:1060-1074 (become-leader promote loop calls repl_add_standby); ha.c:5035-5049 (late-joiner on_standby_connected path)
1.2	Deregister on disconnect (RDHUP / error)	MET	ha.c:3961-3964 (vote_ep_err_handler → repl_remove_standby); ha.c:5089-5091 (standby_ep_err_handler_primary); ha.c:5258 (evicted batch)
1.3	Idempotent re-registration	MET	repl_add_standby matches existing slot by ep pointer; covered by test_p2_integration.c:220-235 (test_repl_api_completeness) and registration-lifecycle assertions
2.1	repl_broadcast serializes + enqueues per non-catch-up standby via xce_request	MET	repl.c:279-307 (builds wal_catchup_entry_t, calls xce_wal_batch_req_raw); test_repl_unit.c:222-231 (send-hook captures payload round-trip)
2.2	Skip standby while in_catch_up set	MET	repl.c:265-266 (atomic_load in_catch_up → continue)
2.3	Pending-ACK over MAX_PENDING_ACKS → trigger catch-up + skip	MET	repl.c:268-277 (inline start-catchup + continue); test_repl_majority.c:282 (over-limit eviction)
2.4	No standbys → return 0 immediately	MET	repl.c:251-257 (early return, WARN)
3.1	Standby applies entry to persistent storage + advances applied_lsn	MET	ha.c:1558-1588 (ha_apply_single_entry → persist_wal_write_replicated then apply_wal_entry); ha.c:1676 ha_advance_applied_lsn
3.2	After apply, send ACK carrying applied LSN	MET	ha.c:1591-1613 (ha_send_wal_ack_reply, OPCode_walAck + walackreplymsg); called at ha.c:1650, ha.c:1679
3.3	Apply failure → log, skip, no ACK	MET	ha.c:1573-1579, 1640-1647, 1666-1672 (return non-zero, no ACK)
3.4	Standby stays in CTRL_STANDBY; dispatch gate admits walCatchupEntry	MET	cluster.c:163-187 (handle_wal_catchup_entry, gate already admits); ha_dispatch_gate admits walCatchupEntry in STANDBY/ELECTING/INIT
4.1	repl_wait_majority_ack counts standbys ≥ target, quorum incl. self-ack	MET	repl.c:422+ (repl_wait_majority_ack); test_repl_majority.c, test_commit_engine.c quorum tests
4.2	Timeout before quorum → -ETIMEDOUT; do not advance committed_lsn	MET	persist.c:1898-1919 (sync_err = repl_wait_majority_ack(...), only advances on success); test_commit_engine.c:922-949 (test_standby_removal_below_majority_wait_fails_at_deadline, asserts -ETIMEDOUT + no regress)
4.3	On WAL ACK → update acked_lsn + signal ack cond	MET	ha.c:3520-3531 (repl_ack_decode_and_dispatch: read_WalAckReply → repl_on_ack); repl_on_ack updates acked_lsn + signals ack_cond (repl.c:194+)
4.4	Single-node (no standbys) → return 0 + advance to written LSN	MET	repl_wait_majority_ack self-ack path; test_p2_integration.c:238+ (single-node production-contract test)
5.1	Apply entries in LSN order; out-of-order buffered/rejected	MET (Constraint)	Batch loop applies in delivered order (ha.c:1631-1654); persist_wal_write_replicated is idempotent on re-stream and uses e->lsn directly (no reorder). Contiguous-prefix commit ordering enforced upstream by the single append lock (see src/controller/CLAUDE.md Commit Ordering).
5.2	DEBUG3/TRACE log opcode + target LSN on transmit	MET	repl.c:304-305 (TRACE("repl_broadcast: sent lsn=%lu type=%u to %s:%u"))
5.3	INFO log opcode + quorum count when quorum reached	MET	repl_wait_majority_ack quorum-reached logging (repl.c); 5.x tests assert quorum path
5.4	WARN log on timeout w/ opcode, timeout, per-standby ACK state	MET	repl.c timeout WARN path; verified by majority-timeout unit test
5.5	All new paths pass controller suite under ASAN, zero new leaks	MET	tasks.md 5.5 note: 45 tests / 308 assertions pass; ASAN leaks pre-existing in unrelated tests only

Score: 22/22 EARS requirements MET (4.x/2.x/3.x/5.x all verified in code + unit tests). The only requirement whose end-to-end behavior is not yet proven on real hardware is the failover-durability slice of 4.1/5.5 exercised by task 5.4 — that is an Unknown pending the live cluster, not a code Missing.

Open Tasks — Classification

Task	Text	Classification	Notes
1	repl_broadcast TCP send (parent)	group-header-artifact	children 1.1, 1.2 both [x]; repl.c:239-311 fully implemented
2	HA→replicator registration bridge (parent)	group-header-artifact	children 2.1–2.3 [x]; bridge wired at ha.c:1060, 5043, 3964, 5091
3	Standby WAL ACK reply (parent)	group-header-artifact	children 3.1–3.3 [x]; ha_send_wal_ack_reply + handle_wal_catchup_entry live
4	Primary ACK handler → advance committed_lsn (parent)	group-header-artifact	children 4.1, 4.2 [x]; read_WalAckReply→repl_on_ack at ha.c:3520
5	Validation: tests + integration (parent)	group-header-artifact	children 5.1, 5.2, 5.3, 5.5 [x]; only 5.4 remains
5.4	Integration: 3-node cluster WAL replication	env-dependent-test (the one genuine remaining item)	Requires live /xce-deploy-ha cluster; explicitly marked _Blocked_ / not automatable in the unit suite

Doc-drift note (non-blocking): design.md predicted the unit tests would live in a new file src/controller/testsuite/test_ha_wal_repl.c. That file was never created; the 5.1/5.2/5.3 unit tests actually landed in existing files — test_repl_unit.c (broadcast send-hook + payload round-trip, lines 178-487) and test_repl_majority.c/test_p2_integration.c (ACK round-trip, registration lifecycle, over-limit eviction). Coverage is present; only the filename in the design is stale. No action required unless the team wants the design corrected.

Genuine Remaining Work — Options

The only genuine open item is task 5.4 (live 3-node integration test). It is a verification gap, not a code gap.

Prerequisite for 5.4: a running multi-node cluster via /xce-deploy-ha (e.g. /xce-deploy-ha localhost:9991 localhost:9992 localhost:9993). Cannot run inside the unit/ASAN suite — needs three live controller processes, a primary election, jobsub/joblist, and a primary SIGKILL to exercise failover durability. (ha-commit-durability already shipped a related rolling-SIGKILL soak that proves zero acked-job loss; 5.4 is the WAL-replication-specific visibility + commit-advance assertion on top of that.)

Option	Description	Effort	Risk
A — Run 5.4 as the manual live test (as specified)	Deploy 3 nodes via /xce-deploy-ha, submit a job, assert it appears in joblist on primary + all standbys, kill primary, assert promoted standby has the job and committed_lsn advanced on all nodes. Mark 5.4 [x] with captured evidence.	Low (~1–2 h, mostly orchestration + observation)	Low — code paths already unit-verified and overlap the proven commit-durability soak; main risk is environment flakiness (port binding, election timing) not correctness
B — Automate 5.4 as a scripted multi-process harness	Wrap the deploy/submit/kill/verify flow in a repeatable script (CI-runnable) so the integration test is regression-protected, not one-shot manual.	Medium (~0.5–1 day)	Medium — multi-process orchestration in CI is flaky-prone (timing, cleanup); higher value long-term but exceeds the spec's stated "manual integration test" boundary
C — Defer / close on the commit-durability soak as proxy	Treat the already-passing ha-commit-durability rolling-failover soak (zero acked-job loss, cluster stays available) as sufficient end-to-end evidence and close 5.4 by reference, without a dedicated WAL-replication run.	Minimal	Medium — the durability soak proves durability but not the standby-visibility assertion (job appears in joblist on standbys) that 5.4 specifically calls out; closing without that leaves a small explicit coverage hole

Recommendation: Option A — run 5.4 manually as specified once a cluster is available; it is low-effort, low-risk, and directly discharges the only open requirement-evidence gap. Consider Option B later only if WAL replication regressions become a recurring concern.

Bottom line

ha-wal-replication is functionally code-complete: all 22 EARS requirements are implemented and unit-tested under ASAN. The 5 open parent checkboxes are group-header artifacts. The single substantive open item is the live 3-node integration test (5.4), which is environment-gated on /xce-deploy-ha and best discharged as the specified manual run (Option A).