feat(ha-wal-replication): HA→replicator standby registration bridge (tasks 2.1-2.3)

Register standbys with replicator on connect (election promotion +
late-joiner), deregister on disconnect (vote err handler + standby err
handler). repl_add_standby now idempotent (duplicate ep returns
existing slot).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

.kiro/specs/ha-wal-replication/tasks.md

@@ -0,0 +1,124 @@
+# Implementation Plan: HA WAL Replication
+
+## Task Overview
+
+5 requirements mapped to 4 implementation groups + 1 validation group. Bridging `repl.c` (replicator) and `ha.c`/`config_repl.c` (HA connection layer) so WAL entries transit from primary to standbys and are majority-committed before the caller is acknowledged.
+
+---
+
+- [ ] 1. repl_broadcast TCP send
+- [x] 1.1 Fill in repl_broadcast send path
+  - Remove `(void)payload; (void)payload_len;` and the `pending_ack_count++` counter.
+  - For each non-catch-up standby (`s->ep != NULL, !in_catch_up`), build a single-entry `wal_catchup_entry_t` with `lsn=hdr->lsn`, `entry_type=hdr->type`, `raw_payload=payload`, `raw_payload_len=payload_len`.
+  - Enqueue via `xce_wal_batch_req_raw(s->ep, &entry, 1)`.
+  - If `xce_wal_batch_req_raw` fails, log WARN and continue to next standby.
+  - Pending-ack-count eviction guard (line 217-225) stays; evicted standbys skip the send.
+  - Compilation succeeds; `/xce-build controller` green.
+  - _Requirements: 2.1, 2.2, 2.3, 2.4_
+  - _Boundary: Replicator (repl.c)_
+
+- [x] 1.2 Verify no standbys → immediate return
+  - Confirm existing guard: when `r->standbys[]` is all-NULL, loop iterates zero times, returns 0.
+  - Add unit test: alloc replicator with zero standbys, call `repl_broadcast`, verify returns 0 without crashing or sending.
+  - _Requirements: 2.4_
+  - _Boundary: Replicator (repl.c)_
+
+---
+
+- [ ] 2. HA→replicator standby registration bridge
+- [x] 2.1 Register standbys on connect
+  - In `raft_cb_become_leader` (ha.c), after promoting vote eps to `ha->standby_endpoints`: for each promoted ep, call `repl_add_standby(&c->persist.repl, ep, peer_host, peer_port)`.
+  - In `on_standby_connected` (ha.c), after ctrlhello is sent: call `repl_add_standby` with the late-joiner's endpoint, host, and port.
+  - Host/port obtained from `c->myconf->peers_cfg.candidates[i]`.
+  - Log DEBUG3 on successful registration.
+  - _Requirements: 1.1_
+  - _Boundary: HA connection layer (ha.c, config_repl.c)_
+
+- [x] 2.2 Deregister standbys on disconnect
+  - In `vote_ep_err_handler` (ha.c): before or after removing ep from `ha->standby_endpoints`, call `repl_remove_standby(&c->persist.repl, ep)`.
+  - In `standby_ep_err_handler_primary` (ha.c): call `repl_remove_standby` before endpoint cleanup.
+  - `repl_remove_standby` is NULL-safe (returns early if ep NULL).
+  - _Requirements: 1.2_
+  - _Boundary: HA connection layer (ha.c)_
+
+- [x] 2.3 Idempotent re-registration
+  - `repl_add_standby` already handles duplicate ep: it finds the existing slot by ep pointer match, resets fields.
+  - Unit test: call `repl_add_standby` twice with same ep, verify no double-count and no crash.
+  - _Requirements: 1.3_
+  - _Boundary: Replicator (repl.c)_
+
+---
+
+- [ ] 3. Standby WAL ACK reply
+- [ ] 3.1 Send WalAckReply after WAL entry applied
+  - In `ha_apply_wal_catchup_entry` (ha.c), after `ha_apply_single_entry` succeeds for an entry: construct a `WalAckReply { term=ha->term, lsn=ent->lsn, success=1 }` and send via `xce_reply` to the endpoint that delivered the entry.
+  - The endpoint is not directly available in `ha_apply_wal_catchup_entry` — it's called from `cluster.c:handle_wal_catchup_entry` which has the ep. Pass `ep` as a new parameter: `ha_apply_wal_catchup_entry(ha, p, entry, ep)`.
+  - _Requirements: 3.1, 3.2_
+  - _Boundary: Standby dispatch (cluster.c, ha.c)_
+
+- [ ] 3.2 Skip ACK on apply failure
+  - In `ha_apply_single_entry` (ha.c): return distinct error codes for skip-vs-crash.
+  - In `ha_apply_wal_catchup_entry`: if `ha_apply_single_entry` returns non-zero, skip the ACK for that entry (already logged as ERROR).
+  - _Requirements: 3.3_
+  - _Boundary: Standby dispatch (ha.c)_
+
+- [ ] 3.3 Standby role guard
+  - The existing `ha_dispatch_gate` already allows `OPCode_walCatchupEntry` in `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT`.
+  - Verify with unit test: standby applies WAL entry while in `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT` — all succeed.
+  - _Requirements: 3.4_
+  - _Boundary: Standby dispatch (cluster.c, ha.c)_
+
+---
+
+- [ ] 4. Primary ACK handler → advance committed_lsn
+- [ ] 4.1 Handle incoming WalAckReply on primary
+  - In `repl_ack_decode_and_dispatch` (ha.c:2506), add case for `RequestReply_data_walackreplymsg`.
+  - Extract `lsn` and `success` from `WalAckReply`. If `success`, call `repl_on_ack(&c->persist.repl, ep, lsn)`.
+  - `repl_on_ack` (repl.c:158) already updates `acked_lsn` and signals `ack_cond` — no changes needed there.
+  - _Requirements: 4.3_
+  - _Boundary: Primary heartbeat handler (ha.c)_
+
+- [ ] 4.2 Verify majority-ack advances committed_lsn
+  - `persist_wal_sync_append` already calls `repl_wait_majority_ack` and advances `committed_lsn` on success.
+  - Integration test: start 3-node cluster, submit a job, verify `committed_lsn > 0` on primary after submit.
+  - _Requirements: 4.1, 4.2, 4.4_
+  - _Boundary: Persistence (persist.c, repl.c)_
+
+---
+
+- [ ] 5. Validation: tests and integration
+- [ ] 5.1 Unit test: repl_broadcast sends to registered standby
+  - Alloc replicator, register a mock endpoint via `repl_add_standby`.
+  - Call `repl_broadcast` with test header + payload.
+  - Verify `xce_wal_batch_req_raw` was invoked (via test hook pattern).
+  - Verify payload bytes round-trip through encode→decode.
+  - _Requirements: 2.1, 5.2_
+  - _Boundary: Test (testsuite/test_ha_wal_repl.c)_
+
+- [ ] 5.2 Unit test: ACK round-trip
+  - Standby side: encode WalAckReply, decode on primary side, call `repl_on_ack`.
+  - Verify `acked_lsn` updated, `ack_cond` signaled, `repl_wait_majority_ack` unblocks.
+  - _Requirements: 4.3, 5.3_
+  - _Boundary: Test (testsuite/test_ha_wal_repl.c)_
+
+- [ ] 5.3 Unit test: registration lifecycle
+  - Register, verify `nstandbys==1` and `standbys[0].ep == ep`.
+  - Register duplicate, verify `nstandbys==1` (idempotent).
+  - Deregister, verify slot freed (`ep==NULL`, `nstandbys==0`).
+  - _Requirements: 1.1, 1.2, 1.3_
+  - _Boundary: Test (testsuite/test_ha_wal_repl.c)_
+
+- [ ] 5.4 Integration test: 3-node cluster WAL replication
+  - Deploy via `/xce-deploy-ha`; verify primary elected.
+  - Submit job via `jobsub`; verify job appears in `joblist` on primary AND on all standbys.
+  - Kill primary; verify promoted standby has the job (failover durability).
+  - Verify `committed_lsn` advanced on all nodes.
+  - _Requirements: 4.1, 5.5_
+  - _Boundary: Integration (deploy-ha)_
+
+- [ ] 5.5 ASAN and regression
+  - Full controller unit suite green under ASAN.
+  - No new ASAN leaks or errors vs baseline.
+  - `/xce-build controller` green.
+  - _Requirements: 5.5_
+  - _Boundary: Build (meson)_