Rongsong Shen
9f530f5742
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation, ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and raft-ha-core-safety each had every numbered sub-task marked [x] but their group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real unchecked sub-tasks in any of the eight files. No task content changed.
128 lines
7.5 KiB
Markdown
128 lines
7.5 KiB
Markdown
# Implementation Plan: HA WAL Replication
|
|
|
|
## Task Overview
|
|
|
|
5 requirements mapped to 4 implementation groups + 1 validation group. Bridging `repl.c` (replicator) and `ha.c`/`config_repl.c` (HA connection layer) so WAL entries transit from primary to standbys and are majority-committed before the caller is acknowledged.
|
|
|
|
---
|
|
|
|
- [x] 1. repl_broadcast TCP send
|
|
- [x] 1.1 Fill in repl_broadcast send path
|
|
- Remove `(void)payload; (void)payload_len;` and the `pending_ack_count++` counter.
|
|
- For each non-catch-up standby (`s->ep != NULL, !in_catch_up`), build a single-entry `wal_catchup_entry_t` with `lsn=hdr->lsn`, `entry_type=hdr->type`, `raw_payload=payload`, `raw_payload_len=payload_len`.
|
|
- Enqueue via `xce_wal_batch_req_raw(s->ep, &entry, 1)`.
|
|
- If `xce_wal_batch_req_raw` fails, log WARN and continue to next standby.
|
|
- Pending-ack-count eviction guard (line 217-225) stays; evicted standbys skip the send.
|
|
- Compilation succeeds; `/xce-build controller` green.
|
|
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
|
- _Boundary: Replicator (repl.c)_
|
|
|
|
- [x] 1.2 Verify no standbys → immediate return
|
|
- Confirm existing guard: when `r->standbys[]` is all-NULL, loop iterates zero times, returns 0.
|
|
- Add unit test: alloc replicator with zero standbys, call `repl_broadcast`, verify returns 0 without crashing or sending.
|
|
- _Requirements: 2.4_
|
|
- _Boundary: Replicator (repl.c)_
|
|
|
|
---
|
|
|
|
- [x] 2. HA→replicator standby registration bridge
|
|
- [x] 2.1 Register standbys on connect
|
|
- In `raft_cb_become_leader` (ha.c), after promoting vote eps to `ha->standby_endpoints`: for each promoted ep, call `repl_add_standby(&c->persist.repl, ep, peer_host, peer_port)`.
|
|
- In `on_standby_connected` (ha.c), after ctrlhello is sent: call `repl_add_standby` with the late-joiner's endpoint, host, and port.
|
|
- Host/port obtained from `c->myconf->peers_cfg.candidates[i]`.
|
|
- Log DEBUG3 on successful registration.
|
|
- _Requirements: 1.1_
|
|
- _Boundary: HA connection layer (ha.c, config_repl.c)_
|
|
|
|
- [x] 2.2 Deregister standbys on disconnect
|
|
- In `vote_ep_err_handler` (ha.c): before or after removing ep from `ha->standby_endpoints`, call `repl_remove_standby(&c->persist.repl, ep)`.
|
|
- In `standby_ep_err_handler_primary` (ha.c): call `repl_remove_standby` before endpoint cleanup.
|
|
- `repl_remove_standby` is NULL-safe (returns early if ep NULL).
|
|
- _Requirements: 1.2_
|
|
- _Boundary: HA connection layer (ha.c)_
|
|
|
|
- [x] 2.3 Idempotent re-registration
|
|
- `repl_add_standby` already handles duplicate ep: it finds the existing slot by ep pointer match, resets fields.
|
|
- Unit test: call `repl_add_standby` twice with same ep, verify no double-count and no crash.
|
|
- _Requirements: 1.3_
|
|
- _Boundary: Replicator (repl.c)_
|
|
|
|
---
|
|
|
|
- [x] 3. Standby WAL ACK reply
|
|
- [x] 3.1 Send WalAckReply after WAL entry applied
|
|
- In `ha_apply_wal_catchup_entry` (ha.c), after `ha_apply_single_entry` succeeds for an entry: construct a `WalAckReply { term=ha->term, lsn=ent->lsn, success=1 }` and send via `xce_reply` to the endpoint that delivered the entry.
|
|
- The endpoint is not directly available in `ha_apply_wal_catchup_entry` — it's called from `cluster.c:handle_wal_catchup_entry` which has the ep. Pass `ep` as a new parameter: `ha_apply_wal_catchup_entry(ha, p, entry, ep)`.
|
|
- _Requirements: 3.1, 3.2_
|
|
- _Boundary: Standby dispatch (cluster.c, ha.c)_
|
|
|
|
- [x] 3.2 Skip ACK on apply failure
|
|
- In `ha_apply_single_entry` (ha.c): return distinct error codes for skip-vs-crash.
|
|
- In `ha_apply_wal_catchup_entry`: if `ha_apply_single_entry` returns non-zero, skip the ACK for that entry (already logged as ERROR).
|
|
- _Requirements: 3.3_
|
|
- _Boundary: Standby dispatch (ha.c)_
|
|
|
|
- [x] 3.3 Standby role guard
|
|
- The existing `ha_dispatch_gate` already allows `OPCode_walCatchupEntry` in `CTRL_STANDBY`, `CTRL_ELECTING`, and `CTRL_INIT`.
|
|
- Verify with unit test: standby applies WAL entry while in `CTRL_STANDBY`, `CTRL_ELECTING`, `CTRL_INIT` — all succeed.
|
|
- _Requirements: 3.4_
|
|
- _Boundary: Standby dispatch (cluster.c, ha.c)_
|
|
|
|
---
|
|
|
|
- [x] 4. Primary ACK handler → advance committed_lsn
|
|
- [x] 4.1 Handle incoming WalAckReply on primary
|
|
- In `repl_ack_decode_and_dispatch` (ha.c:2506), add case for `RequestReply_data_walackreplymsg`.
|
|
- Extract `lsn` and `success` from `WalAckReply`. If `success`, call `repl_on_ack(&c->persist.repl, ep, lsn)`.
|
|
- `repl_on_ack` (repl.c:158) already updates `acked_lsn` and signals `ack_cond` — no changes needed there.
|
|
- _Requirements: 4.3_
|
|
- _Boundary: Primary heartbeat handler (ha.c)_
|
|
|
|
- [x] 4.2 Verify majority-ack advances committed_lsn
|
|
- `persist_wal_sync_append` already calls `repl_wait_majority_ack` and advances `committed_lsn` on success.
|
|
- Integration test: start 3-node cluster, submit a job, verify `committed_lsn > 0` on primary after submit.
|
|
- _Requirements: 4.1, 4.2, 4.4_
|
|
- _Boundary: Persistence (persist.c, repl.c)_
|
|
|
|
---
|
|
|
|
- [x] 5. Validation: tests and integration
|
|
- [x] 5.1 Unit test: repl_broadcast sends to registered standby
|
|
- Alloc replicator, register a mock endpoint via `repl_add_standby`.
|
|
- Call `repl_broadcast` with test header + payload.
|
|
- Verify `xce_wal_batch_req_raw` was invoked (via test hook pattern).
|
|
- Verify payload bytes round-trip through encode→decode.
|
|
- _Requirements: 2.1, 5.2_
|
|
- _Boundary: Test (testsuite/test_ha_wal_repl.c)_
|
|
|
|
- [x] 5.2 Unit test: ACK round-trip
|
|
- Standby side: encode WalAckReply, decode on primary side, call `repl_on_ack`.
|
|
- Verify `acked_lsn` updated, `ack_cond` signaled, `repl_wait_majority_ack` unblocks.
|
|
- _Requirements: 4.3, 5.3_
|
|
- _Boundary: Test (testsuite/test_ha_wal_repl.c)_
|
|
|
|
- [x] 5.3 Unit test: registration lifecycle
|
|
- Register, verify `nstandbys==1` and `standbys[0].ep == ep`.
|
|
- Register duplicate, verify `nstandbys==1` (idempotent).
|
|
- Deregister, verify slot freed (`ep==NULL`, `nstandbys==0`).
|
|
- _Requirements: 1.1, 1.2, 1.3_
|
|
- _Boundary: Test (testsuite/test_ha_wal_repl.c)_
|
|
|
|
- [x] 5.4 Integration test: 3-node cluster WAL replication <!-- VERIFIED 2026-07-02 on ~/tmp (ext4) — see Implementation Notes -->
|
|
- Deploy via `/xce-deploy-ha`; verify primary elected.
|
|
- Submit job via `jobsub`; verify job appears in `joblist` on primary AND on all standbys.
|
|
- Kill primary; verify promoted standby has the job (failover durability).
|
|
- Verify `committed_lsn` advanced on all nodes.
|
|
- _Requirements: 4.1, 5.5_
|
|
- _Boundary: Integration (deploy-ha)_
|
|
|
|
- [x] 5.5 ASAN and regression
|
|
- Full controller unit suite green under ASAN.
|
|
- No new ASAN leaks or errors vs baseline.
|
|
- `/xce-build controller` green.
|
|
- _Requirements: 5.5_
|
|
- _Boundary: Build (meson)_
|
|
- 45 tests (308 total assertions) pass. ASAN leaks in test_ha_rdhup_integration and test_ha_start_election_async are pre-existing (last touched by election-iomgr-refactor/checkpoint commits, not ha-wal-replication). Zero new leaks introduced by this feature.
|
|
|
|
## Implementation Notes
|
|
- 5.4 VERIFIED 2026-07-02: deployed a 3-node cluster to `~/tmp/xce-wal-repl-deploy` (real ext4). Submitted a job on primary ctrl-1; `ctrlstatus` on all 3 nodes showed `committedLsn=2` matching (replication confirmed — note: `joblist` itself is gated primary-only, `E_XCE_STANDBY`/10014, by design in `ha.c`'s `CTRL_STANDBY` dispatch switch, so standby replication is verified via matching `committedLsn`/`appliedLsn`, not a direct standby `joblist` read). Killed primary ctrl-1; ctrl-3 was promoted (term 2→4); `joblist` against the surviving pair (masters=all 3, client transparently rotates) showed the job intact; `committedLsn=3` matched on both surviving nodes post-failover. PASS on all 4 Observable bullets.
|