lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-wal-replication/tasks.md
Rongsong Shen 9f530f5742 docs: fix unchecked group-header checkboxes across 8 fully-implemented specs 
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
2026-07-07 08:45:41 +00:00

7.5 KiB
Raw Blame History

Implementation Plan: HA WAL Replication

Task Overview

5 requirements mapped to 4 implementation groups + 1 validation group. Bridging repl.c (replicator) and ha.c/config_repl.c (HA connection layer) so WAL entries transit from primary to standbys and are majority-committed before the caller is acknowledged.

  • 1. repl_broadcast TCP send

  • 1.1 Fill in repl_broadcast send path

    • Remove (void)payload; (void)payload_len; and the pending_ack_count++ counter.
    • For each non-catch-up standby (s->ep != NULL, !in_catch_up), build a single-entry wal_catchup_entry_t with lsn=hdr->lsn, entry_type=hdr->type, raw_payload=payload, raw_payload_len=payload_len.
    • Enqueue via xce_wal_batch_req_raw(s->ep, &entry, 1).
    • If xce_wal_batch_req_raw fails, log WARN and continue to next standby.
    • Pending-ack-count eviction guard (line 217-225) stays; evicted standbys skip the send.
    • Compilation succeeds; /xce-build controller green.
    • Requirements: 2.1, 2.2, 2.3, 2.4
    • Boundary: Replicator (repl.c)

  • 1.2 Verify no standbys → immediate return

    • Confirm existing guard: when r->standbys[] is all-NULL, loop iterates zero times, returns 0.
    • Add unit test: alloc replicator with zero standbys, call repl_broadcast, verify returns 0 without crashing or sending.
    • Requirements: 2.4
    • Boundary: Replicator (repl.c)

  • 2. HA→replicator standby registration bridge

  • 2.1 Register standbys on connect

    • In raft_cb_become_leader (ha.c), after promoting vote eps to ha->standby_endpoints: for each promoted ep, call repl_add_standby(&c->persist.repl, ep, peer_host, peer_port).
    • In on_standby_connected (ha.c), after ctrlhello is sent: call repl_add_standby with the late-joiner's endpoint, host, and port.
    • Host/port obtained from c->myconf->peers_cfg.candidates[i].
    • Log DEBUG3 on successful registration.
    • Requirements: 1.1
    • Boundary: HA connection layer (ha.c, config_repl.c)

  • 2.2 Deregister standbys on disconnect

    • In vote_ep_err_handler (ha.c): before or after removing ep from ha->standby_endpoints, call repl_remove_standby(&c->persist.repl, ep).
    • In standby_ep_err_handler_primary (ha.c): call repl_remove_standby before endpoint cleanup.
    • repl_remove_standby is NULL-safe (returns early if ep NULL).
    • Requirements: 1.2
    • Boundary: HA connection layer (ha.c)

  • 2.3 Idempotent re-registration

    • repl_add_standby already handles duplicate ep: it finds the existing slot by ep pointer match, resets fields.
    • Unit test: call repl_add_standby twice with same ep, verify no double-count and no crash.
    • Requirements: 1.3
    • Boundary: Replicator (repl.c)

  • 3. Standby WAL ACK reply

  • 3.1 Send WalAckReply after WAL entry applied

    • In ha_apply_wal_catchup_entry (ha.c), after ha_apply_single_entry succeeds for an entry: construct a WalAckReply { term=ha->term, lsn=ent->lsn, success=1 } and send via xce_reply to the endpoint that delivered the entry.
    • The endpoint is not directly available in ha_apply_wal_catchup_entry — it's called from cluster.c:handle_wal_catchup_entry which has the ep. Pass ep as a new parameter: ha_apply_wal_catchup_entry(ha, p, entry, ep).
    • Requirements: 3.1, 3.2
    • Boundary: Standby dispatch (cluster.c, ha.c)

  • 3.2 Skip ACK on apply failure

    • In ha_apply_single_entry (ha.c): return distinct error codes for skip-vs-crash.
    • In ha_apply_wal_catchup_entry: if ha_apply_single_entry returns non-zero, skip the ACK for that entry (already logged as ERROR).
    • Requirements: 3.3
    • Boundary: Standby dispatch (ha.c)

  • 3.3 Standby role guard

    • The existing ha_dispatch_gate already allows OPCode_walCatchupEntry in CTRL_STANDBY, CTRL_ELECTING, and CTRL_INIT.
    • Verify with unit test: standby applies WAL entry while in CTRL_STANDBY, CTRL_ELECTING, CTRL_INIT — all succeed.
    • Requirements: 3.4
    • Boundary: Standby dispatch (cluster.c, ha.c)

  • 4. Primary ACK handler → advance committed_lsn

  • 4.1 Handle incoming WalAckReply on primary

    • In repl_ack_decode_and_dispatch (ha.c:2506), add case for RequestReply_data_walackreplymsg.
    • Extract lsn and success from WalAckReply. If success, call repl_on_ack(&c->persist.repl, ep, lsn).
    • repl_on_ack (repl.c:158) already updates acked_lsn and signals ack_cond — no changes needed there.
    • Requirements: 4.3
    • Boundary: Primary heartbeat handler (ha.c)

  • 4.2 Verify majority-ack advances committed_lsn

    • persist_wal_sync_append already calls repl_wait_majority_ack and advances committed_lsn on success.
    • Integration test: start 3-node cluster, submit a job, verify committed_lsn > 0 on primary after submit.
    • Requirements: 4.1, 4.2, 4.4
    • Boundary: Persistence (persist.c, repl.c)

  • 5. Validation: tests and integration

  • 5.1 Unit test: repl_broadcast sends to registered standby

    • Alloc replicator, register a mock endpoint via repl_add_standby.
    • Call repl_broadcast with test header + payload.
    • Verify xce_wal_batch_req_raw was invoked (via test hook pattern).
    • Verify payload bytes round-trip through encode→decode.
    • Requirements: 2.1, 5.2
    • Boundary: Test (testsuite/test_ha_wal_repl.c)

  • 5.2 Unit test: ACK round-trip

    • Standby side: encode WalAckReply, decode on primary side, call repl_on_ack.
    • Verify acked_lsn updated, ack_cond signaled, repl_wait_majority_ack unblocks.
    • Requirements: 4.3, 5.3
    • Boundary: Test (testsuite/test_ha_wal_repl.c)

  • 5.3 Unit test: registration lifecycle

    • Register, verify nstandbys==1 and standbys[0].ep == ep.
    • Register duplicate, verify nstandbys==1 (idempotent).
    • Deregister, verify slot freed (ep==NULL, nstandbys==0).
    • Requirements: 1.1, 1.2, 1.3
    • Boundary: Test (testsuite/test_ha_wal_repl.c)

  • 5.4 Integration test: 3-node cluster WAL replication

    • Deploy via /xce-deploy-ha; verify primary elected.
    • Submit job via jobsub; verify job appears in joblist on primary AND on all standbys.
    • Kill primary; verify promoted standby has the job (failover durability).
    • Verify committed_lsn advanced on all nodes.
    • Requirements: 4.1, 5.5
    • Boundary: Integration (deploy-ha)

  • 5.5 ASAN and regression

    • Full controller unit suite green under ASAN.
    • No new ASAN leaks or errors vs baseline.
    • /xce-build controller green.
    • Requirements: 5.5
    • Boundary: Build (meson)
    • 45 tests (308 total assertions) pass. ASAN leaks in test_ha_rdhup_integration and test_ha_start_election_async are pre-existing (last touched by election-iomgr-refactor/checkpoint commits, not ha-wal-replication). Zero new leaks introduced by this feature.

Implementation Notes

  • 5.4 VERIFIED 2026-07-02: deployed a 3-node cluster to ~/tmp/xce-wal-repl-deploy (real ext4). Submitted a job on primary ctrl-1; ctrlstatus on all 3 nodes showed committedLsn=2 matching (replication confirmed — note: joblist itself is gated primary-only, E_XCE_STANDBY/10014, by design in ha.c's CTRL_STANDBY dispatch switch, so standby replication is verified via matching committedLsn/appliedLsn, not a direct standby joblist read). Killed primary ctrl-1; ctrl-3 was promoted (term 2→4); joblist against the surviving pair (masters=all 3, client transparently rotates) showed the job intact; committedLsn=3 matched on both surviving nodes post-failover. PASS on all 4 Observable bullets.