myxce/.kiro/specs/ha-peer-connection-reuse/tasks.md

Implementation Plan: ha-peer-connection-reuse

• 1. Foundation: track outgoing vote endpoint per peer

• 1.1 Add ep field to ha_vote_entry_t and populate it during vote endpoint creation

  • Add struct endpoint_t *ep field to ha_vote_entry_t in ha.h (forward-declare endpoint_t if not already present in that header)
  • In raft_cb_send_vote_request (ha.c): after iomgr_monitor(ep) succeeds, store ep in ectx->votes[votes_idx].ep; leave votes[votes_idx].ep = NULL on connect or monitor failure
  • In the test helper vote_ep_ctx_build_for_test: also accept an endpoint_t *ep parameter and store it in votes[peer_idx].ep so tests can verify the field
  • Observable: ASAN-clean build; ectx->votes[i].ep is non-NULL for successfully connected peers and NULL for failed peers in existing vote-send tests
  • Requirements: 1.1, 1.2, 1.3, 1.4, 4.1

• 2. Core: implement state-aware message dispatcher

• 2.1 (P) Implement peer_ctrl_msg_handler as a role-dispatching iomgr handler

  • Add static int peer_ctrl_msg_handler(void *ep_raw) in ha.c
  • On CTRL_ELECTING: call vote_reply_decode_and_dispatch(ep, vctx) — factor the body of the existing vote_reply_msg_handler into this static helper (preserving XCE_TESTING term-fencing and xce_vote_reply_hook seam)
  • On CTRL_PRIMARY_COMMITTING or CTRL_PRIMARY: call repl_ack_decode_and_dispatch(ep, vctx) — new static helper that decodes an inbound capnp packet, checks opcode == OPCode_replCatchUpDone, then calls ha_repl_ack_received(ha, c); logs WARN and drops on unexpected opcode
  • On any other role (CTRL_STANDBY, CTRL_INIT): free packet, return 0, log WARN
  • Observable: unit test verifies that injecting a vote-reply packet with role=ELECTING calls xce_vote_reply_hook; injecting a replCatchUpDone packet with role=COMMITTING calls the repl-ack hook; injecting with role=STANDBY results in packet dropped and no hook called
  • Requirements: 1.2
  • Boundary: peer_ctrl_msg_handler

• 3. Core: promote winner's vote endpoints to standby control channels

• 3.1 (P) Extend raft_cb_become_leader to populate ha->standby_endpoints from surviving vote endpoints

  • Read ha->elect_ctx; if NULL (single-node path), skip the loop
  • Iterate ctx->votes[i].ep for i in [0, ctx->n_peers) (note: votes[0] is self in the raft library; peer eps start at index 1 — verify against raft_cb_send_vote_request peer_idx mapping)
  • For each non-NULL votes[i].ep: acquire ha->lock, append to ha->standby_endpoints, release lock
  • Log INFO "HA: promoted vote ep[%d] → standby channel" for each promoted endpoint
  • Observable: after raft_cb_become_leader fires in a unit test with 2 populated votes[i].ep, ha->standby_endpoints->count == 2; subsequent call to ha_send_ctrlhello_to_all_standbys iterates the promoted endpoints
  • Requirements: 1.3, 5.1, 5.2
  • Boundary: raft_cb_become_leader

• 4. Core: passive standby path — close vote endpoints and arm watchdog

• 4.1 (P) Extend raft_cb_become_follower to close vote endpoints and arm heartbeat watchdog; remove fetch_config_from_primary call

  • Read ha->elect_ctx; if NULL, skip close loop
  • Iterate ctx->votes[i].ep for all i: if non-NULL, call endpoint_fini(votes[i].ep), set votes[i].ep = NULL
  • Log INFO "HA: closing vote ep[%d] (loser path)" for each closed endpoint
  • Remove the #ifndef XCE_TESTING block that calls fetch_config_from_primary via delay_task_ms — delete it entirely
  • After ha->role = CTRL_STANDBY is stored, call ha_arm_heartbeat_timeout_vtask(ha, c) (no-op when c == NULL in tests)
  • Observable: unit test with mocked ha_arm_heartbeat_timeout_vtask hook verifies it is called exactly once; votes[i].ep == NULL for all i after the call; no delay_task_ms for fetch_config_from_primary is dispatched
  • Requirements: 1.4, 2.1, 2.3, 2.4, 3.3, 4.1
  • Boundary: raft_cb_become_follower

• 5. Core: designate primary control channel from ctrlhello

• 5.1 (P) Extend handle_ctrlhello to set primary_ctrl_fd on first ctrlhello when role is CTRL_STANDBY

  • In the ctrlhello handler (config_repl.c): if ha->role == CTRL_STANDBY and ha->primary_ctrl_fd == -1, set ha->primary_ctrl_fd = ep->fd
  • Call ha_on_heartbeat_received(ha) to reset the watchdog deadline (marks the first live message from primary)
  • Guard with primary_ctrl_fd == -1 for idempotency — repeated ctrlhellos on the same connection do not re-register
  • Observable: unit test calls handle_ctrlhello with ha->role = CTRL_STANDBY, ha->primary_ctrl_fd = -1; afterward ha->primary_ctrl_fd == ep->fd; calling again with the same ep leaves primary_ctrl_fd unchanged
  • Requirements: 3.1, 3.2, 3.3
  • Boundary: handle_ctrlhello

• 6. Integration: wire dispatcher and promoted-connection lifecycle

• 6.1 Replace vote_reply_msg_handler assignment in on_vote_connected with peer_ctrl_msg_handler

  • In on_vote_connected (ha.c): change ep->msg_handler = vote_reply_msg_handler to ep->msg_handler = peer_ctrl_msg_handler
  • vote_reply_msg_handler retained for vote_reply_msg_handler_for_test under XCE_TESTING; body factored into vote_reply_decode_and_dispatch
  • Observable: existing vote-send tests still pass with no behavior change; the xce_vote_reply_hook seam path is reached via peer_ctrl_msg_handler → vote_reply_decode_and_dispatch
  • Depends: 2.1
  • Requirements: 1.2

• 6.2 Extend vote_ep_err_handler to remove promoted endpoint from standby_endpoints on disconnect

  • When vote_ep_err_handler fires and ha->role is CTRL_PRIMARY_COMMITTING or CTRL_PRIMARY: acquire ha->lock, scan ha->standby_endpoints for ep, remove it, release lock
  • Existing deny-vote path (fired during CTRL_ELECTING) is preserved unchanged — role gate ensures no double-action
  • Observable: unit test populates standby_endpoints with a mock endpoint, sets role=COMMITTING, calls vote_ep_err_handler; verifies endpoint removed from array and standby_endpoints->count decremented
  • Requirements: 4.2, 5.5

• 7. Validation: tests and regression coverage

• 7.1 Unit tests for dispatcher, promotion, and passive path

  • peer_ctrl_msg_handler role-routing: three scenarios (ELECTING → vote hook; COMMITTING → repl-ack hook; STANDBY → drop), each confirmed by mock hooks
  • raft_cb_become_leader promotion: standby_endpoints populated from non-NULL votes[i].ep entries; NULL entries skipped
  • raft_cb_become_follower close path: all votes[i].ep set to NULL; ha_arm_heartbeat_timeout_vtask called; no fetch_config_from_primary dispatched
  • handle_ctrlhello idempotency: first call sets primary_ctrl_fd; second call is a no-op
  • vote_ep_err_handler standby removal: endpoint removed from standby_endpoints when role=COMMITTING
  • Observable: make test for controller testsuite passes with no new failures; ASAN reports no leaks or UAF
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.1, 4.2, 5.5

• 7.2 Integration test: standby receives ctrlhello without active outbound primary connect

  • Using existing two-node test harness in test_ha_bootstrap.c or test_p3_integration.c: boot two nodes, verify the standby never invokes ha_find_and_connect_to_primary after election
  • Verify ctrlhello arrives on the standby within election_timeout_ms + 200ms via the promoted inbound connection
  • Verify ha->primary_ctrl_fd is set on the standby after ctrlhello
  • Observable: test passes end-to-end; ASAN reports no new errors compared to baseline
  • Requirements: 2.2, 2.4, 3.1, 5.3, 5.5

• 7.3 Integration test: promoted connection carries WAL replication ACK

  • Verify OPCode_replCatchUpDone decoded on the promoted ep increments ha->repl_ack_count and triggers ha_promote_to_primary when majority threshold reached
  • Observable: repl_ack_count reaches quorum - 1 and role transitions to CTRL_PRIMARY; ASAN clean
  • Requirements: 5.2, 5.4