# Implementation Plan: ha-peer-connection-reuse

- [x] 1. Foundation: track outgoing vote endpoint per peer

- [x] 1.1 Add `ep` field to `ha_vote_entry_t` and populate it during vote endpoint creation
  - Add `struct endpoint_t *ep` field to `ha_vote_entry_t` in `ha.h` (forward-declare `endpoint_t` if not already present in that header)
  - In `raft_cb_send_vote_request` (ha.c): after `iomgr_monitor(ep)` succeeds, store `ep` in `ectx->votes[votes_idx].ep`; leave `votes[votes_idx].ep = NULL` on connect or monitor failure
  - In the test helper `vote_ep_ctx_build_for_test`: also accept an `endpoint_t *ep` parameter and store it in `votes[peer_idx].ep` so tests can verify the field
  - Observable: ASAN-clean build; `ectx->votes[i].ep` is non-NULL for successfully connected peers and NULL for failed peers in existing vote-send tests
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 4.1_

- [x] 2. Core: implement state-aware message dispatcher

- [x] 2.1 (P) Implement `peer_ctrl_msg_handler` as a role-dispatching iomgr handler
  - Add `static int peer_ctrl_msg_handler(void *ep_raw)` in ha.c
  - On `CTRL_ELECTING`: call `vote_reply_decode_and_dispatch(ep, vctx)` — factor the body of the existing `vote_reply_msg_handler` into this static helper (preserving XCE_TESTING term-fencing and `xce_vote_reply_hook` seam)
  - On `CTRL_PRIMARY_COMMITTING` or `CTRL_PRIMARY`: call `repl_ack_decode_and_dispatch(ep, vctx)` — new static helper that decodes an inbound capnp packet, checks opcode == `OPCode_replCatchUpDone`, then calls `ha_repl_ack_received(ha, c)`; logs WARN and drops on unexpected opcode
  - On any other role (`CTRL_STANDBY`, `CTRL_INIT`): free packet, return 0, log WARN
  - Observable: unit test verifies that injecting a vote-reply packet with role=ELECTING calls `xce_vote_reply_hook`; injecting a replCatchUpDone packet with role=COMMITTING calls the repl-ack hook; injecting with role=STANDBY results in packet dropped and no hook called
  - _Requirements: 1.2_
  - _Boundary: peer_ctrl_msg_handler_

- [x] 3. Core: promote winner's vote endpoints to standby control channels

- [x] 3.1 (P) Extend `raft_cb_become_leader` to populate `ha->standby_endpoints` from surviving vote endpoints
  - Read `ha->elect_ctx`; if NULL (single-node path), skip the loop
  - Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)` (note: votes[0] is self in the raft library; peer eps start at index 1 — verify against `raft_cb_send_vote_request` peer_idx mapping)
  - For each non-NULL `votes[i].ep`: acquire `ha->lock`, append to `ha->standby_endpoints`, release lock
  - Log `INFO "HA: promoted vote ep[%d] → standby channel"` for each promoted endpoint
  - Observable: after `raft_cb_become_leader` fires in a unit test with 2 populated `votes[i].ep`, `ha->standby_endpoints->count == 2`; subsequent call to `ha_send_ctrlhello_to_all_standbys` iterates the promoted endpoints
  - _Requirements: 1.3, 5.1, 5.2_
  - _Boundary: raft_cb_become_leader_

- [x] 4. Core: passive standby path — close vote endpoints and arm watchdog

- [x] 4.1 (P) Extend `raft_cb_become_follower` to close vote endpoints and arm heartbeat watchdog; remove `fetch_config_from_primary` call
  - Read `ha->elect_ctx`; if NULL, skip close loop
  - Iterate `ctx->votes[i].ep` for all i: if non-NULL, call `endpoint_fini(votes[i].ep)`, set `votes[i].ep = NULL`
  - Log `INFO "HA: closing vote ep[%d] (loser path)"` for each closed endpoint
  - Remove the `#ifndef XCE_TESTING` block that calls `fetch_config_from_primary` via `delay_task_ms` — delete it entirely
  - After `ha->role = CTRL_STANDBY` is stored, call `ha_arm_heartbeat_timeout_vtask(ha, c)` (no-op when `c == NULL` in tests)
  - Observable: unit test with mocked `ha_arm_heartbeat_timeout_vtask` hook verifies it is called exactly once; `votes[i].ep == NULL` for all i after the call; no `delay_task_ms` for `fetch_config_from_primary` is dispatched
  - _Requirements: 1.4, 2.1, 2.3, 2.4, 3.3, 4.1_
  - _Boundary: raft_cb_become_follower_

- [x] 5. Core: designate primary control channel from ctrlhello

- [x] 5.1 (P) Extend `handle_ctrlhello` to set `primary_ctrl_fd` on first ctrlhello when role is CTRL_STANDBY
  - In the ctrlhello handler (config_repl.c): if `ha->role == CTRL_STANDBY` and `ha->primary_ctrl_fd == -1`, set `ha->primary_ctrl_fd = ep->fd`
  - Call `ha_on_heartbeat_received(ha)` to reset the watchdog deadline (marks the first live message from primary)
  - Guard with `primary_ctrl_fd == -1` for idempotency — repeated ctrlhellos on the same connection do not re-register
  - Observable: unit test calls `handle_ctrlhello` with `ha->role = CTRL_STANDBY`, `ha->primary_ctrl_fd = -1`; afterward `ha->primary_ctrl_fd == ep->fd`; calling again with the same ep leaves `primary_ctrl_fd` unchanged
  - _Requirements: 3.1, 3.2, 3.3_
  - _Boundary: handle_ctrlhello_

- [x] 6. Integration: wire dispatcher and promoted-connection lifecycle

- [x] 6.1 Replace `vote_reply_msg_handler` assignment in `on_vote_connected` with `peer_ctrl_msg_handler`
  - In `on_vote_connected` (ha.c): change `ep->msg_handler = vote_reply_msg_handler` to `ep->msg_handler = peer_ctrl_msg_handler`
  - `vote_reply_msg_handler` retained for `vote_reply_msg_handler_for_test` under XCE_TESTING; body factored into `vote_reply_decode_and_dispatch`
  - Observable: existing vote-send tests still pass with no behavior change; the `xce_vote_reply_hook` seam path is reached via `peer_ctrl_msg_handler` → `vote_reply_decode_and_dispatch`
  - _Depends: 2.1_
  - _Requirements: 1.2_

- [x] 6.2 Extend `vote_ep_err_handler` to remove promoted endpoint from `standby_endpoints` on disconnect
  - When `vote_ep_err_handler` fires and `ha->role` is `CTRL_PRIMARY_COMMITTING` or `CTRL_PRIMARY`: acquire `ha->lock`, scan `ha->standby_endpoints` for `ep`, remove it, release lock
  - Existing deny-vote path (fired during CTRL_ELECTING) is preserved unchanged — role gate ensures no double-action
  - Observable: unit test populates `standby_endpoints` with a mock endpoint, sets role=COMMITTING, calls `vote_ep_err_handler`; verifies endpoint removed from array and `standby_endpoints->count` decremented
  - _Requirements: 4.2, 5.5_

- [x] 7. Validation: tests and regression coverage

- [x] 7.1 Unit tests for dispatcher, promotion, and passive path
  - `peer_ctrl_msg_handler` role-routing: three scenarios (ELECTING → vote hook; COMMITTING → repl-ack hook; STANDBY → drop), each confirmed by mock hooks
  - `raft_cb_become_leader` promotion: `standby_endpoints` populated from non-NULL `votes[i].ep` entries; NULL entries skipped
  - `raft_cb_become_follower` close path: all `votes[i].ep` set to NULL; `ha_arm_heartbeat_timeout_vtask` called; no `fetch_config_from_primary` dispatched
  - `handle_ctrlhello` idempotency: first call sets `primary_ctrl_fd`; second call is a no-op
  - `vote_ep_err_handler` standby removal: endpoint removed from `standby_endpoints` when role=COMMITTING
  - Observable: `make test` for controller testsuite passes with no new failures; ASAN reports no leaks or UAF
  - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.1, 4.2, 5.5_

- [x] 7.2 Integration test: standby receives ctrlhello without active outbound primary connect
  - Using existing two-node test harness in `test_ha_bootstrap.c` or `test_p3_integration.c`: boot two nodes, verify the standby never invokes `ha_find_and_connect_to_primary` after election
  - Verify ctrlhello arrives on the standby within `election_timeout_ms + 200ms` via the promoted inbound connection
  - Verify `ha->primary_ctrl_fd` is set on the standby after ctrlhello
  - Observable: test passes end-to-end; ASAN reports no new errors compared to baseline
  - _Requirements: 2.2, 2.4, 3.1, 5.3, 5.5_

- [x] 7.3 Integration test: promoted connection carries WAL replication ACK
  - Verify `OPCode_replCatchUpDone` decoded on the promoted ep increments `ha->repl_ack_count` and triggers `ha_promote_to_primary` when majority threshold reached
  - Observable: `repl_ack_count` reaches quorum - 1 and role transitions to CTRL_PRIMARY; ASAN clean
  - _Requirements: 5.2, 5.4_