myxce/.kiro/specs/ha-peer-connection-reuse/design.md

Design Document: ha-peer-connection-reuse

Overview

The XCE controller election currently discards outgoing vote connections after the vote exchange, forcing standbys to create new outbound connections to the primary for the persistent control channel. This produces a 10-second heartbeat timeout delay before standbys detect the primary, spurious retry warnings from premature config-fetch attempts, and a complex tier-1/2/3 primary discovery path.

This spec reuses each candidate's outgoing vote endpoint as the persistent peer control channel: the election winner promotes its connections to replication channels; the loser closes its connections and waits passively for the primary's ctrlhello to arrive on its listener.

The entire change is confined to ha.c and cluster.c. iomgr, the capnp wire protocol, and the raft library are not modified.

Goals

• Eliminate the double-connection round-trip after election
• Remove the standby's active primary discovery path (fetch_config_from_primary)
• Arm the standby heartbeat watchdog immediately when CTRL_STANDBY is entered
• Preserve existing HA behaviors: failover, WAL replication, config push, heartbeat detection

Non-Goals

• SSL/TLS certificate management changes
• Vote grant/deny protocol logic changes
• WAL entry format or replication content changes
• Config file format changes
• Cluster membership changes (add/remove peers)

---

Boundary Commitments

This Spec Owns

• The lifecycle of outgoing vote endpoints from creation through promotion or close
• The peer_ctrl_msg_handler state-aware dispatcher (replaces vote_reply_msg_handler + explicit reassignment)
• Population of ha->standby_endpoints from promoted vote connections on the primary side
• Arming the standby heartbeat watchdog from raft_cb_become_follower instead of after ha_find_and_connect_to_primary
• Setting ha->primary_ctrl_fd from the inbound ctrlhello connection on the standby side
• Removing the fetch_config_from_primary delay task from the standby path

Out of Boundary

• SSL/TLS negotiation details — handled by existing client_ssl_ctx / on_vote_connected path
• iomgr internals — zero changes to src/lib/netio.c or src/include/netio.h
• Raft library — zero changes to src/lib/raft/raft.c
• Heartbeat watchdog thread implementation — only the arming callsite changes
• The existing ha_find_and_connect_to_primary function — it remains for future use; the standby path no longer calls it after election

Allowed Dependencies

• ha_state_t, ha_elect_ctx_t, ha_vote_entry_t — extended with new fields
• vote_ep_ctx_t — reused as-is for promoted connections
• endpoint_t, iomgr_monitor, endpoint_set_msg_handler — used unchanged
• ha->standby_endpoints (zarray) — populated from promoted connections
• raft_cb_become_leader / raft_cb_become_follower — extended to populate/close endpoints
• ha_arm_heartbeat_timeout_vtask — called from raft_cb_become_follower

Revalidation Triggers

• Changing ha_vote_entry_t struct layout (tasks depending on votes[].ep)
• Changing vote_ep_ctx_t fields or lifetime contract
• Changing raft_cb_become_leader / raft_cb_become_follower signatures
• Adding new roles to ctrl_role_t (the dispatcher switch must be updated)
• Changing ha->standby_endpoints population semantics

---

Architecture

Existing Architecture Analysis

The current election flow:

1. Candidate creates one endpoint_t per peer (outgoing vote), registers with iomgr
2. on_vote_connected sends the vote request and reassigns ep->msg_handler = vote_reply_msg_handler
3. After the election, the outgoing vote endpoints are closed by ha_election_cleanup
4. The winning standby calls ha_find_and_connect_to_primary to create a new inbound connection to the primary
5. The heartbeat watchdog is armed after this reconnection succeeds

The promoted-connection flow (this spec):

1. Same vote endpoint creation
2. on_vote_connected sets ep->msg_handler = peer_ctrl_msg_handler (state-aware, never reassigned)
3. In raft_cb_become_leader: each surviving vote endpoint is added to ha->standby_endpoints (primary)
4. In raft_cb_become_follower: outgoing vote endpoints are closed; standby waits passively; watchdog armed immediately
5. Primary's ctrlhello arrives on the standby's listener, setting ha->primary_ctrl_fd

Architecture Pattern & Boundary Map

graph TD
    subgraph Election_Phase
        A[raft_cb_send_vote_request] --> B[endpoint_connect + iomgr_monitor]
        B --> C[on_vote_connected sets peer_ctrl_msg_handler]
        C --> D[send RequestVote via xce_request]
    end

    subgraph Election_Result
        E[raft_cb_become_leader] --> F[promote: add ep to standby_endpoints]
        G[raft_cb_become_follower] --> H[close: endpoint_fini each vote ep]
        G --> I[arm heartbeat watchdog]
    end

    subgraph Promoted_Connection_PRIMARY
        F --> J[peer_ctrl_msg_handler role=COMMITTING or PRIMARY]
        J --> K[repl_ack_decode: ha_repl_ack_received]
    end

    subgraph Standby_Passive
        H --> L[wait for ctrlhello on listener]
        L --> M[handle_ctrlhello sets primary_ctrl_fd]
        I --> N[watchdog arm: fires on silence timeout]
    end

    subgraph Dispatcher
        D --> O[peer_ctrl_msg_handler]
        O -- role=ELECTING --> P[vote_reply_decode_and_dispatch]
        O -- role=COMMITTING or PRIMARY --> K
        O -- other role --> Q[drop packet]
    end

Technology Stack

Layer	Choice	Role	Notes
HA election	ha.c (C, gnu17)	Endpoint lifecycle, dispatcher	All changes here
Cluster runtime	cluster.c (C)	standby_endpoints population, ctrlhello	Minor additions
iomgr	netio.c (C)	Unchanged	No modifications
Raft library	lib/raft/raft.c	Unchanged	No modifications

---

File Structure Plan

Modified Files

• src/controller/ha.c — All behavioral changes: peer_ctrl_msg_handler, vote_ep_ctx_t promoted lifetime, ha_vote_entry_t.ep tracking, raft_cb_become_leader promotion path, raft_cb_become_follower passive path + watchdog arm, remove fetch_config_from_primary delay task
• src/controller/ha.h — Add endpoint_t *ep to ha_vote_entry_t; add peer_ctrl_msg_handler forward declaration or make it accessible via a new test-seam hook; update comments on ha_find_and_connect_to_primary noting standby post-election path no longer calls it
• src/controller/cluster.c — handle_ctrlhello: set ha->primary_ctrl_fd = ep->fd when role == CTRL_STANDBY and ctrlhello arrives for the first time
• src/controller/testsuite/test_ha_election.c — Add tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello

---

System Flows

Election + Promotion (Winner Path)

sequenceDiagram
    participant C as Candidate ha.c
    participant I as iomgr
    participant P as Peer Standby

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>P: xce_request(OPCode_requestVote)
    note over C: ep->msg_handler = peer_ctrl_msg_handler
    P->>C: RequestVoteReply
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: vote_reply_decode_and_dispatch → raft_on_vote_reply
    C->>C: raft_cb_become_leader fires
    C->>C: votes[i].ep added to ha->standby_endpoints
    C->>P: push_config_to_standbys (on promoted ep)
    C->>P: ha_send_ctrlhello_to_all_standbys (on promoted ep)
    P->>C: OPCode_replCatchUpDone on same ep
    I->>C: peer_ctrl_msg_handler(role=COMMITTING)
    C->>C: ha_repl_ack_received → ha_promote_to_primary

Election + Passive Wait (Loser Path)

sequenceDiagram
    participant C as Loser ha.c
    participant I as iomgr
    participant W as Winner Primary

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>W: xce_request(OPCode_requestVote)
    W->>C: RequestVoteReply (denied)
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: raft_cb_become_follower fires
    C->>C: endpoint_fini(votes[i].ep) — close all vote endpoints
    C->>C: ha_arm_heartbeat_timeout_vtask
    note over C: CTRL_STANDBY — passive wait on listener
    W->>C: ctrlhello arrives on listener (existing iomgr path)
    C->>C: handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd
    C->>C: ha_on_heartbeat_received resets watchdog deadline

---

Requirements Traceability

Requirement	Summary	Component	Flow
1.1	Candidate establishes one outbound TLS connection per peer for vote RPC	raft_cb_send_vote_request (unchanged)	Election Phase
1.2	Candidate reuses each surviving outbound vote connection as persistent peer channel	raft_cb_become_leader promotes ep; peer_ctrl_msg_handler dispatches by role	Promotion
1.3	Winner promotes vote connections to standby replication channels	raft_cb_become_leader: add votes[i].ep to standby_endpoints	Winner Path
1.4	Loser closes all outbound vote connections	raft_cb_become_follower: endpoint_fini(votes[i].ep)	Loser Path
2.1	CTRL_STANDBY does not initiate outbound TCP connections	Remove fetch_config_from_primary delay task from raft_cb_become_follower	Loser Path
2.2	Standby accepts ctrlhello / config / WAL on inbound connection from winner	Existing vtaskmgr path unchanged; handle_ctrlhello sets primary_ctrl_fd	Standby Passive
2.3	No heartbeat from primary within timeout → new election	ha_arm_heartbeat_timeout_vtask called in raft_cb_become_follower	Loser Path
2.4	CTRL_STANDBY does not invoke tier-1/2/3 discovery	Remove fetch_config_from_primary call; ha_find_and_connect_to_primary not called	Loser Path
3.1	First ctrlhello on promoted connection → designate as primary control channel	handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd when role == CTRL_STANDBY	Standby Passive
3.2	Primary control channel RDHUP → detect within one epoll cycle → CTRL_ELECTING	Existing ha_on_primary_ctrl_fd_rdhup unchanged; armed from ctrlhello ep	Standby Passive
3.3	Arm heartbeat watchdog with same semantics	ha_arm_heartbeat_timeout_vtask called in raft_cb_become_follower	Loser Path
4.1	Loser explicitly closes all outbound vote endpoints on election loss	raft_cb_become_follower: iterate ha->elect_ctx->votes[i].ep, call endpoint_fini	Loser Path
4.2	Failed outbound vote connection → count peer as denied, no reconnect	Existing vote_ep_err_handler unchanged	Error
4.3	New election → fresh outbound connections	ha_start_election_async creates new endpoints each time (unchanged)	Election Phase
5.1	Primary pushes config via promoted connection	push_config_to_standbys uses standby_endpoints (populated from promoted eps)	Winner Path
5.2	WAL replication via promoted connections	Existing catch-up path uses standby_endpoints	Winner Path
5.3	Single-node self-elect without peer connections	Existing raft library single-node path unchanged	Election Phase
5.4	Committing timeout → degraded PRIMARY	Existing committing_timeout_fn unchanged	Winner Path
5.5	No new ASAN errors	Promoted ep lifetime owned by vote_ep_err_handler / raft_cb_become_follower cleanup	All

---

Components and Interfaces

Component	File	Intent	Req Coverage
peer_ctrl_msg_handler	ha.c	State-aware dispatcher; replaces vote_reply_msg_handler	1.2
ha_vote_entry_t.ep field	ha.h	Track per-peer outgoing endpoint for promotion/close	1.3, 1.4, 4.1
raft_cb_become_leader (extended)	ha.c	Promote vote eps → standby_endpoints	1.3, 5.1, 5.2
raft_cb_become_follower (extended)	ha.c	Close vote eps, arm watchdog, remove fetch_config call	1.4, 2.1, 2.3, 2.4, 3.3
handle_ctrlhello (extended)	cluster.c	Set primary_ctrl_fd from ctrlhello ep	3.1, 3.2

HA Election Layer

peer_ctrl_msg_handler

Field	Detail
Intent	Single iomgr message handler for outgoing vote endpoints; routes by current role
Requirements	1.2
File	src/controller/ha.c

Responsibilities & Constraints

• Set once in on_vote_connected; never reassigned
• Reads ha->role atomically at dispatch time
• Routes CTRL_ELECTING to vote reply decode (existing vote_reply_msg_handler logic, factored out as vote_reply_decode_and_dispatch)
• Routes CTRL_PRIMARY_COMMITTING / CTRL_PRIMARY to repl-ack decode (repl_ack_decode_and_dispatch, calls ha_repl_ack_received)
• Any other role: drop packet (free packet, return 0)
• Runs in iomgr thread — must not block

Contracts: State [ x ]

State Management

/* Signature: matches msg_handler_t = int (*)(void*) */
static int peer_ctrl_msg_handler(void *ep_raw);

/* Internal helpers (static, same file) */
static int vote_reply_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx);
static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx);

• State model: branches on atomic_load_explicit(&ha->role, memory_order_acquire)
• ELECTING path: existing vote-reply capnp decode → vote_reply_dispatch → raft_on_vote_reply
• COMMITTING/PRIMARY path: decode OPCode_replCatchUpDone → ha_repl_ack_received(ha, c)
• Other: free packet, no-op

Implementation Notes

• vote_reply_decode_and_dispatch is exactly the body of the existing vote_reply_msg_handler (factored out); the XCE_TESTING term-fencing and xce_vote_reply_hook seams are preserved
• repl_ack_decode_and_dispatch decodes an inbound capnp packet expecting OPCode_replCatchUpDone then calls ha_repl_ack_received; the opcode check guards against unexpected packets on the promoted endpoint during the COMMITTING phase

raft_cb_become_leader (extended)

Field	Detail
Intent	After election win: populate ha->standby_endpoints from promoted vote endpoints
Requirements	1.3, 5.1, 5.2
File	src/controller/ha.c

Responsibilities & Constraints

• Read ha->elect_ctx to get ha_elect_ctx_t*
• Iterate ctx->votes[i].ep for i in [0, ctx->n_peers)
• If votes[i].ep != NULL (connect succeeded): add to ha->standby_endpoints under ha->lock
• Existing push_config_to_standbys and ha_send_ctrlhello_to_all_standbys calls are unchanged; they iterate the now-populated array

Implementation Notes

• votes[0] is reserved for self in the raft library; peer endpoints start at index 1. Verify index mapping against raft_cb_send_vote_request peer_idx.
• If ha->elect_ctx is NULL when raft_cb_become_leader fires (e.g., single-node), skip promotion loop (no peers to promote).
• Risk: votes[i].ep may be NULL if the peer's connect failed; the NULL check guards this.

raft_cb_become_follower (extended)

Field	Detail
Intent	After election loss: close vote endpoints, arm heartbeat watchdog, remove fetch_config call
Requirements	1.4, 2.1, 2.3, 2.4, 3.3
File	src/controller/ha.c

Responsibilities & Constraints

• Read ha->elect_ctx, iterate votes[i].ep, call endpoint_fini(votes[i].ep) for non-NULL entries, set votes[i].ep = NULL
• Do NOT call ha_find_and_connect_to_primary or schedule fetch_config_from_primary
• Call ha_arm_heartbeat_timeout_vtask(ha, c) after role is set to CTRL_STANDBY
• Existing cluster_init_phase2 call and ha->phase2_done guard are preserved

Implementation Notes

• endpoint_fini closes the fd and releases the endpoint slot; iomgr is notified via epoll RDHUP on the peer side, which is expected
• Watchdog arm must happen after ha->role = CTRL_STANDBY is stored (role gate in watchdog logic)
• XCE_TESTING: ha_arm_heartbeat_timeout_vtask is a no-op when c == NULL; the test seam is preserved

Cluster Runtime Layer

handle_ctrlhello (extended)

Field	Detail
Intent	Set ha->primary_ctrl_fd when ctrlhello arrives on standby's inbound connection
Requirements	3.1, 3.2, 3.3
File	src/controller/cluster.c

Responsibilities & Constraints

• Existing ctrlhello handler already processes OP_hello and triggers WAL catch-up
• Add: if ha->role == CTRL_STANDBY and ha->primary_ctrl_fd == -1, set ha->primary_ctrl_fd = ep->fd
• Call ha_on_heartbeat_received(ha) to reset the watchdog deadline (already done by heartbeat path; add to ctrlhello path for first message)
• No other changes to the ctrlhello handler

Implementation Notes

• Guard with primary_ctrl_fd == -1 for idempotency; a repeat ctrlhello on the same connection does not need re-registration
• ha_on_primary_ctrl_fd_rdhup already handles fd close and triggers re-election; no changes needed there

---

Data Models

ha_vote_entry_t (modified)

typedef struct {
    xce_candidate_t *peer;          /* Peer being solicited */
    uint64_t         my_term;       /* Term sent in OP_requestVote */
    uint64_t         my_tag;        /* config_tag sent in request */
    uint64_t         my_lsn;        /* WAL LSN sent in request */
    int              vote_granted;  /* 1 if peer granted vote */
    int              replied;       /* 1 if peer replied */
    uint64_t         peer_tag;      /* peer's config_tag from reply */
    uint64_t         peer_lsn;      /* peer's WAL LSN from reply */
    struct endpoint_t *ep;          /* NEW: outgoing vote endpoint; NULL if connect failed */
} ha_vote_entry_t;

Invariants:

• ep is set by raft_cb_send_vote_request when iomgr_monitor succeeds
• ep is NULL if endpoint_connect or iomgr_monitor failed for this peer
• Winner: ep is transferred to standby_endpoints; winner does NOT call endpoint_fini on promoted eps
• Loser: ep is closed via endpoint_fini in raft_cb_become_follower; set to NULL after close

vote_ep_ctx_t (unchanged)

No structural changes. The promoted flag is NOT added — role is read from ha->role at dispatch time, which is the authoritative state. The ctx carries ha, c, peer_idx which are sufficient for both phases.

---

Error Handling

Error Strategy

Treat connection failures and packet decode errors identically to the existing vote path: count the peer as denied (or drop the packet for non-vote phases).

Error Categories

Connect failure (vote_ep_err_handler):

• Existing behavior preserved: handle_request_vote_reply(..., 0, ...) counts as denied vote
• Promoted connection disconnect: vote_ep_err_handler fires when the standby drops. For COMMITTING/PRIMARY role, remove ep from ha->standby_endpoints under ha->lock; call ha_repl_ack_received or degrade gracefully based on n_peers

Packet decode error in repl_ack_decode_and_dispatch:

• Drop packet, return 0 (iomgr continues monitoring endpoint)
• Log WARN with opcode for debugging

Unexpected role in peer_ctrl_msg_handler:

• Drop packet, return 0 (CTRL_INIT, CTRL_STANDBY on outgoing ep: should not happen normally)
• Log WARN

Monitoring

• Add INFO log in raft_cb_become_leader for each promoted endpoint: "HA: promoted vote ep[%d] → standby channel"
• Add INFO log in raft_cb_become_follower for endpoint close: "HA: closing vote ep[%d] (loser path)"
• Existing heartbeat watchdog log messages unchanged

---

Testing Strategy

Unit Tests (test_ha_election.c)

1. Dispatcher routes by role: set ep->msg_handler = peer_ctrl_msg_handler; inject packet with role=ELECTING → verify xce_vote_reply_hook fires; inject packet with role=COMMITTING → verify repl-ack hook fires; inject with role=STANDBY → verify packet dropped
2. Winner promotes endpoints: call raft_cb_become_leader with ha->elect_ctx populated; verify ha->standby_endpoints->count == n_peers_connected
3. Loser closes endpoints: call raft_cb_become_follower; verify votes[i].ep == NULL for all i; verify endpoint_fini called (ASAN validates no leak)
4. Watchdog armed on CTRL_STANDBY: mock ha_arm_heartbeat_timeout_vtask via test hook; verify it is called exactly once from raft_cb_become_follower
5. fetch_config_from_primary NOT called: verify no delay_task_ms call for fetch_config_from_primary in raft_cb_become_follower (use test seam to intercept delay_task_ms)
6. primary_ctrl_fd set on ctrlhello: call handle_ctrlhello with role=CTRL_STANDBY and primary_ctrl_fd=-1; verify primary_ctrl_fd == ep->fd
7. Promoted ep err_handler removes from standby_endpoints: populate standby_endpoints with a mock ep; call vote_ep_err_handler with role=COMMITTING; verify ep removed from array

Integration Tests (test_ha_bootstrap.c / test_p3_integration.c)

1. Two-node election → standby receives ctrlhello without outbound connect: boot two nodes; verify standby never calls ha_find_and_connect_to_primary; verify ctrlhello arrives within election_timeout_ms + 200ms
2. Promoted connection carries WAL replication: verify OPCode_replCatchUpDone decoded on promoted ep increments repl_ack_count
3. Primary failover → fresh connections on re-election: kill primary; verify standby transitions CTRL_ELECTING and creates fresh endpoints (not reusing closed ones)