myxce/.kiro/specs/ha-peer-connection-reuse/design.md

Design Document: ha-peer-connection-reuse

Overview

The XCE controller election currently discards outgoing vote connections after the vote exchange, forcing standbys to create new outbound connections to the primary for the persistent control channel. This produces a 10-second heartbeat timeout delay before standbys detect the primary, spurious retry warnings from premature config-fetch attempts, and a complex tier-1/2/3 primary discovery path.

This spec reuses each candidate's outgoing vote endpoint as the persistent peer control channel: the election winner promotes its connections to replication channels; the loser closes its connections and waits passively for the primary's ctrlhello to arrive on its listener.

The entire change is confined to ha.c and cluster.c. iomgr, the capnp wire protocol, and the raft library are not modified.

Goals

• Eliminate the double-connection round-trip after election
• Remove the standby's active primary discovery path (fetch_config_from_primary)
• Arm the standby heartbeat watchdog immediately when CTRL_STANDBY is entered
• Preserve existing HA behaviors: failover, WAL replication, config push, heartbeat detection

Non-Goals

• SSL/TLS certificate management changes
• Vote grant/deny protocol logic changes
• WAL entry format or replication content changes
• Config file format changes
• Cluster membership changes (add/remove peers)

---

Boundary Commitments

This Spec Owns

• The lifecycle of outgoing vote endpoints from creation through promotion or close
• The peer_ctrl_msg_handler state-aware dispatcher (replaces vote_reply_msg_handler + explicit reassignment)
• Population of ha->standby_endpoints from promoted vote connections on the primary side
• Arming the standby heartbeat watchdog from raft_cb_become_follower instead of after ha_find_and_connect_to_primary
• Setting ha->primary_ctrl_fd from the inbound ctrlhello connection on the standby side
• Removing the fetch_config_from_primary delay task from the standby path

Out of Boundary

• SSL/TLS negotiation details — handled by existing client_ssl_ctx / on_vote_connected path
• iomgr internals — zero changes to src/lib/netio.c or src/include/netio.h
• Raft library — zero changes to src/lib/raft/raft.c
• Heartbeat watchdog thread implementation — only the arming callsite changes
• The existing ha_find_and_connect_to_primary function — originally scoped to remain for future use; drifted: it was removed entirely during implementation (exceeds this design's original intent — see this spec's own research.md "Gap Re-Audit 2026-06-18", finding R4). The standby path no longer calls it, and the function no longer exists in the tree.

Allowed Dependencies

• ha_state_t, ha_elect_ctx_t, ha_vote_entry_t — extended with new fields
• vote_ep_ctx_t — reused as-is for promoted connections
• endpoint_t, iomgr_monitor, endpoint_set_msg_handler — used unchanged
• ha->standby_endpoints (zarray) — populated from promoted connections
• raft_cb_become_leader / raft_cb_become_follower — extended to populate/close endpoints
• ha_arm_heartbeat_timeout_vtask — called from raft_cb_become_follower

Revalidation Triggers

• Changing ha_vote_entry_t struct layout (tasks depending on votes[].ep)
• Changing vote_ep_ctx_t fields or lifetime contract
• Changing raft_cb_become_leader / raft_cb_become_follower signatures
• Adding new roles to ctrl_role_t (the dispatcher switch must be updated)
• Changing ha->standby_endpoints population semantics

---

Architecture

Existing Architecture Analysis

The current election flow:

1. Candidate creates one endpoint_t per peer (outgoing vote), registers with iomgr
2. on_vote_connected sends the vote request and reassigns ep->msg_handler = vote_reply_msg_handler
3. After the election, the outgoing vote endpoints are closed by ha_election_cleanup
4. The winning standby calls ha_find_and_connect_to_primary to create a new inbound connection to the primary
5. The heartbeat watchdog is armed after this reconnection succeeds

The promoted-connection flow (this spec):

1. Same vote endpoint creation
2. on_vote_connected sets ep->msg_handler = peer_ctrl_msg_handler (state-aware, never reassigned)
3. In raft_cb_become_leader: each surviving vote endpoint is added to ha->standby_endpoints (primary)
4. In raft_cb_become_follower: outgoing vote endpoints are closed; standby waits passively; watchdog armed immediately
5. Primary's ctrlhello arrives on the standby's listener, setting ha->primary_ctrl_fd

Architecture Pattern & Boundary Map

graph TD
    subgraph Election_Phase
        A[raft_cb_send_vote_request] --> B[endpoint_connect + iomgr_monitor]
        B --> C[on_vote_connected sets peer_ctrl_msg_handler]
        C --> D[send RequestVote via xce_request]
    end

    subgraph Election_Result
        E[raft_cb_become_leader] --> F[promote: add ep to standby_endpoints]
        G[raft_cb_become_follower] --> H[close: endpoint_fini each vote ep]
        G --> I[arm heartbeat watchdog]
    end

    subgraph Promoted_Connection_PRIMARY
        F --> J[peer_ctrl_msg_handler role=PRIMARY]
        J --> K[repl_ack_decode_and_dispatch: no-op ack, INFO log]
    end
    %% Note: CTRL_PRIMARY_COMMITTING was removed (superseded by ha-consistency-invariants /
    %% ha-single-leader-role, 2026-06-11); ha_repl_ack_received no longer exists -- promotion
    %% is now driven by the heartbeat decision loop + commit engine, not ack-counting on this ep.

    subgraph Standby_Passive
        H --> L[wait for ctrlhello on listener]
        L --> M[handle_ctrlhello sets primary_ctrl_fd]
        I --> N[watchdog arm: fires on silence timeout]
    end

    subgraph Dispatcher
        D --> O[peer_ctrl_msg_handler]
        O -- role=ELECTING --> P[vote_reply_decode_and_dispatch]
        O -- role=PRIMARY --> K
        O -- role=STANDBY + PreVote round --> R[pre_vote_reply_decode_and_dispatch]
        O -- other role --> Q[drop packet]
    end
    %% pre_vote_reply_decode_and_dispatch branch added later by ha-prevote (2026-07-18) --
    %% shares this dispatcher but was not part of this spec's original scope.

Technology Stack

Layer	Choice	Role	Notes
HA election	ha.c (C, gnu17)	Endpoint lifecycle, dispatcher	All changes here
Cluster runtime	cluster.c (C)	standby_endpoints population, ctrlhello	Minor additions
iomgr	netio.c (C)	Unchanged	No modifications
Raft library	lib/raft/raft.c	Unchanged	No modifications

---

File Structure Plan

Modified Files

• src/controller/ha.c — All behavioral changes: peer_ctrl_msg_handler, vote_ep_ctx_t promoted lifetime, ha_vote_entry_t.ep tracking, raft_cb_become_leader promotion path, raft_cb_become_follower passive path + watchdog arm, remove fetch_config_from_primary delay task
• src/controller/ha.h — Add endpoint_t *ep to ha_vote_entry_t; add peer_ctrl_msg_handler forward declaration or make it accessible via a new test-seam hook; update comments on ha_find_and_connect_to_primary noting standby post-election path no longer calls it
• src/controller/config_repl.c — handle_ctrlhello: set ha->primary_ctrl_fd = ep->fd when role == CTRL_STANDBY and ctrlhello arrives for the first time (Note: this design originally cited cluster.c; handle_ctrlhello actually lives in config_repl.c — drift already flagged in this spec's own research.md R4, corrected here)
• src/controller/testsuite/test_ha_connection_reuse.c — Tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello (Note: implemented under this filename, not test_ha_election.c as originally planned — that file is a pre-existing, unrelated quorum-election test suite)

---

System Flows

Election + Promotion (Winner Path)

sequenceDiagram
    participant C as Candidate ha.c
    participant I as iomgr
    participant P as Peer Standby

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>P: xce_request(OPCode_requestVote)
    note over C: ep->msg_handler = peer_ctrl_msg_handler
    P->>C: RequestVoteReply
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: vote_reply_decode_and_dispatch → raft_on_vote_reply
    C->>C: raft_cb_become_leader fires
    C->>C: votes[i].ep added to ha->standby_endpoints
    C->>P: push_config_to_standbys (on promoted ep)
    C->>P: ha_send_ctrlhello_to_all_standbys (on promoted ep)
    C->>C: repl_add_standby registers promoted ep with WAL replicator (ha-commit-engine addition)
    P->>C: OPCode_replCatchUpDone on same ep
    I->>C: peer_ctrl_msg_handler(role=PRIMARY)
    C->>C: repl_ack_decode_and_dispatch: replCatchUpDone is now a no-op (ack-counting removed)

Note: the original role=COMMITTING step and ha_repl_ack_received → ha_promote_to_primary ack-counting tail no longer exist — CTRL_PRIMARY_COMMITTING and ha_repl_ack_received were deleted by ha-consistency-invariants/ha-single-leader-role (2026-06-11); promotion is now driven by the heartbeat decision loop + commit engine (ha-commit-engine, ha-repl-hardening), not by counting replCatchUpDone acks on the promoted ep. See those specs' design.md.

Election + Passive Wait (Loser Path)

sequenceDiagram
    participant C as Loser ha.c
    participant I as iomgr
    participant W as Winner Primary

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>W: xce_request(OPCode_requestVote)
    W->>C: RequestVoteReply (denied)
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: raft_cb_become_follower fires
    C->>C: endpoint_fini(votes[i].ep) — close all vote endpoints
    C->>C: ha_arm_heartbeat_timeout_vtask
    note over C: CTRL_STANDBY — passive wait on listener
    W->>C: ctrlhello arrives on listener (existing iomgr path)
    C->>C: handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd
    C->>C: ha_on_heartbeat_received resets watchdog deadline

---

Requirements Traceability

Requirement	Summary	Component	Flow
1.1	Candidate establishes one outbound TLS connection per peer for vote RPC	raft_cb_send_vote_request (unchanged)	Election Phase
1.2	Candidate reuses each surviving outbound vote connection as persistent peer channel	raft_cb_become_leader promotes ep; peer_ctrl_msg_handler dispatches by role	Promotion
1.3	Winner promotes vote connections to standby replication channels	raft_cb_become_leader: add votes[i].ep to standby_endpoints	Winner Path
1.4	Loser closes all outbound vote connections	raft_cb_become_follower: endpoint_fini(votes[i].ep)	Loser Path
2.1	CTRL_STANDBY does not initiate outbound TCP connections	Remove fetch_config_from_primary delay task from raft_cb_become_follower	Loser Path
2.2	Standby accepts ctrlhello / config / WAL on inbound connection from winner	Existing vtaskmgr path unchanged; handle_ctrlhello sets primary_ctrl_fd	Standby Passive
2.3	No heartbeat from primary within timeout → new election	ha_arm_heartbeat_timeout_vtask called in raft_cb_become_follower	Loser Path
2.4	CTRL_STANDBY does not invoke tier-1/2/3 discovery	Remove fetch_config_from_primary call; ha_find_and_connect_to_primary not called	Loser Path
3.1	First ctrlhello on promoted connection → designate as primary control channel	handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd when role == CTRL_STANDBY	Standby Passive
3.2	Primary control channel RDHUP → detect within one epoll cycle → CTRL_ELECTING	Existing ha_on_primary_ctrl_fd_rdhup unchanged; armed from ctrlhello ep	Standby Passive
3.3	Arm heartbeat watchdog with same semantics	ha_arm_heartbeat_timeout_vtask called in raft_cb_become_follower	Loser Path
4.1	Loser explicitly closes all outbound vote endpoints on election loss	raft_cb_become_follower: iterate ha->elect_ctx->votes[i].ep, call endpoint_fini	Loser Path
4.2	Failed outbound vote connection → count peer as denied, no reconnect	Existing vote_ep_err_handler unchanged	Error
4.3	New election → fresh outbound connections	ha_start_election_async creates new endpoints each time (unchanged)	Election Phase
5.1	Primary pushes config via promoted connection	push_config_to_standbys uses standby_endpoints (populated from promoted eps)	Winner Path
5.2	WAL replication via promoted connections	Existing catch-up path uses standby_endpoints; majority-ack commit tracking now also registers the promoted ep with the replicator via repl_add_standby (Note: added by ha-commit-engine, 2026-06-12 — see that spec's design.md)	Winner Path
5.3	Single-node self-elect without peer connections	Existing raft library single-node path unchanged	Election Phase
5.4	Committing timeout → degraded PRIMARY	committing_timeout_fn no longer exists (Note: deleted along with CTRL_PRIMARY_COMMITTING by ha-consistency-invariants, 2026-06-11; replaced by promotion-fence/own-term-gate machinery — ha_arm_own_term_gate, ha_arm_promotion_fence_timer — see that spec's design.md)	Winner Path
5.5	No new ASAN errors	Promoted ep lifetime owned by vote_ep_err_handler / raft_cb_become_follower cleanup	All

---

Components and Interfaces

Component	File	Intent	Req Coverage
peer_ctrl_msg_handler	ha.c	State-aware dispatcher; replaces vote_reply_msg_handler	1.2
ha_vote_entry_t.ep field	ha.h	Track per-peer outgoing endpoint for promotion/close	1.3, 1.4, 4.1
raft_cb_become_leader (extended)	ha.c	Promote vote eps → standby_endpoints	1.3, 5.1, 5.2
raft_cb_become_follower (extended)	ha.c	Close vote eps, arm watchdog, remove fetch_config call	1.4, 2.1, 2.3, 2.4, 3.3
handle_ctrlhello (extended)	config_repl.c (design originally cited cluster.c — corrected)	Set primary_ctrl_fd from ctrlhello ep	3.1, 3.2

HA Election Layer

peer_ctrl_msg_handler

Field	Detail
Intent	Single iomgr message handler for outgoing vote endpoints; routes by current role
Requirements	1.2
File	src/controller/ha.c

Responsibilities & Constraints

• Set once in on_vote_connected; never reassigned
• Reads ha->role atomically at dispatch time
• Routes CTRL_ELECTING to vote reply decode (existing vote_reply_msg_handler logic, factored out as vote_reply_decode_and_dispatch)
• Routes CTRL_PRIMARY to repl-ack decode (repl_ack_decode_and_dispatch). (Note: CTRL_PRIMARY_COMMITTING no longer exists — removed by ha-consistency-invariants/ha-single-leader-role, 2026-06-11; repl_ack_decode_and_dispatch's OPCode_replCatchUpDone branch is now an INFO-logged no-op and ha_repl_ack_received was deleted along with the ack-counting promotion path — see those specs' design.md)
• Also routes CTRL_STANDBY with an in-progress PreVote round to pre_vote_reply_decode_and_dispatch — added later by ha-prevote (2026-07-18), not part of this spec's original scope
• Any other role: drop packet (free packet, return 0)
• Runs in iomgr thread — must not block, for the default (dispatch-flag-OFF) path; raft-drive-async (2026-07-18) later added an optional vtask-pool dispatch mode (ha_peerctrl_dispatch_enabled) that can run this body off the iomgr thread — see that spec's design.md

Contracts: State [ x ]

State Management

/* Signature: matches msg_handler_t = int (*)(void*) */
static int peer_ctrl_msg_handler(void *ep_raw);

/* Internal helpers (static, same file) — current signatures (corrected; both
 * gained an io_packet_t *pkt parameter, and vote_reply_decode_and_dispatch
 * dropped its endpoint_t *ep parameter, as part of the raft-drive-async
 * pkt-ownership refactor, 2026-07-18) */
static int vote_reply_decode_and_dispatch(vote_ep_ctx_t *vctx, io_packet_t *pkt);
static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx, io_packet_t *pkt);

• State model: branches on atomic_load_explicit(&ha->role, memory_order_acquire)
• ELECTING path: existing vote-reply capnp decode → vote_reply_dispatch → raft_on_vote_reply
• PRIMARY path: decode OPCode_replCatchUpDone — now a no-op INFO log (ack-counting/ha_repl_ack_received removed, see note above); the same handler also decodes heartbeat-reply stateReport for convergence tracking
• Other: free packet, no-op

Implementation Notes

• vote_reply_decode_and_dispatch is exactly the body of the existing vote_reply_msg_handler (factored out); the XCE_TESTING term-fencing and xce_vote_reply_hook seams are preserved
• repl_ack_decode_and_dispatch decodes an inbound capnp packet; OPCode_replCatchUpDone is now a no-op (superseded — see note above), and it separately decodes heartbeat-reply stateReport messages for convergence tracking. The original "COMMITTING phase" framing no longer applies since that role was removed.

raft_cb_become_leader (extended)

Field	Detail
Intent	After election win: populate ha->standby_endpoints from promoted vote endpoints
Requirements	1.3, 5.1, 5.2
File	src/controller/ha.c

Responsibilities & Constraints

• Read ha->elect_ctx to get ha_elect_ctx_t*
• Iterate ctx->votes[i].ep for i in [0, ctx->n_peers)
• If votes[i].ep != NULL (connect succeeded): add to ha->standby_endpoints under ha->lock
• Existing push_config_to_standbys and ha_send_ctrlhello_to_all_standbys calls are unchanged; they iterate the now-populated array
• (Note: ha-commit-engine, 2026-06-12, added a further registration in this same loop — each promoted ep is also registered with the WAL replicator via repl_add_standby(&c->persist.repl, ...), which is what majority-ack commit tracking now uses; standby_endpoints remains the config-push/ctrlhello target only. See that spec's design.md.)

Implementation Notes

• votes[0] is reserved for self in the raft library; peer endpoints start at index 1. Verify index mapping against raft_cb_send_vote_request peer_idx.
• If ha->elect_ctx is NULL when raft_cb_become_leader fires (e.g., single-node), skip promotion loop (no peers to promote).
• Risk: votes[i].ep may be NULL if the peer's connect failed; the NULL check guards this.

raft_cb_become_follower (extended)

Field	Detail
Intent	After election loss: close vote endpoints, arm heartbeat watchdog, remove fetch_config call
Requirements	1.4, 2.1, 2.3, 2.4, 3.3
File	src/controller/ha.c

Responsibilities & Constraints

• Read ha->elect_ctx, iterate votes[i].ep, call endpoint_fini(votes[i].ep) for non-NULL entries, set votes[i].ep = NULL
• Do NOT call ha_find_and_connect_to_primary or schedule fetch_config_from_primary
• Call ha_arm_heartbeat_timeout_vtask(ha, c) after role is set to CTRL_STANDBY
• Existing cluster_init_phase2 call and ha->phase2_done guard are preserved

Implementation Notes

• endpoint_fini closes the fd and releases the endpoint slot; iomgr is notified via epoll RDHUP on the peer side, which is expected
• Watchdog arm must happen after ha->role = CTRL_STANDBY is stored (role gate in watchdog logic)
• XCE_TESTING: ha_arm_heartbeat_timeout_vtask is a no-op when c == NULL; the test seam is preserved

Cluster Runtime Layer

handle_ctrlhello (extended)

Field	Detail
Intent	Set ha->primary_ctrl_fd when ctrlhello arrives on standby's inbound connection
Requirements	3.1, 3.2, 3.3
File	src/controller/cluster.c

Responsibilities & Constraints

• Existing ctrlhello handler already processes OP_hello and triggers WAL catch-up
• Add: if ha->role == CTRL_STANDBY and ha->primary_ctrl_fd == -1, set ha->primary_ctrl_fd = ep->fd
• Call ha_on_heartbeat_received(ha) to reset the watchdog deadline (already done by heartbeat path; add to ctrlhello path for first message)
• No other changes to the ctrlhello handler

Implementation Notes

• Guard with primary_ctrl_fd == -1 for idempotency; a repeat ctrlhello on the same connection does not need re-registration
• ha_on_primary_ctrl_fd_rdhup already handles fd close and triggers re-election; no changes needed there

---

Data Models

ha_vote_entry_t (modified)

typedef struct {
    xce_candidate_t *peer;          /* Peer being solicited */
    uint64_t         my_term;       /* Term sent in OP_requestVote */
    uint64_t         my_tag;        /* config_tag sent in request */
    uint64_t         my_lsn;        /* WAL LSN sent in request */
    int              vote_granted;  /* 1 if peer granted vote */
    int              replied;       /* 1 if peer replied */
    uint64_t         peer_tag;      /* peer's config_tag from reply */
    uint64_t         peer_lsn;      /* peer's WAL LSN from reply */
    struct endpoint_t *ep;          /* NEW: outgoing vote endpoint; NULL if connect failed */
} ha_vote_entry_t;

Invariants:

• ep is set by raft_cb_send_vote_request when iomgr_monitor succeeds
• ep is NULL if endpoint_connect or iomgr_monitor failed for this peer
• Winner: ep is transferred to standby_endpoints; winner does NOT call endpoint_fini on promoted eps
• Loser: ep is closed via endpoint_fini in raft_cb_become_follower; set to NULL after close

vote_ep_ctx_t

The promoted flag is NOT added — role is read from ha->role at dispatch time, which is the authoritative state, as this design intended. (Note: despite the original "(unchanged)" heading, the struct is no longer literally unchanged since this spec closed — ha-term-safety (2026-07-17) added my_last_log_term/my_last_log_index fields, and ha-elect-ctx-lifecycle (2026-07-18) changed its lifetime contract: the endpoint's on_fini hook now calls ha_elect_ctx_release to decrement a refcount on the owning ha_elect_ctx_t, rather than the simple election-cleanup-only lifetime this design describes. See those specs' design.md.)

---

Error Handling

Error Strategy

Treat connection failures and packet decode errors identically to the existing vote path: count the peer as denied (or drop the packet for non-vote phases).

Error Categories

Connect failure (vote_ep_err_handler):

• Existing behavior preserved: a connect-time or post-connect failure counts as a denied vote through the production tally (vote_reply_dispatch), not the legacy handle_request_vote_reply path (hardened later by raft-adapter-conformance)
• Promoted connection disconnect: vote_ep_err_handler fires when the standby drops. For PRIMARY role (Note: the COMMITTING role named here was removed — see the peer_ctrl_msg_handler note above), remove ep from ha->standby_endpoints under ha->lock; current code also calls repl_remove_standby to deregister from the WAL replicator (ha-commit-engine addition) — ha_repl_ack_received no longer exists

Packet decode error in repl_ack_decode_and_dispatch:

• Drop packet, return 0 (iomgr continues monitoring endpoint)
• Log WARN with opcode for debugging

Unexpected role in peer_ctrl_msg_handler:

• Drop packet, return 0 (CTRL_INIT: should not happen normally; CTRL_STANDBY on outgoing ep is now an EXPECTED case when a PreVote round is in progress — routed to pre_vote_reply_decode_and_dispatch rather than dropped, added by ha-prevote, 2026-07-18)
• Log WARN

Monitoring

• Add INFO log in raft_cb_become_leader for each promoted endpoint: "HA: promoted vote ep[%d] → standby channel"
• Add INFO log in raft_cb_become_follower for endpoint close: "HA: closing vote ep[%d] (loser path)"
• Existing heartbeat watchdog log messages unchanged

---

Testing Strategy

Unit Tests (test_ha_election.c)

1. Dispatcher routes by role: set ep->msg_handler = peer_ctrl_msg_handler; inject packet with role=ELECTING → verify xce_vote_reply_hook fires; inject packet with role=COMMITTING → verify repl-ack hook fires; inject with role=STANDBY → verify packet dropped
2. Winner promotes endpoints: call raft_cb_become_leader with ha->elect_ctx populated; verify ha->standby_endpoints->count == n_peers_connected
3. Loser closes endpoints: call raft_cb_become_follower; verify votes[i].ep == NULL for all i; verify endpoint_fini called (ASAN validates no leak)
4. Watchdog armed on CTRL_STANDBY: mock ha_arm_heartbeat_timeout_vtask via test hook; verify it is called exactly once from raft_cb_become_follower
5. fetch_config_from_primary NOT called: verify no delay_task_ms call for fetch_config_from_primary in raft_cb_become_follower (use test seam to intercept delay_task_ms)
6. primary_ctrl_fd set on ctrlhello: call handle_ctrlhello with role=CTRL_STANDBY and primary_ctrl_fd=-1; verify primary_ctrl_fd == ep->fd
7. Promoted ep err_handler removes from standby_endpoints: populate standby_endpoints with a mock ep; call vote_ep_err_handler with role=COMMITTING; verify ep removed from array

Integration Tests (test_ha_bootstrap.c / test_p3_integration.c)

1. Two-node election → standby receives ctrlhello without outbound connect: boot two nodes; verify standby never calls ha_find_and_connect_to_primary; verify ctrlhello arrives within election_timeout_ms + 200ms
2. Promoted connection carries WAL replication: verify OPCode_replCatchUpDone decoded on promoted ep increments repl_ack_count
3. Primary failover → fresh connections on re-election: kill primary; verify standby transitions CTRL_ELECTING and creates fresh endpoints (not reusing closed ones)