myxce/.kiro/specs/ha-peer-connection-reuse/design.md

# Design Document: ha-peer-connection-reuse

## Overview

The XCE controller election currently discards outgoing vote connections after the vote exchange, forcing standbys to create new outbound connections to the primary for the persistent control channel. This produces a 10-second heartbeat timeout delay before standbys detect the primary, spurious retry warnings from premature config-fetch attempts, and a complex tier-1/2/3 primary discovery path.

This spec reuses each candidate's outgoing vote endpoint as the persistent peer control channel: the election winner promotes its connections to replication channels; the loser closes its connections and waits passively for the primary's ctrlhello to arrive on its listener.

The entire change is confined to `ha.c` and `cluster.c`. iomgr, the capnp wire protocol, and the raft library are not modified.

### Goals

- Eliminate the double-connection round-trip after election
- Remove the standby's active primary discovery path (fetch_config_from_primary)
- Arm the standby heartbeat watchdog immediately when CTRL_STANDBY is entered
- Preserve existing HA behaviors: failover, WAL replication, config push, heartbeat detection

### Non-Goals

- SSL/TLS certificate management changes
- Vote grant/deny protocol logic changes
- WAL entry format or replication content changes
- Config file format changes
- Cluster membership changes (add/remove peers)

---

## Boundary Commitments

### This Spec Owns

- The lifecycle of outgoing vote endpoints from creation through promotion or close
- The `peer_ctrl_msg_handler` state-aware dispatcher (replaces `vote_reply_msg_handler` + explicit reassignment)
- Population of `ha->standby_endpoints` from promoted vote connections on the primary side
- Arming the standby heartbeat watchdog from `raft_cb_become_follower` instead of after `ha_find_and_connect_to_primary`
- Setting `ha->primary_ctrl_fd` from the inbound ctrlhello connection on the standby side
- Removing the `fetch_config_from_primary` delay task from the standby path

### Out of Boundary

- SSL/TLS negotiation details — handled by existing `client_ssl_ctx` / `on_vote_connected` path
- iomgr internals — zero changes to `src/lib/netio.c` or `src/include/netio.h`
- Raft library — zero changes to `src/lib/raft/raft.c`
- Heartbeat watchdog thread implementation — only the arming callsite changes
- The existing `ha_find_and_connect_to_primary` function — originally scoped to remain for future use; **drifted**: it was removed entirely during implementation (exceeds this design's original intent — see this spec's own research.md "Gap Re-Audit 2026-06-18", finding R4). The standby path no longer calls it, and the function no longer exists in the tree.

### Allowed Dependencies

- `ha_state_t`, `ha_elect_ctx_t`, `ha_vote_entry_t` — extended with new fields
- `vote_ep_ctx_t` — reused as-is for promoted connections
- `endpoint_t`, `iomgr_monitor`, `endpoint_set_msg_handler` — used unchanged
- `ha->standby_endpoints` (zarray) — populated from promoted connections
- `raft_cb_become_leader` / `raft_cb_become_follower` — extended to populate/close endpoints
- `ha_arm_heartbeat_timeout_vtask` — called from `raft_cb_become_follower`

### Revalidation Triggers

- Changing `ha_vote_entry_t` struct layout (tasks depending on `votes[].ep`)
- Changing `vote_ep_ctx_t` fields or lifetime contract
- Changing `raft_cb_become_leader` / `raft_cb_become_follower` signatures
- Adding new roles to `ctrl_role_t` (the dispatcher switch must be updated)
- Changing `ha->standby_endpoints` population semantics

---

## Architecture

### Existing Architecture Analysis

The current election flow:
1. Candidate creates one `endpoint_t` per peer (outgoing vote), registers with iomgr
2. `on_vote_connected` sends the vote request and reassigns `ep->msg_handler = vote_reply_msg_handler`
3. After the election, the outgoing vote endpoints are closed by `ha_election_cleanup`
4. The winning standby calls `ha_find_and_connect_to_primary` to create a new inbound connection to the primary
5. The heartbeat watchdog is armed after this reconnection succeeds

The promoted-connection flow (this spec):
1. Same vote endpoint creation
2. `on_vote_connected` sets `ep->msg_handler = peer_ctrl_msg_handler` (state-aware, never reassigned)
3. In `raft_cb_become_leader`: each surviving vote endpoint is added to `ha->standby_endpoints` (primary)
4. In `raft_cb_become_follower`: outgoing vote endpoints are closed; standby waits passively; watchdog armed immediately
5. Primary's ctrlhello arrives on the standby's listener, setting `ha->primary_ctrl_fd`

### Architecture Pattern & Boundary Map

```mermaid
graph TD
    subgraph Election_Phase
        A[raft_cb_send_vote_request] --> B[endpoint_connect + iomgr_monitor]
        B --> C[on_vote_connected sets peer_ctrl_msg_handler]
        C --> D[send RequestVote via xce_request]
    end

    subgraph Election_Result
        E[raft_cb_become_leader] --> F[promote: add ep to standby_endpoints]
        G[raft_cb_become_follower] --> H[close: endpoint_fini each vote ep]
        G --> I[arm heartbeat watchdog]
    end

    subgraph Promoted_Connection_PRIMARY
        F --> J[peer_ctrl_msg_handler role=PRIMARY]
        J --> K[repl_ack_decode_and_dispatch: no-op ack, INFO log]
    end
    %% Note: CTRL_PRIMARY_COMMITTING was removed (superseded by ha-consistency-invariants /
    %% ha-single-leader-role, 2026-06-11); ha_repl_ack_received no longer exists -- promotion
    %% is now driven by the heartbeat decision loop + commit engine, not ack-counting on this ep.

    subgraph Standby_Passive
        H --> L[wait for ctrlhello on listener]
        L --> M[handle_ctrlhello sets primary_ctrl_fd]
        I --> N[watchdog arm: fires on silence timeout]
    end

    subgraph Dispatcher
        D --> O[peer_ctrl_msg_handler]
        O -- role=ELECTING --> P[vote_reply_decode_and_dispatch]
        O -- role=PRIMARY --> K
        O -- role=STANDBY + PreVote round --> R[pre_vote_reply_decode_and_dispatch]
        O -- other role --> Q[drop packet]
    end
    %% pre_vote_reply_decode_and_dispatch branch added later by ha-prevote (2026-07-18) --
    %% shares this dispatcher but was not part of this spec's original scope.
```

### Technology Stack

| Layer | Choice | Role | Notes |
|-------|--------|------|-------|
| HA election | ha.c (C, gnu17) | Endpoint lifecycle, dispatcher | All changes here |
| Cluster runtime | cluster.c (C) | standby_endpoints population, ctrlhello | Minor additions |
| iomgr | netio.c (C) | Unchanged | No modifications |
| Raft library | lib/raft/raft.c | Unchanged | No modifications |

---

## File Structure Plan

### Modified Files

- `src/controller/ha.c` — All behavioral changes: `peer_ctrl_msg_handler`, `vote_ep_ctx_t` promoted lifetime, `ha_vote_entry_t.ep` tracking, `raft_cb_become_leader` promotion path, `raft_cb_become_follower` passive path + watchdog arm, remove `fetch_config_from_primary` delay task
- `src/controller/ha.h` — Add `endpoint_t *ep` to `ha_vote_entry_t`; add `peer_ctrl_msg_handler` forward declaration or make it accessible via a new test-seam hook; update comments on `ha_find_and_connect_to_primary` noting standby post-election path no longer calls it
- `src/controller/config_repl.c` — `handle_ctrlhello`: set `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY and ctrlhello arrives for the first time (Note: this design originally cited `cluster.c`; `handle_ctrlhello` actually lives in `config_repl.c` — drift already flagged in this spec's own research.md R4, corrected here)
- `src/controller/testsuite/test_ha_connection_reuse.c` — Tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello (Note: implemented under this filename, not `test_ha_election.c` as originally planned — that file is a pre-existing, unrelated quorum-election test suite)

---

## System Flows

### Election + Promotion (Winner Path)

```mermaid
sequenceDiagram
    participant C as Candidate ha.c
    participant I as iomgr
    participant P as Peer Standby

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>P: xce_request(OPCode_requestVote)
    note over C: ep->msg_handler = peer_ctrl_msg_handler
    P->>C: RequestVoteReply
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: vote_reply_decode_and_dispatch → raft_on_vote_reply
    C->>C: raft_cb_become_leader fires
    C->>C: votes[i].ep added to ha->standby_endpoints
    C->>P: push_config_to_standbys (on promoted ep)
    C->>P: ha_send_ctrlhello_to_all_standbys (on promoted ep)
    C->>C: repl_add_standby registers promoted ep with WAL replicator (ha-commit-engine addition)
    P->>C: OPCode_replCatchUpDone on same ep
    I->>C: peer_ctrl_msg_handler(role=PRIMARY)
    C->>C: repl_ack_decode_and_dispatch: replCatchUpDone is now a no-op (ack-counting removed)
```

> Note: the original `role=COMMITTING` step and `ha_repl_ack_received → ha_promote_to_primary`
> ack-counting tail no longer exist — `CTRL_PRIMARY_COMMITTING` and `ha_repl_ack_received` were
> deleted by `ha-consistency-invariants`/`ha-single-leader-role` (2026-06-11); promotion is now
> driven by the heartbeat decision loop + commit engine (`ha-commit-engine`, `ha-repl-hardening`),
> not by counting `replCatchUpDone` acks on the promoted ep. See those specs' design.md.

### Election + Passive Wait (Loser Path)

```mermaid
sequenceDiagram
    participant C as Loser ha.c
    participant I as iomgr
    participant W as Winner Primary

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>W: xce_request(OPCode_requestVote)
    W->>C: RequestVoteReply (denied)
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: raft_cb_become_follower fires
    C->>C: endpoint_fini(votes[i].ep) — close all vote endpoints
    C->>C: ha_arm_heartbeat_timeout_vtask
    note over C: CTRL_STANDBY — passive wait on listener
    W->>C: ctrlhello arrives on listener (existing iomgr path)
    C->>C: handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd
    C->>C: ha_on_heartbeat_received resets watchdog deadline
```

---

## Requirements Traceability

| Requirement | Summary | Component | Flow |
|-------------|---------|-----------|------|
| 1.1 | Candidate establishes one outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` (unchanged) | Election Phase |
| 1.2 | Candidate reuses each surviving outbound vote connection as persistent peer channel | `raft_cb_become_leader` promotes ep; `peer_ctrl_msg_handler` dispatches by role | Promotion |
| 1.3 | Winner promotes vote connections to standby replication channels | `raft_cb_become_leader`: add `votes[i].ep` to `standby_endpoints` | Winner Path |
| 1.4 | Loser closes all outbound vote connections | `raft_cb_become_follower`: `endpoint_fini(votes[i].ep)` | Loser Path |
| 2.1 | CTRL_STANDBY does not initiate outbound TCP connections | Remove `fetch_config_from_primary` delay task from `raft_cb_become_follower` | Loser Path |
| 2.2 | Standby accepts ctrlhello / config / WAL on inbound connection from winner | Existing vtaskmgr path unchanged; `handle_ctrlhello` sets `primary_ctrl_fd` | Standby Passive |
| 2.3 | No heartbeat from primary within timeout → new election | `ha_arm_heartbeat_timeout_vtask` called in `raft_cb_become_follower` | Loser Path |
| 2.4 | CTRL_STANDBY does not invoke tier-1/2/3 discovery | Remove `fetch_config_from_primary` call; `ha_find_and_connect_to_primary` not called | Loser Path |
| 3.1 | First ctrlhello on promoted connection → designate as primary control channel | `handle_ctrlhello` sets `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY | Standby Passive |
| 3.2 | Primary control channel RDHUP → detect within one epoll cycle → CTRL_ELECTING | Existing `ha_on_primary_ctrl_fd_rdhup` unchanged; armed from ctrlhello ep | Standby Passive |
| 3.3 | Arm heartbeat watchdog with same semantics | `ha_arm_heartbeat_timeout_vtask` called in `raft_cb_become_follower` | Loser Path |
| 4.1 | Loser explicitly closes all outbound vote endpoints on election loss | `raft_cb_become_follower`: iterate `ha->elect_ctx->votes[i].ep`, call `endpoint_fini` | Loser Path |
| 4.2 | Failed outbound vote connection → count peer as denied, no reconnect | Existing `vote_ep_err_handler` unchanged | Error |
| 4.3 | New election → fresh outbound connections | `ha_start_election_async` creates new endpoints each time (unchanged) | Election Phase |
| 5.1 | Primary pushes config via promoted connection | `push_config_to_standbys` uses `standby_endpoints` (populated from promoted eps) | Winner Path |
| 5.2 | WAL replication via promoted connections | Existing catch-up path uses `standby_endpoints`; majority-ack commit tracking now also registers the promoted ep with the replicator via `repl_add_standby` (Note: added by `ha-commit-engine`, 2026-06-12 — see that spec's design.md) | Winner Path |
| 5.3 | Single-node self-elect without peer connections | Existing raft library single-node path unchanged | Election Phase |
| 5.4 | Committing timeout → degraded PRIMARY | `committing_timeout_fn` no longer exists (Note: deleted along with `CTRL_PRIMARY_COMMITTING` by `ha-consistency-invariants`, 2026-06-11; replaced by promotion-fence/own-term-gate machinery — `ha_arm_own_term_gate`, `ha_arm_promotion_fence_timer` — see that spec's design.md) | Winner Path |
| 5.5 | No new ASAN errors | Promoted ep lifetime owned by `vote_ep_err_handler` / `raft_cb_become_follower` cleanup | All |

---

## Components and Interfaces

| Component | File | Intent | Req Coverage |
|-----------|------|--------|--------------|
| `peer_ctrl_msg_handler` | ha.c | State-aware dispatcher; replaces `vote_reply_msg_handler` | 1.2 |
| `ha_vote_entry_t.ep` field | ha.h | Track per-peer outgoing endpoint for promotion/close | 1.3, 1.4, 4.1 |
| `raft_cb_become_leader` (extended) | ha.c | Promote vote eps → standby_endpoints | 1.3, 5.1, 5.2 |
| `raft_cb_become_follower` (extended) | ha.c | Close vote eps, arm watchdog, remove fetch_config call | 1.4, 2.1, 2.3, 2.4, 3.3 |
| `handle_ctrlhello` (extended) | config_repl.c (design originally cited cluster.c — corrected) | Set primary_ctrl_fd from ctrlhello ep | 3.1, 3.2 |

### HA Election Layer

#### peer_ctrl_msg_handler

| Field | Detail |
|-------|--------|
| Intent | Single iomgr message handler for outgoing vote endpoints; routes by current role |
| Requirements | 1.2 |
| File | `src/controller/ha.c` |

**Responsibilities & Constraints**

- Set once in `on_vote_connected`; never reassigned
- Reads `ha->role` atomically at dispatch time
- Routes `CTRL_ELECTING` to vote reply decode (existing `vote_reply_msg_handler` logic, factored out as `vote_reply_decode_and_dispatch`)
- Routes `CTRL_PRIMARY` to repl-ack decode (`repl_ack_decode_and_dispatch`). (Note: `CTRL_PRIMARY_COMMITTING` no longer exists — removed by `ha-consistency-invariants`/`ha-single-leader-role`, 2026-06-11; `repl_ack_decode_and_dispatch`'s `OPCode_replCatchUpDone` branch is now an INFO-logged no-op and `ha_repl_ack_received` was deleted along with the ack-counting promotion path — see those specs' design.md)
- Also routes `CTRL_STANDBY` with an in-progress PreVote round to `pre_vote_reply_decode_and_dispatch` — added later by `ha-prevote` (2026-07-18), not part of this spec's original scope
- Any other role: drop packet (free packet, return 0)
- Runs in iomgr thread — must not block, for the default (dispatch-flag-OFF) path; `raft-drive-async` (2026-07-18) later added an optional vtask-pool dispatch mode (`ha_peerctrl_dispatch_enabled`) that can run this body off the iomgr thread — see that spec's design.md

**Contracts**: State [ x ]

##### State Management

```c
/* Signature: matches msg_handler_t = int (*)(void*) */
static int peer_ctrl_msg_handler(void *ep_raw);

/* Internal helpers (static, same file) — current signatures (corrected; both
 * gained an io_packet_t *pkt parameter, and vote_reply_decode_and_dispatch
 * dropped its endpoint_t *ep parameter, as part of the raft-drive-async
 * pkt-ownership refactor, 2026-07-18) */
static int vote_reply_decode_and_dispatch(vote_ep_ctx_t *vctx, io_packet_t *pkt);
static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx, io_packet_t *pkt);
```

- State model: branches on `atomic_load_explicit(&ha->role, memory_order_acquire)`
- ELECTING path: existing vote-reply capnp decode → `vote_reply_dispatch` → `raft_on_vote_reply`
- PRIMARY path: decode `OPCode_replCatchUpDone` — now a no-op INFO log (ack-counting/`ha_repl_ack_received` removed, see note above); the same handler also decodes heartbeat-reply `stateReport` for convergence tracking
- Other: free packet, no-op

**Implementation Notes**

- `vote_reply_decode_and_dispatch` is exactly the body of the existing `vote_reply_msg_handler` (factored out); the XCE_TESTING term-fencing and `xce_vote_reply_hook` seams are preserved
- `repl_ack_decode_and_dispatch` decodes an inbound capnp packet; `OPCode_replCatchUpDone` is now a no-op (superseded — see note above), and it separately decodes heartbeat-reply `stateReport` messages for convergence tracking. The original "COMMITTING phase" framing no longer applies since that role was removed.

#### raft_cb_become_leader (extended)

| Field | Detail |
|-------|--------|
| Intent | After election win: populate `ha->standby_endpoints` from promoted vote endpoints |
| Requirements | 1.3, 5.1, 5.2 |
| File | `src/controller/ha.c` |

**Responsibilities & Constraints**

- Read `ha->elect_ctx` to get `ha_elect_ctx_t*`
- Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)`
- If `votes[i].ep != NULL` (connect succeeded): add to `ha->standby_endpoints` under `ha->lock`
- Existing `push_config_to_standbys` and `ha_send_ctrlhello_to_all_standbys` calls are unchanged; they iterate the now-populated array
- (Note: `ha-commit-engine`, 2026-06-12, added a further registration in this same loop — each promoted ep is also registered with the WAL replicator via `repl_add_standby(&c->persist.repl, ...)`, which is what majority-ack commit tracking now uses; `standby_endpoints` remains the config-push/ctrlhello target only. See that spec's design.md.)

**Implementation Notes**

- `votes[0]` is reserved for self in the raft library; peer endpoints start at index 1. Verify index mapping against `raft_cb_send_vote_request` peer_idx.
- If `ha->elect_ctx` is NULL when `raft_cb_become_leader` fires (e.g., single-node), skip promotion loop (no peers to promote).
- Risk: `votes[i].ep` may be NULL if the peer's connect failed; the NULL check guards this.

#### raft_cb_become_follower (extended)

| Field | Detail |
|-------|--------|
| Intent | After election loss: close vote endpoints, arm heartbeat watchdog, remove fetch_config call |
| Requirements | 1.4, 2.1, 2.3, 2.4, 3.3 |
| File | `src/controller/ha.c` |

**Responsibilities & Constraints**

- Read `ha->elect_ctx`, iterate `votes[i].ep`, call `endpoint_fini(votes[i].ep)` for non-NULL entries, set `votes[i].ep = NULL`
- Do NOT call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary`
- Call `ha_arm_heartbeat_timeout_vtask(ha, c)` after role is set to CTRL_STANDBY
- Existing `cluster_init_phase2` call and `ha->phase2_done` guard are preserved

**Implementation Notes**

- `endpoint_fini` closes the fd and releases the endpoint slot; iomgr is notified via epoll RDHUP on the peer side, which is expected
- Watchdog arm must happen after `ha->role = CTRL_STANDBY` is stored (role gate in watchdog logic)
- XCE_TESTING: `ha_arm_heartbeat_timeout_vtask` is a no-op when `c == NULL`; the test seam is preserved

### Cluster Runtime Layer

#### handle_ctrlhello (extended)

| Field | Detail |
|-------|--------|
| Intent | Set `ha->primary_ctrl_fd` when ctrlhello arrives on standby's inbound connection |
| Requirements | 3.1, 3.2, 3.3 |
| File | `src/controller/cluster.c` |

**Responsibilities & Constraints**

- Existing ctrlhello handler already processes OP_hello and triggers WAL catch-up
- Add: if `ha->role == CTRL_STANDBY` and `ha->primary_ctrl_fd == -1`, set `ha->primary_ctrl_fd = ep->fd`
- Call `ha_on_heartbeat_received(ha)` to reset the watchdog deadline (already done by heartbeat path; add to ctrlhello path for first message)
- No other changes to the ctrlhello handler

**Implementation Notes**

- Guard with `primary_ctrl_fd == -1` for idempotency; a repeat ctrlhello on the same connection does not need re-registration
- `ha_on_primary_ctrl_fd_rdhup` already handles fd close and triggers re-election; no changes needed there

---

## Data Models

### ha_vote_entry_t (modified)

```c
typedef struct {
    xce_candidate_t *peer;          /* Peer being solicited */
    uint64_t         my_term;       /* Term sent in OP_requestVote */
    uint64_t         my_tag;        /* config_tag sent in request */
    uint64_t         my_lsn;        /* WAL LSN sent in request */
    int              vote_granted;  /* 1 if peer granted vote */
    int              replied;       /* 1 if peer replied */
    uint64_t         peer_tag;      /* peer's config_tag from reply */
    uint64_t         peer_lsn;      /* peer's WAL LSN from reply */
    struct endpoint_t *ep;          /* NEW: outgoing vote endpoint; NULL if connect failed */
} ha_vote_entry_t;
```

**Invariants**:
- `ep` is set by `raft_cb_send_vote_request` when `iomgr_monitor` succeeds
- `ep` is NULL if `endpoint_connect` or `iomgr_monitor` failed for this peer
- Winner: `ep` is transferred to `standby_endpoints`; winner does NOT call `endpoint_fini` on promoted eps
- Loser: `ep` is closed via `endpoint_fini` in `raft_cb_become_follower`; set to NULL after close

### vote_ep_ctx_t

The `promoted` flag is NOT added — role is read from `ha->role` at dispatch time, which is the authoritative state, as this design intended. (Note: despite the original "(unchanged)" heading, the struct is no longer literally unchanged since this spec closed — `ha-term-safety` (2026-07-17) added `my_last_log_term`/`my_last_log_index` fields, and `ha-elect-ctx-lifecycle` (2026-07-18) changed its lifetime contract: the endpoint's `on_fini` hook now calls `ha_elect_ctx_release` to decrement a refcount on the owning `ha_elect_ctx_t`, rather than the simple election-cleanup-only lifetime this design describes. See those specs' design.md.)

---

## Error Handling

### Error Strategy

Treat connection failures and packet decode errors identically to the existing vote path: count the peer as denied (or drop the packet for non-vote phases).

### Error Categories

**Connect failure** (vote_ep_err_handler):
- Existing behavior preserved: a connect-time or post-connect failure counts as a denied vote through the production tally (`vote_reply_dispatch`), not the legacy `handle_request_vote_reply` path (hardened later by `raft-adapter-conformance`)
- Promoted connection disconnect: `vote_ep_err_handler` fires when the standby drops. For `PRIMARY` role (Note: the `COMMITTING` role named here was removed — see the peer_ctrl_msg_handler note above), remove `ep` from `ha->standby_endpoints` under `ha->lock`; current code also calls `repl_remove_standby` to deregister from the WAL replicator (`ha-commit-engine` addition) — `ha_repl_ack_received` no longer exists

**Packet decode error in repl_ack_decode_and_dispatch**:
- Drop packet, return 0 (iomgr continues monitoring endpoint)
- Log WARN with opcode for debugging

**Unexpected role in peer_ctrl_msg_handler**:
- Drop packet, return 0 (CTRL_INIT: should not happen normally; CTRL_STANDBY on outgoing ep is now an EXPECTED case when a PreVote round is in progress — routed to `pre_vote_reply_decode_and_dispatch` rather than dropped, added by `ha-prevote`, 2026-07-18)
- Log WARN

### Monitoring

- Add `INFO` log in `raft_cb_become_leader` for each promoted endpoint: `"HA: promoted vote ep[%d] → standby channel"`
- Add `INFO` log in `raft_cb_become_follower` for endpoint close: `"HA: closing vote ep[%d] (loser path)"`
- Existing heartbeat watchdog log messages unchanged

---

## Testing Strategy

### Unit Tests (test_ha_election.c)

1. **Dispatcher routes by role**: set `ep->msg_handler = peer_ctrl_msg_handler`; inject packet with role=ELECTING → verify `xce_vote_reply_hook` fires; inject packet with role=COMMITTING → verify repl-ack hook fires; inject with role=STANDBY → verify packet dropped
2. **Winner promotes endpoints**: call `raft_cb_become_leader` with `ha->elect_ctx` populated; verify `ha->standby_endpoints->count == n_peers_connected`
3. **Loser closes endpoints**: call `raft_cb_become_follower`; verify `votes[i].ep == NULL` for all i; verify `endpoint_fini` called (ASAN validates no leak)
4. **Watchdog armed on CTRL_STANDBY**: mock `ha_arm_heartbeat_timeout_vtask` via test hook; verify it is called exactly once from `raft_cb_become_follower`
5. **fetch_config_from_primary NOT called**: verify no `delay_task_ms` call for `fetch_config_from_primary` in `raft_cb_become_follower` (use test seam to intercept delay_task_ms)
6. **primary_ctrl_fd set on ctrlhello**: call `handle_ctrlhello` with role=CTRL_STANDBY and `primary_ctrl_fd=-1`; verify `primary_ctrl_fd == ep->fd`
7. **Promoted ep err_handler removes from standby_endpoints**: populate `standby_endpoints` with a mock ep; call `vote_ep_err_handler` with role=COMMITTING; verify ep removed from array

### Integration Tests (test_ha_bootstrap.c / test_p3_integration.c)

1. **Two-node election → standby receives ctrlhello without outbound connect**: boot two nodes; verify standby never calls `ha_find_and_connect_to_primary`; verify ctrlhello arrives within `election_timeout_ms + 200ms`
2. **Promoted connection carries WAL replication**: verify `OPCode_replCatchUpDone` decoded on promoted ep increments `repl_ack_count`
3. **Primary failover → fresh connections on re-election**: kill primary; verify standby transitions CTRL_ELECTING and creates fresh endpoints (not reusing closed ones)