# Design Document: ha-peer-connection-reuse

## Overview

The XCE controller election currently discards outgoing vote connections after the vote exchange, forcing standbys to create new outbound connections to the primary for the persistent control channel. This produces a 10-second heartbeat timeout delay before standbys detect the primary, spurious retry warnings from premature config-fetch attempts, and a complex tier-1/2/3 primary discovery path.

This spec reuses each candidate's outgoing vote endpoint as the persistent peer control channel: the election winner promotes its connections to replication channels; the loser closes its connections and waits passively for the primary's ctrlhello to arrive on its listener.

The entire change is confined to `ha.c` and `cluster.c`. iomgr, the capnp wire protocol, and the raft library are not modified.

### Goals

- Eliminate the double-connection round-trip after election
- Remove the standby's active primary discovery path (fetch_config_from_primary)
- Arm the standby heartbeat watchdog immediately when CTRL_STANDBY is entered
- Preserve existing HA behaviors: failover, WAL replication, config push, heartbeat detection

### Non-Goals

- SSL/TLS certificate management changes
- Vote grant/deny protocol logic changes
- WAL entry format or replication content changes
- Config file format changes
- Cluster membership changes (add/remove peers)

---

## Boundary Commitments

### This Spec Owns

- The lifecycle of outgoing vote endpoints from creation through promotion or close
- The `peer_ctrl_msg_handler` state-aware dispatcher (replaces `vote_reply_msg_handler` + explicit reassignment)
- Population of `ha->standby_endpoints` from promoted vote connections on the primary side
- Arming the standby heartbeat watchdog from `raft_cb_become_follower` instead of after `ha_find_and_connect_to_primary`
- Setting `ha->primary_ctrl_fd` from the inbound ctrlhello connection on the standby side
- Removing the `fetch_config_from_primary` delay task from the standby path

### Out of Boundary

- SSL/TLS negotiation details — handled by existing `client_ssl_ctx` / `on_vote_connected` path
- iomgr internals — zero changes to `src/lib/netio.c` or `src/include/netio.h`
- Raft library — zero changes to `src/lib/raft/raft.c`
- Heartbeat watchdog thread implementation — only the arming callsite changes
- The existing `ha_find_and_connect_to_primary` function — it remains for future use; the standby path no longer calls it after election

### Allowed Dependencies

- `ha_state_t`, `ha_elect_ctx_t`, `ha_vote_entry_t` — extended with new fields
- `vote_ep_ctx_t` — reused as-is for promoted connections
- `endpoint_t`, `iomgr_monitor`, `endpoint_set_msg_handler` — used unchanged
- `ha->standby_endpoints` (zarray) — populated from promoted connections
- `raft_cb_become_leader` / `raft_cb_become_follower` — extended to populate/close endpoints
- `ha_arm_heartbeat_timeout_vtask` — called from `raft_cb_become_follower`

### Revalidation Triggers

- Changing `ha_vote_entry_t` struct layout (tasks depending on `votes[].ep`)
- Changing `vote_ep_ctx_t` fields or lifetime contract
- Changing `raft_cb_become_leader` / `raft_cb_become_follower` signatures
- Adding new roles to `ctrl_role_t` (the dispatcher switch must be updated)
- Changing `ha->standby_endpoints` population semantics

---

## Architecture

### Existing Architecture Analysis

The current election flow:
1. Candidate creates one `endpoint_t` per peer (outgoing vote), registers with iomgr
2. `on_vote_connected` sends the vote request and reassigns `ep->msg_handler = vote_reply_msg_handler`
3. After the election, the outgoing vote endpoints are closed by `ha_election_cleanup`
4. The winning standby calls `ha_find_and_connect_to_primary` to create a new inbound connection to the primary
5. The heartbeat watchdog is armed after this reconnection succeeds

The promoted-connection flow (this spec):
1. Same vote endpoint creation
2. `on_vote_connected` sets `ep->msg_handler = peer_ctrl_msg_handler` (state-aware, never reassigned)
3. In `raft_cb_become_leader`: each surviving vote endpoint is added to `ha->standby_endpoints` (primary)
4. In `raft_cb_become_follower`: outgoing vote endpoints are closed; standby waits passively; watchdog armed immediately
5. Primary's ctrlhello arrives on the standby's listener, setting `ha->primary_ctrl_fd`

### Architecture Pattern & Boundary Map

```mermaid
graph TD
    subgraph Election_Phase
        A[raft_cb_send_vote_request] --> B[endpoint_connect + iomgr_monitor]
        B --> C[on_vote_connected sets peer_ctrl_msg_handler]
        C --> D[send RequestVote via xce_request]
    end

    subgraph Election_Result
        E[raft_cb_become_leader] --> F[promote: add ep to standby_endpoints]
        G[raft_cb_become_follower] --> H[close: endpoint_fini each vote ep]
        G --> I[arm heartbeat watchdog]
    end

    subgraph Promoted_Connection_PRIMARY
        F --> J[peer_ctrl_msg_handler role=COMMITTING or PRIMARY]
        J --> K[repl_ack_decode: ha_repl_ack_received]
    end

    subgraph Standby_Passive
        H --> L[wait for ctrlhello on listener]
        L --> M[handle_ctrlhello sets primary_ctrl_fd]
        I --> N[watchdog arm: fires on silence timeout]
    end

    subgraph Dispatcher
        D --> O[peer_ctrl_msg_handler]
        O -- role=ELECTING --> P[vote_reply_decode_and_dispatch]
        O -- role=COMMITTING or PRIMARY --> K
        O -- other role --> Q[drop packet]
    end
```

### Technology Stack

| Layer | Choice | Role | Notes |
|-------|--------|------|-------|
| HA election | ha.c (C, gnu17) | Endpoint lifecycle, dispatcher | All changes here |
| Cluster runtime | cluster.c (C) | standby_endpoints population, ctrlhello | Minor additions |
| iomgr | netio.c (C) | Unchanged | No modifications |
| Raft library | lib/raft/raft.c | Unchanged | No modifications |

---

## File Structure Plan

### Modified Files

- `src/controller/ha.c` — All behavioral changes: `peer_ctrl_msg_handler`, `vote_ep_ctx_t` promoted lifetime, `ha_vote_entry_t.ep` tracking, `raft_cb_become_leader` promotion path, `raft_cb_become_follower` passive path + watchdog arm, remove `fetch_config_from_primary` delay task
- `src/controller/ha.h` — Add `endpoint_t *ep` to `ha_vote_entry_t`; add `peer_ctrl_msg_handler` forward declaration or make it accessible via a new test-seam hook; update comments on `ha_find_and_connect_to_primary` noting standby post-election path no longer calls it
- `src/controller/cluster.c` — `handle_ctrlhello`: set `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY and ctrlhello arrives for the first time
- `src/controller/testsuite/test_ha_election.c` — Add tests for: dispatcher routes by role, winner promotes endpoints, loser closes endpoints, watchdog armed on CTRL_STANDBY, standby sets primary_ctrl_fd on ctrlhello

---

## System Flows

### Election + Promotion (Winner Path)

```mermaid
sequenceDiagram
    participant C as Candidate ha.c
    participant I as iomgr
    participant P as Peer Standby

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>P: xce_request(OPCode_requestVote)
    note over C: ep->msg_handler = peer_ctrl_msg_handler
    P->>C: RequestVoteReply
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: vote_reply_decode_and_dispatch → raft_on_vote_reply
    C->>C: raft_cb_become_leader fires
    C->>C: votes[i].ep added to ha->standby_endpoints
    C->>P: push_config_to_standbys (on promoted ep)
    C->>P: ha_send_ctrlhello_to_all_standbys (on promoted ep)
    P->>C: OPCode_replCatchUpDone on same ep
    I->>C: peer_ctrl_msg_handler(role=COMMITTING)
    C->>C: ha_repl_ack_received → ha_promote_to_primary
```

### Election + Passive Wait (Loser Path)

```mermaid
sequenceDiagram
    participant C as Loser ha.c
    participant I as iomgr
    participant W as Winner Primary

    C->>I: endpoint_connect + iomgr_monitor (per peer)
    I->>C: on_vote_connected fires
    C->>W: xce_request(OPCode_requestVote)
    W->>C: RequestVoteReply (denied)
    I->>C: peer_ctrl_msg_handler(role=ELECTING)
    C->>C: raft_cb_become_follower fires
    C->>C: endpoint_fini(votes[i].ep) — close all vote endpoints
    C->>C: ha_arm_heartbeat_timeout_vtask
    note over C: CTRL_STANDBY — passive wait on listener
    W->>C: ctrlhello arrives on listener (existing iomgr path)
    C->>C: handle_ctrlhello sets ha->primary_ctrl_fd = ep->fd
    C->>C: ha_on_heartbeat_received resets watchdog deadline
```

---

## Requirements Traceability

| Requirement | Summary | Component | Flow |
|-------------|---------|-----------|------|
| 1.1 | Candidate establishes one outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` (unchanged) | Election Phase |
| 1.2 | Candidate reuses each surviving outbound vote connection as persistent peer channel | `raft_cb_become_leader` promotes ep; `peer_ctrl_msg_handler` dispatches by role | Promotion |
| 1.3 | Winner promotes vote connections to standby replication channels | `raft_cb_become_leader`: add `votes[i].ep` to `standby_endpoints` | Winner Path |
| 1.4 | Loser closes all outbound vote connections | `raft_cb_become_follower`: `endpoint_fini(votes[i].ep)` | Loser Path |
| 2.1 | CTRL_STANDBY does not initiate outbound TCP connections | Remove `fetch_config_from_primary` delay task from `raft_cb_become_follower` | Loser Path |
| 2.2 | Standby accepts ctrlhello / config / WAL on inbound connection from winner | Existing vtaskmgr path unchanged; `handle_ctrlhello` sets `primary_ctrl_fd` | Standby Passive |
| 2.3 | No heartbeat from primary within timeout → new election | `ha_arm_heartbeat_timeout_vtask` called in `raft_cb_become_follower` | Loser Path |
| 2.4 | CTRL_STANDBY does not invoke tier-1/2/3 discovery | Remove `fetch_config_from_primary` call; `ha_find_and_connect_to_primary` not called | Loser Path |
| 3.1 | First ctrlhello on promoted connection → designate as primary control channel | `handle_ctrlhello` sets `ha->primary_ctrl_fd = ep->fd` when role == CTRL_STANDBY | Standby Passive |
| 3.2 | Primary control channel RDHUP → detect within one epoll cycle → CTRL_ELECTING | Existing `ha_on_primary_ctrl_fd_rdhup` unchanged; armed from ctrlhello ep | Standby Passive |
| 3.3 | Arm heartbeat watchdog with same semantics | `ha_arm_heartbeat_timeout_vtask` called in `raft_cb_become_follower` | Loser Path |
| 4.1 | Loser explicitly closes all outbound vote endpoints on election loss | `raft_cb_become_follower`: iterate `ha->elect_ctx->votes[i].ep`, call `endpoint_fini` | Loser Path |
| 4.2 | Failed outbound vote connection → count peer as denied, no reconnect | Existing `vote_ep_err_handler` unchanged | Error |
| 4.3 | New election → fresh outbound connections | `ha_start_election_async` creates new endpoints each time (unchanged) | Election Phase |
| 5.1 | Primary pushes config via promoted connection | `push_config_to_standbys` uses `standby_endpoints` (populated from promoted eps) | Winner Path |
| 5.2 | WAL replication via promoted connections | Existing catch-up path uses `standby_endpoints` | Winner Path |
| 5.3 | Single-node self-elect without peer connections | Existing raft library single-node path unchanged | Election Phase |
| 5.4 | Committing timeout → degraded PRIMARY | Existing `committing_timeout_fn` unchanged | Winner Path |
| 5.5 | No new ASAN errors | Promoted ep lifetime owned by `vote_ep_err_handler` / `raft_cb_become_follower` cleanup | All |

---

## Components and Interfaces

| Component | File | Intent | Req Coverage |
|-----------|------|--------|--------------|
| `peer_ctrl_msg_handler` | ha.c | State-aware dispatcher; replaces `vote_reply_msg_handler` | 1.2 |
| `ha_vote_entry_t.ep` field | ha.h | Track per-peer outgoing endpoint for promotion/close | 1.3, 1.4, 4.1 |
| `raft_cb_become_leader` (extended) | ha.c | Promote vote eps → standby_endpoints | 1.3, 5.1, 5.2 |
| `raft_cb_become_follower` (extended) | ha.c | Close vote eps, arm watchdog, remove fetch_config call | 1.4, 2.1, 2.3, 2.4, 3.3 |
| `handle_ctrlhello` (extended) | cluster.c | Set primary_ctrl_fd from ctrlhello ep | 3.1, 3.2 |

### HA Election Layer

#### peer_ctrl_msg_handler

| Field | Detail |
|-------|--------|
| Intent | Single iomgr message handler for outgoing vote endpoints; routes by current role |
| Requirements | 1.2 |
| File | `src/controller/ha.c` |

**Responsibilities & Constraints**

- Set once in `on_vote_connected`; never reassigned
- Reads `ha->role` atomically at dispatch time
- Routes `CTRL_ELECTING` to vote reply decode (existing `vote_reply_msg_handler` logic, factored out as `vote_reply_decode_and_dispatch`)
- Routes `CTRL_PRIMARY_COMMITTING` / `CTRL_PRIMARY` to repl-ack decode (`repl_ack_decode_and_dispatch`, calls `ha_repl_ack_received`)
- Any other role: drop packet (free packet, return 0)
- Runs in iomgr thread — must not block

**Contracts**: State [ x ]

##### State Management

```c
/* Signature: matches msg_handler_t = int (*)(void*) */
static int peer_ctrl_msg_handler(void *ep_raw);

/* Internal helpers (static, same file) */
static int vote_reply_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx);
static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx);
```

- State model: branches on `atomic_load_explicit(&ha->role, memory_order_acquire)`
- ELECTING path: existing vote-reply capnp decode → `vote_reply_dispatch` → `raft_on_vote_reply`
- COMMITTING/PRIMARY path: decode `OPCode_replCatchUpDone` → `ha_repl_ack_received(ha, c)`
- Other: free packet, no-op

**Implementation Notes**

- `vote_reply_decode_and_dispatch` is exactly the body of the existing `vote_reply_msg_handler` (factored out); the XCE_TESTING term-fencing and `xce_vote_reply_hook` seams are preserved
- `repl_ack_decode_and_dispatch` decodes an inbound capnp packet expecting `OPCode_replCatchUpDone` then calls `ha_repl_ack_received`; the opcode check guards against unexpected packets on the promoted endpoint during the COMMITTING phase

#### raft_cb_become_leader (extended)

| Field | Detail |
|-------|--------|
| Intent | After election win: populate `ha->standby_endpoints` from promoted vote endpoints |
| Requirements | 1.3, 5.1, 5.2 |
| File | `src/controller/ha.c` |

**Responsibilities & Constraints**

- Read `ha->elect_ctx` to get `ha_elect_ctx_t*`
- Iterate `ctx->votes[i].ep` for `i` in `[0, ctx->n_peers)`
- If `votes[i].ep != NULL` (connect succeeded): add to `ha->standby_endpoints` under `ha->lock`
- Existing `push_config_to_standbys` and `ha_send_ctrlhello_to_all_standbys` calls are unchanged; they iterate the now-populated array

**Implementation Notes**

- `votes[0]` is reserved for self in the raft library; peer endpoints start at index 1. Verify index mapping against `raft_cb_send_vote_request` peer_idx.
- If `ha->elect_ctx` is NULL when `raft_cb_become_leader` fires (e.g., single-node), skip promotion loop (no peers to promote).
- Risk: `votes[i].ep` may be NULL if the peer's connect failed; the NULL check guards this.

#### raft_cb_become_follower (extended)

| Field | Detail |
|-------|--------|
| Intent | After election loss: close vote endpoints, arm heartbeat watchdog, remove fetch_config call |
| Requirements | 1.4, 2.1, 2.3, 2.4, 3.3 |
| File | `src/controller/ha.c` |

**Responsibilities & Constraints**

- Read `ha->elect_ctx`, iterate `votes[i].ep`, call `endpoint_fini(votes[i].ep)` for non-NULL entries, set `votes[i].ep = NULL`
- Do NOT call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary`
- Call `ha_arm_heartbeat_timeout_vtask(ha, c)` after role is set to CTRL_STANDBY
- Existing `cluster_init_phase2` call and `ha->phase2_done` guard are preserved

**Implementation Notes**

- `endpoint_fini` closes the fd and releases the endpoint slot; iomgr is notified via epoll RDHUP on the peer side, which is expected
- Watchdog arm must happen after `ha->role = CTRL_STANDBY` is stored (role gate in watchdog logic)
- XCE_TESTING: `ha_arm_heartbeat_timeout_vtask` is a no-op when `c == NULL`; the test seam is preserved

### Cluster Runtime Layer

#### handle_ctrlhello (extended)

| Field | Detail |
|-------|--------|
| Intent | Set `ha->primary_ctrl_fd` when ctrlhello arrives on standby's inbound connection |
| Requirements | 3.1, 3.2, 3.3 |
| File | `src/controller/cluster.c` |

**Responsibilities & Constraints**

- Existing ctrlhello handler already processes OP_hello and triggers WAL catch-up
- Add: if `ha->role == CTRL_STANDBY` and `ha->primary_ctrl_fd == -1`, set `ha->primary_ctrl_fd = ep->fd`
- Call `ha_on_heartbeat_received(ha)` to reset the watchdog deadline (already done by heartbeat path; add to ctrlhello path for first message)
- No other changes to the ctrlhello handler

**Implementation Notes**

- Guard with `primary_ctrl_fd == -1` for idempotency; a repeat ctrlhello on the same connection does not need re-registration
- `ha_on_primary_ctrl_fd_rdhup` already handles fd close and triggers re-election; no changes needed there

---

## Data Models

### ha_vote_entry_t (modified)

```c
typedef struct {
    xce_candidate_t *peer;          /* Peer being solicited */
    uint64_t         my_term;       /* Term sent in OP_requestVote */
    uint64_t         my_tag;        /* config_tag sent in request */
    uint64_t         my_lsn;        /* WAL LSN sent in request */
    int              vote_granted;  /* 1 if peer granted vote */
    int              replied;       /* 1 if peer replied */
    uint64_t         peer_tag;      /* peer's config_tag from reply */
    uint64_t         peer_lsn;      /* peer's WAL LSN from reply */
    struct endpoint_t *ep;          /* NEW: outgoing vote endpoint; NULL if connect failed */
} ha_vote_entry_t;
```

**Invariants**:
- `ep` is set by `raft_cb_send_vote_request` when `iomgr_monitor` succeeds
- `ep` is NULL if `endpoint_connect` or `iomgr_monitor` failed for this peer
- Winner: `ep` is transferred to `standby_endpoints`; winner does NOT call `endpoint_fini` on promoted eps
- Loser: `ep` is closed via `endpoint_fini` in `raft_cb_become_follower`; set to NULL after close

### vote_ep_ctx_t (unchanged)

No structural changes. The `promoted` flag is NOT added — role is read from `ha->role` at dispatch time, which is the authoritative state. The ctx carries `ha`, `c`, `peer_idx` which are sufficient for both phases.

---

## Error Handling

### Error Strategy

Treat connection failures and packet decode errors identically to the existing vote path: count the peer as denied (or drop the packet for non-vote phases).

### Error Categories

**Connect failure** (vote_ep_err_handler):
- Existing behavior preserved: `handle_request_vote_reply(..., 0, ...)` counts as denied vote
- Promoted connection disconnect: `vote_ep_err_handler` fires when the standby drops. For COMMITTING/PRIMARY role, remove `ep` from `ha->standby_endpoints` under `ha->lock`; call `ha_repl_ack_received` or degrade gracefully based on `n_peers`

**Packet decode error in repl_ack_decode_and_dispatch**:
- Drop packet, return 0 (iomgr continues monitoring endpoint)
- Log WARN with opcode for debugging

**Unexpected role in peer_ctrl_msg_handler**:
- Drop packet, return 0 (CTRL_INIT, CTRL_STANDBY on outgoing ep: should not happen normally)
- Log WARN

### Monitoring

- Add `INFO` log in `raft_cb_become_leader` for each promoted endpoint: `"HA: promoted vote ep[%d] → standby channel"`
- Add `INFO` log in `raft_cb_become_follower` for endpoint close: `"HA: closing vote ep[%d] (loser path)"`
- Existing heartbeat watchdog log messages unchanged

---

## Testing Strategy

### Unit Tests (test_ha_election.c)

1. **Dispatcher routes by role**: set `ep->msg_handler = peer_ctrl_msg_handler`; inject packet with role=ELECTING → verify `xce_vote_reply_hook` fires; inject packet with role=COMMITTING → verify repl-ack hook fires; inject with role=STANDBY → verify packet dropped
2. **Winner promotes endpoints**: call `raft_cb_become_leader` with `ha->elect_ctx` populated; verify `ha->standby_endpoints->count == n_peers_connected`
3. **Loser closes endpoints**: call `raft_cb_become_follower`; verify `votes[i].ep == NULL` for all i; verify `endpoint_fini` called (ASAN validates no leak)
4. **Watchdog armed on CTRL_STANDBY**: mock `ha_arm_heartbeat_timeout_vtask` via test hook; verify it is called exactly once from `raft_cb_become_follower`
5. **fetch_config_from_primary NOT called**: verify no `delay_task_ms` call for `fetch_config_from_primary` in `raft_cb_become_follower` (use test seam to intercept delay_task_ms)
6. **primary_ctrl_fd set on ctrlhello**: call `handle_ctrlhello` with role=CTRL_STANDBY and `primary_ctrl_fd=-1`; verify `primary_ctrl_fd == ep->fd`
7. **Promoted ep err_handler removes from standby_endpoints**: populate `standby_endpoints` with a mock ep; call `vote_ep_err_handler` with role=COMMITTING; verify ep removed from array

### Integration Tests (test_ha_bootstrap.c / test_p3_integration.c)

1. **Two-node election → standby receives ctrlhello without outbound connect**: boot two nodes; verify standby never calls `ha_find_and_connect_to_primary`; verify ctrlhello arrives within `election_timeout_ms + 200ms`
2. **Promoted connection carries WAL replication**: verify `OPCode_replCatchUpDone` decoded on promoted ep increments `repl_ack_count`
3. **Primary failover → fresh connections on re-election**: kill primary; verify standby transitions CTRL_ELECTING and creates fresh endpoints (not reusing closed ones)