myxce/.kiro/specs/election-iomgr-refactor/design.md

# Design Document: election-iomgr-refactor

## Overview

The XCE controller was originally fully event-driven: `cluster_init` → `cluster_start` →
`cluster_run` (shutdown-check loop). The HA additions broke this by inserting blocking calls
before `cluster_run`: `ha_wait_for_heartbeat_or_timeout` and `am_i_primary_quorum` block the
main thread, causing SSL context confusion, capnp UAF, and mixed-I/O races.

This design restores full event-driven operation with no blocking calls after phase1.
`myapp_run` does setup then immediately enters `cluster_run`. All election I/O flows through
iomgr callbacks. When the election reaches a terminal state, `ha_elect_evaluate_and_signal`
calls `cluster_init_phase2` inline from vtask context (phase2 is already thin — managers were
moved to phase1 in a prior refactor), then atomically sets `ha->role`. The dispatch gate
(`ha_dispatch_gate`) opens automatically: the next client request sees PRIMARY or STANDBY and
is handled normally. No semaphores, no wbuf signals, no changes to `cluster_run`.

### Goals
- Restore event-driven model: `myapp_run` → `cluster_run` with zero blocking between them
- All election I/O through one epoll event loop (iomgr)
- `ha_elect_evaluate_and_signal` calls thin `cluster_init_phase2` from vtask context; sets role atomically
- `cluster_run` unchanged — still just the shutdown-check loop
- `myreq_handler` / `ha_dispatch_gate` is the sole FSM dispatcher — no new state machinery needed
- Role-based vote gate: PRIMARY and STANDBY-with-live-primary deny votes

### Non-Goals
- Changes to `cluster_run` or `wbuf` signal values
- Incoming vote RPC handling (already correct in cluster.c)
- Heartbeat watchdog thread (existing, unchanged)
- Config replication, RDHUP paths (unchanged)
- Async DNS resolution (deferred)
- TLS certificate verification for vote connections

## Boundary Commitments

### This Spec Owns
- `vote_request_thread` + `am_i_primary_quorum` blocking loop → replaced by `ha_start_election_async()`
- `on_connected` callback added to `netio.c`
- `client_ssl_ctx` in `ha_state_t` + `myapp_run`
- `vote_reply_msg_handler` for iomgr-native reply handling
- Election timeout watchdog thread
- `ha_elect_evaluate_and_signal` extended to call `cluster_init_phase2` then set role
- `handle_request_vote` extended with role-based deny gate
- `myapp_run` restructured: phase1 + SSL + `ha_start_election_async` + `cluster_run` directly

### Out of Boundary
- `cluster_run` internals (unchanged)
- `case OPCode_requestVote:` in cluster.c (already correct, no changes)
- `ha_wait_for_heartbeat_or_timeout` (existing; now called internally by `ha_start_election_async`)
- Config replication SSL

### Allowed Dependencies
- `netio.c` / `endpoint_t` (add `on_connected` callback)
- `ha_elect_ctx_t` (existing; mutex guards concurrent access)
- `iomgr_t` / `iomgr_monitor` (existing)
- `xce_request()` / `endpoint_enq()` (existing)
- `handle_request_vote_reply()` (existing)
- `cluster_init_phase2()` (called from vtask context after election; must be vtask-safe)

### Revalidation Triggers
- Changes to `endpoint_t` structure
- Changes to `cluster_init_phase2` preconditions (anything it accesses that may not be ready)
- Changes to `ha_dispatch_gate` CTRL_ELECTING allowlist

## Architecture

### Before: Blocking startup
```mermaid
graph TD
    myapp_run --> cluster_init_phase1
    cluster_init_phase1 --> ha_wait_hb_or_timeout
    ha_wait_hb_or_timeout -->|blocks 2s| timeout
    timeout --> am_i_primary_quorum
    am_i_primary_quorum -->|pthreads block on sem| result
    result --> cluster_init_phase2
    cluster_init_phase2 --> cluster_start
    cluster_start --> cluster_run
```

### After: Fully event-driven
```mermaid
graph TD
    myapp_run --> cluster_init_phase1
    cluster_init_phase1 --> ssl_setup
    ssl_setup --> cluster_register_listener
    cluster_register_listener --> iomgr_vtask_start
    iomgr_vtask_start --> ha_start_election_async
    ha_start_election_async -->|returns immediately| cluster_run

    iomgr_worker -->|EPOLLOUT handshake| on_vote_connected
    on_vote_connected --> xce_request_vote
    iomgr_worker -->|EPOLLIN reply| vtask_vote_reply_msg_handler
    vtask_vote_reply_msg_handler --> handle_request_vote_reply
    handle_request_vote_reply --> ha_elect_evaluate_and_signal
    ha_elect_evaluate_and_signal --> cluster_init_phase2_thin
    cluster_init_phase2_thin --> set_ha_role_atomic
    set_ha_role_atomic -->|gate opens| serve_traffic

    election_watchdog -->|timeout| ha_elect_evaluate_and_signal
    heartbeat_watchdog -->|hb timeout or received| ha_elect_evaluate_and_signal
```

### Key Decisions

- **`ha_start_election_async()` is fully non-blocking**: Arms heartbeat watchdog + election timeout watchdog, connects outgoing vote endpoints, registers them with iomgr. Returns immediately.
- **Two promotion paths in `ha_elect_evaluate_and_signal`**:
  - *Initial startup* (`ha->phase2_done == 0`): calls `cluster_init_phase2` (loads config data, inits structures), sets `ha->phase2_done = 1`. Phase2 runs exactly once per process lifetime — config upgrade always does a full restart.
  - *RDHUP failover* (`ha->phase2_done == 1`): state is current via continuous WAL replication. Skip phase2; enter COMMITTING directly.
- **`is_first_boot` = `ha->phase2_done == 0`**: Not `!c->running` (which is 0 until `cluster_run` sets it, making all elections look like first boot). `phase2_done` is a dedicated bool in `ha_state_t`.
- **`CTRL_PRIMARY_COMMITTING`**: New FSM state between winning election and serving traffic. Allows repl acks; denies job dispatch. Also denies vote grants (challenger must not displace a committing primary).
- **`cluster_run` unchanged**: Still just `cbuf_get(&wbuf)` for stop signal.
- **`delaymgr_start` and `persist_start` move to `myapp_run`**: Currently called inside `cluster_start()`. In the new design `cluster_start` is not called from `myapp_run`. These two calls move explicitly into `myapp_run` before `ha_start_election_async`.
- **`bootstrap_config_if_needed` removed entirely**: Replaced by the freshness gate in voting + primary replication push. No pre-election config probe needed.
- **`ha_send_ctrlhello_to_all_standbys`**: Iterates `ha->standby_endpoints` (populated when standbys connect and send `OPCode_hello`), sends `CtrlHelloRequest` to each. This triggers the existing catch-up replication in `handle_ctrlhello`. After catch-up, standby sends `OPCode_replCatchUpDone` → `ha_repl_ack_received`.
- **`committing_timeout_ms`**: Added to `ha_config_t` with default `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 10000` (10s). Covers the case where standbys are down at startup (single-node degraded mode).
- **Timeout mechanism — `delay_task_ms` + cancelled flag**: delaymgr is enhanced with one new API:
  - `delay_task_ms(mgr, delay_ms, runcount, fn, arg)` — millisecond-granularity one-shot task using `clock_gettime(CLOCK_MONOTONIC)`. Precision bounded by `mgr->precision_ms`.
  - **Cancellation via state flag**: each context struct (`election_timeout_ctx_t`, `committing_timeout_ctx_t`) carries an `_Atomic int cancelled` field. Setting it to 1 makes the callback a no-op when it fires. This is O(1), lock-free, and race-free — even if the worker dequeued the task and is mid-execution, the flag check at callback entry handles it correctly. No ordq scan or ID tracking needed.
  ```c
  typedef struct {
      _Atomic int    cancelled;
      ha_state_t    *ha;
      cluster_t     *c;
      ha_elect_ctx_t *ctx;
  } election_timeout_ctx_t;

  static int election_timeout_fn(void *arg) {
      election_timeout_ctx_t *t = arg;
      if (atomic_load(&t->cancelled)) return 0;
      ha_elect_evaluate_and_signal(t->ha, t->ctx);
      return 0;
  }

  /* Schedule: */
  delay_task_ms(&c->delaymgr, ha->config.election_timeout_ms, 1, election_timeout_fn, ctx);

  /* "Cancel" when quorum already reached: */
  atomic_store(&ctx->cancelled, 1);
  ```
  The existing heartbeat watchdog keeps its dedicated `sem_timedwait` pthread (precise rearm on every heartbeat — not suitable for delaymgr's polling model).

## File Structure Plan

```
src/lib/
├── netio.c              # Add on_connected callback to endpoint_t; invoke in io_handler
├── delay.c              # Add delay_task_ms() — ms-granularity using clock_gettime(CLOCK_MONOTONIC)
│                        # Cancellation via caller-owned _Atomic int cancelled flag (no API needed)
└── delay.h              # Expose delay_task_ms()

src/controller/
├── ha.h                 # Add to ha_state_t: client_ssl_ctx, phase2_done, repl_ack_count, n_peers
│                        # Add to ha_config_t: committing_timeout_ms (default 10000ms)
│                        # Add CTRL_PRIMARY_COMMITTING to ctrl_role_t enum
│                        # Declare: ha_start_election_async(), ha_election_cleanup(),
│                        #          ha_repl_ack_received(), ha_arm/disarm_committing_timeout(),
│                        #          ha_send_ctrlhello_to_all_standbys()
├── ha.c                 # Remove: vote_request_thread, vote_thread_ctx_t
│                        # Add: vote_ep_ctx_t, on_vote_connected(), vote_reply_msg_handler()
│                        # Add: ha_start_election_async(), election_timeout_watchdog
│                        # Extend ha_elect_evaluate_and_signal: COMMITTING path + replication push
│                        # Extend handle_request_vote: role-based deny gate
│                        # Add: ha_repl_ack_received(), ha_arm/disarm_committing_timeout()
│                        # Extend ha_dispatch_gate: CTRL_PRIMARY_COMMITTING case
├── cluster.c            # cluster_init_phase2: vtask-safe (managers already in phase1)
│                        # ha_send_ctrlhello_to_all_standbys() or reuse handle_ctrlhello path
│                        # cluster_run: UNCHANGED
└── myapp.c              # Restructured: phase1 + SSL + delaymgr/persist_start + ha_start_election_async + cluster_run
                         # Remove: ha_wait_for_heartbeat_or_timeout, am_i_primary_quorum,
                         #         for(;;) retry loop, cluster_init_phase2, cluster_start calls,
                         #         bootstrap_config_if_needed (replaced by freshness gate + primary push)
```

### Modified Files
- `src/lib/netio.c` — Add `on_connected` to `endpoint_t`; fire in `io_handler` after `ssl_handshake_done=1`
- `src/lib/delay.c` — Add `delay_task_ms()` using `clock_gettime(CLOCK_MONOTONIC)` for ms precision; precision still bounded by `mgr->precision_ms`
- `src/lib/delay.h` — Expose `delay_task_ms()`
- `src/controller/ha.h` — Add `client_ssl_ctx`, `phase2_done`, `repl_ack_count`, `CTRL_PRIMARY_COMMITTING`; declare new functions
- `src/controller/ha.c` — Main changes; `ha_dispatch_gate` extended for COMMITTING; uses `delay_task_ms`/`delay_task_cancel` for watchdogs
- `src/controller/cluster.c` — `ha_send_ctrlhello_to_all_standbys` helper; `OPCode_replCatchUpDone` calls `ha_repl_ack_received`; `cluster_run` unchanged
- `src/controller/myapp.c` — Restructured: non-blocking after phase1; `delaymgr_start`, `persist_start` moved here; `bootstrap_config_if_needed` removed

## System Flows

### 3-node startup — ctrl-a wins (with COMMITTING phase)
```mermaid
sequenceDiagram
    participant Main as myapp_run
    participant Async as ha_start_election_async
    participant iomgr as iomgr Worker
    participant vtask as vtaskmgr Worker
    participant PeerB as ctrl-b endpoint
    participant PeerC as ctrl-c endpoint

    Main->>Main: cluster_init_phase1 + SSL + register_listener + delaymgr_start + persist_start
    Main->>Async: ha_start_election_async (non-blocking)
    Async->>PeerB: endpoint_connect(client_ssl_ctx)
    Async->>PeerC: endpoint_connect(client_ssl_ctx)
    Async->>iomgr: iomgr_monitor(peer_eps)
    Async-->>Main: returns immediately

    Main->>Main: cluster_run() — wbuf shutdown loop

    iomgr->>PeerB: handshake → on_vote_connected → OP_requestVote
    iomgr->>PeerC: handshake → on_vote_connected → OP_requestVote
    iomgr->>vtask: vote replies → quorum reached
    vtask->>vtask: ha_elect_evaluate_and_signal → cluster_init_phase2 (first boot)
    vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
    vtask->>PeerB: push_config_to_standbys + ha_send_ctrlhello (triggers WAL catch-up)
    vtask->>PeerC: push_config_to_standbys + ha_send_ctrlhello

    iomgr->>vtask: OPCode_replCatchUpDone from ctrl-b → ha_repl_ack_received (ack=1)
    iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=2 ≥ quorum-1)
    vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
    Note over vtask: dispatch gate fully open — serves job traffic
```

### Primary crash — heartbeat timeout triggers promotion (RDHUP failover)
```mermaid
sequenceDiagram
    participant HBWatch as Heartbeat Watchdog on ctrl-b
    participant vtask as vtaskmgr Worker
    participant iomgr as iomgr on ctrl-b
    participant PeerC as ctrl-c endpoint

    Note over HBWatch: heartbeat_timeout_ms — primary silent
    HBWatch->>vtask: ha_on_primary_ctrl_fd_rdhup → ha->role = CTRL_ELECTING
    HBWatch->>iomgr: ha_start_election_async (retry — is_first_boot=false)
    iomgr->>PeerC: vote exchange → ctrl-b wins
    vtask->>vtask: ha_elect_evaluate_and_signal (is_first_boot=false → skip phase2)
    vtask->>vtask: ha->role = CTRL_PRIMARY_COMMITTING
    vtask->>PeerC: ha_send_ctrlhello (WAL already current — catch-up is fast)
    iomgr->>vtask: OPCode_replCatchUpDone from ctrl-c → ha_repl_ack_received (ack=1 ≥ quorum-1)
    vtask->>vtask: ha_promote_to_primary → ha->role = CTRL_PRIMARY
    Note over vtask: ctrl-b is new primary — state already synchronized via prior replication
```

### Incoming request during election (NOT_READY)
```mermaid
sequenceDiagram
    participant Client
    participant iomgr as iomgr Worker
    participant vtask as vtaskmgr Worker

    Client->>iomgr: TCP connect + OP_jobsubmit
    iomgr->>vtask: myreq_handler
    vtask->>vtask: ha_dispatch_gate(CTRL_ELECTING, OPCode_jobsubmit) → XCE_ERR_NOT_READY
    vtask->>Client: reply status = XCE_ERR_NOT_READY (client retries)
```

## Requirements Traceability

| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1 | No pthreads for vote I/O | `ha_start_election_async` | 3-node startup |
| 1.2 | Vote RPC via iomgr path | `on_vote_connected`, `xce_request` | 3-node startup |
| 1.3 | Non-responding peer → deny | election watchdog → evaluate | — |
| 2.1 | Async TCP+TLS via iomgr | `endpoint_connect`, `iomgr_monitor` | 3-node startup |
| 2.2 | TLS client context | `client_ssl_ctx` in `ha_state_t` | 3-node startup |
| 2.3 | Send vote after handshake | `on_vote_connected` callback | 3-node startup |
| 2.4 | Connect/TLS failure → deny | `vote_ep_err_handler` | — |
| 3.1 | Reply via iomgr packet | `vote_reply_msg_handler` | 3-node startup |
| 3.2 | No capnp UAF | vtask handler reads before free | 3-node startup |
| 3.3 | Extract reply fields | `vote_reply_msg_handler` decodes | 3-node startup |
| 3.4 | Term fencing on reply | `handle_request_vote_reply` (existing) | — |
| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING | `ha_elect_evaluate_and_signal` → committing state | 3-node startup |
| 4.2 | Freshness gate guarantees ground-truth | `handle_request_vote` config_tag check | — |
| 4.3 | Cold start: only highest config_tag wins | freshness gate + role gate combined | — |
| 4.4–4.7 | Retry, timeout, heartbeat abort | watchdog, re-arm, heartbeat watchdog | Primary crash |
| 5.1 | ELECTING gate: NOT_READY for non-election | `ha_dispatch_gate` (existing + COMMITTING) | NOT_READY flow |
| 5.2 | COMMITTING: only repl acks allowed | `ha_dispatch_gate` new COMMITTING case | 3-node startup |
| 5.3 | PRIMARY: full traffic | `ha_dispatch_gate` (existing) | — |
| 5.4–5.5 | STANDBY gate, term persistence | `ha_dispatch_gate`, `ha_persist_term` (existing) | — |
| 6.1 | PRIMARY denies all votes | `handle_request_vote` role gate | — |
| 6.2 | STANDBY+fresh hb denies votes | `handle_request_vote` heartbeat check | — |
| 6.3 | HB timeout → ELECTING | heartbeat watchdog fires RDHUP path | Primary crash |
| 6.4 | INIT/ELECTING evaluates Raft gates | `handle_request_vote` existing gates | — |
| 6.5 | Primary crash → new primary | HB timeout → re-election → COMMITTING → PRIMARY | Primary crash |
| 7.1 | Winner pushes config to standbys | `push_config_to_standbys()` in committing path | 3-node startup |
| 7.2 | Winner replicates WAL to standbys | `handle_ctrlhello()` catch-up (existing) | 3-node startup |
| 7.3 | config_tag=0 standby gets config via push | no independent fetch; primary pushes | 3-node startup |
| 7.4 | Ack tracked per standby | `ha_state_t.repl_ack_count` | 3-node startup |
| 8.1 | COMMITTING blocks job traffic | `ha_dispatch_gate` COMMITTING case | 3-node startup |
| 8.2 | Majority ack → CTRL_PRIMARY | `ha_repl_ack_received()` threshold check | 3-node startup |
| 8.3 | CTRL_PRIMARY opens full traffic | `ha_dispatch_gate` PRIMARY case (existing) | 3-node startup |
| 8.4 | Ack timeout → degraded open | timeout watchdog in committing state | — |
| 9.1–9.4 | Regressions | Single-node, ground-truth invariant, RDHUP hooks-only, priority stagger | — |

## Components and Interfaces

### Summary

| Component | File | Intent | Req Coverage |
|-----------|------|--------|--------------|
| `endpoint_t.on_connected` | netio.c | Async handshake notification | 2.1, 2.3 |
| `ha_state_t.client_ssl_ctx` | ha.h/myapp.c | TLS client context | 2.2 |
| `vote_ep_ctx_t` | ha.c | Per-peer context for callbacks | 1.1, 2.1–2.4 |
| `on_vote_connected()` | ha.c | Send vote RPC after handshake | 2.3, 3.1 |
| `vote_reply_msg_handler()` | ha.c | Decode reply; update tally | 3.1–3.4 |
| `ha_start_election_async()` | ha.c | Non-blocking: connect peers + arm watchdog | 1.1–1.3, 4.1–4.3 |
| Election timeout watchdog | ha.c | Fire evaluate if deadline expires | 1.3, 4.2, 4.3 |
| `handle_request_vote` role gate | ha.c | Deny votes from PRIMARY/healthy-STANDBY | 6.1–6.4 |
| `ha_elect_evaluate_and_signal` extended | ha.c | Enter COMMITTING; push config+WAL; set role | 4.1, 7.1–7.4 |
| `CTRL_PRIMARY_COMMITTING` state | ha.h + ha_dispatch_gate | Block job traffic; allow repl acks | 5.2, 8.1 |
| `ha_repl_ack_received()` | ha.c | Count standby acks; transition to PRIMARY on quorum | 8.2–8.3 |
| `ha_state_t.repl_ack_count` | ha.h | Atomic ack counter per election round | 7.4, 8.2 |

### Key Interfaces

#### `ha_start_election_async()`
```c
/* Non-blocking. Arms heartbeat watchdog + election watchdog.
 * Connects outgoing vote endpoints. On terminal state:
 * ha_elect_evaluate_and_signal() calls cluster_init_phase2() then sets ha->role. */
void ha_start_election_async(ha_state_t      *ha,
                             cluster_t        *c,
                             int               n_peers,
                             xce_candidate_t **peers,
                             uint64_t          my_config_tag,
                             uint64_t          my_lsn,
                             const char       *my_name);
```

#### `ha_elect_evaluate_and_signal` (extended)

```c
/* After election determines role: */
int is_first_boot = !ha->phase2_done;  /* phase2_done set after first cluster_init_phase2 call */

if (result == CTRL_PRIMARY) {
    if (is_first_boot) {
        cluster_init_phase2(c);    /* load config+data; runs once per process lifetime */
        ha->phase2_done = 1;
    }
    /* Enter COMMITTING: enable hooks but block job traffic until majority ack (Req 8.1) */
    atomic_store(&ha->role, CTRL_PRIMARY_COMMITTING, memory_order_release);
    ha->repl_ack_count = 0;        /* reset ack counter for this round */
    /* Push config and replicate WAL to standbys (Req 7.1, 7.2) */
    push_config_to_standbys(c, ...);
    ha_send_ctrlhello_to_all_standbys(c);   /* triggers catch-up replication */
    /* Arm committing timeout (degraded-mode fallback, Req 8.4) */
    ha_arm_committing_timeout(ha, c);
} else {
    if (is_first_boot) {
        cluster_init_phase2(c);    /* init structures; config arrives via primary push */
        ha->phase2_done = 1;
    }
    atomic_store(&ha->role, CTRL_STANDBY, memory_order_release);
    /* re-arm: heartbeat watchdog fires ha_on_primary_ctrl_fd_rdhup on next timeout */
}
```

#### `ha_repl_ack_received()` — called when a standby acks config+WAL catch-up

```c
void ha_repl_ack_received(ha_state_t *ha, cluster_t *c) {
    int acks = atomic_fetch_add(&ha->repl_ack_count, 1) + 1;
    int quorum = (ha->n_peers + 2) / 2;  /* majority of total cluster */
    if (acks >= quorum - 1) {             /* -1 because primary counts itself */
        ha_disarm_committing_timeout(ha);
        ha_promote_to_primary(c);         /* sets role=CTRL_PRIMARY, enables hooks */
    }
}
```

Called from the existing `handle_ctrlhello` reply path (CtrlHelloReply ack from standby after catch-up) and from `push_config_to_standbys` ack handler.

#### `handle_request_vote` role gate (extended)
```c
ctrl_role_t my_role = atomic_load_explicit(&ha->role, memory_order_acquire);
if (my_role == CTRL_PRIMARY || my_role == CTRL_PRIMARY_COMMITTING)
    return 0;  /* deny — I am primary (or committing); don't grant votes to challengers */
if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(ha))
    return 0;  /* deny — my primary is alive */
/* CTRL_INIT, CTRL_ELECTING, or STANDBY without live primary: evaluate Raft gates below */
```

#### `ha_heartbeat_is_fresh()`
```c
static int ha_heartbeat_is_fresh(const ha_state_t *ha) {
    struct timespec now; clock_gettime(CLOCK_MONOTONIC, &now);
    long ms = (now.tv_sec - ha->last_heartbeat.tv_sec) * 1000
            + (now.tv_nsec - ha->last_heartbeat.tv_nsec) / 1000000;
    return ms < (long)ha->config.heartbeat_timeout_ms;
}
```

#### `myapp_run` (restructured — no blocking)
```c
int myapp_run(myconf_t *myconf) {
    cluster_t c; memset(&c, 0, sizeof(c)); c.myconf = myconf;
    cluster_init_phase1(&c);              /* tunables, managers, endpoints zarray */
    initialize_openssl();
    c.ha.ssl_ctx        = create_ssl_context(myconf->certfile, myconf->keyfile);
    c.ha.client_ssl_ctx = SSL_CTX_new(TLS_client_method());
    SSL_CTX_set_verify(c.ha.client_ssl_ctx, SSL_VERIFY_NONE, NULL);
    iomgr_init(&c.iomgr); iomgr_set_ssl_ctx(&c.iomgr, c.ha.ssl_ctx);
    vtaskmgr_init(&c.vtaskmgr, nvcpus);
    iomgr_start(&c.iomgr); vtaskmgr_start(&c.vtaskmgr);
    cluster_register_listener(&c);        /* listener in epoll before election */
    delaymgr_init(&c.delaymgr); delaymgr_start(&c.delaymgr);  /* moved from cluster_start */
    persist_init(&c.persist, ...); persist_start(&c.persist);   /* moved from cluster_start */
    load_primary_config(&c);              /* pre-load config_tag for freshness gate */
    /* bootstrap_config_if_needed() REMOVED — freshness gate + primary push replaces it */
    /* Build peer array from myconf->peers_cfg ... */
    ha_start_election_async(&c.ha, &c, n_peers, peer_ptrs, my_tag, my_lsn, myconf->name);
    return cluster_run(&c);               /* unchanged — just waits for stop signal */
}
```

**Note**: `vote_ep_ctx_t` carries `cluster_t *c` so that `ha_elect_evaluate_and_signal`
(called from vtask context via `vote_reply_msg_handler`) has access to `cluster_t` for
`cluster_init_phase2`, `push_config_to_standbys`, and `ha_send_ctrlhello_to_all_standbys`.

#### `ha_dispatch_gate` — CTRL_PRIMARY_COMMITTING case

```c
case CTRL_PRIMARY_COMMITTING:
    /* Allow replication acks (config push ack, CtrlHelloReply, WAL ack) so
     * ha_repl_ack_received() can count them toward majority.
     * Deny job dispatch and queue/resource management until majority ack. */
    switch (opcode) {
    case OPCode_heartbeat:
    case OPCode_replConfig:        /* config push ack from standby */
    case OPCode_replConfigPush:
    case OPCode_upgradeConfigAck:
    case OPCode_hello:             /* CtrlHelloReply ack from standby after catch-up */
    case OPCode_walAck:
    case OPCode_replEntry:
    case OPCode_replAck:
    case OPCode_replCatchUpDone:   /* standby signals catch-up complete → count ack */
        return 0;
    default:
        return -XCE_ERR_NOT_READY;
    }
```

`OPCode_replCatchUpDone` handler in `cluster.c` calls `ha_repl_ack_received(ha, c)` to
increment the ack counter and conditionally transition to `CTRL_PRIMARY`.

## Error Handling

- **Connect failure**: endpoint skipped; peer absent from election. After watchdog timeout, election evaluates with available votes only.
- **SSL handshake failure**: `vote_ep_err_handler` calls `handle_request_vote_reply(grant=0)`.
- **`cluster_init_phase2` failure from vtask**: logs ERROR; posts stop to wbuf → clean shutdown.
- **Client during CTRL_ELECTING or CTRL_PRIMARY_COMMITTING**: `ha_dispatch_gate` returns `XCE_ERR_NOT_READY`; client retries.
- **Committing timeout (Req 8.4)**: if majority ack not received within `committing_timeout_ms`, watchdog fires `ha_promote_to_primary` anyway (degraded mode) with a WARN log. Prevents deadlock when standbys are unavailable at startup.
- **Vote re-arm on STANDBY**: `ha_elect_evaluate_and_signal` re-arms `ha_start_election_async`. Heartbeat watchdog fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING → re-election.

## Testing Strategy

### Unit Tests
1. `ha_start_election_async` returns in < 5 ms without blocking
2. `on_vote_connected` encodes and sends correct `OP_requestVote`
3. `vote_reply_msg_handler` reads capnp fields before free — ASAN clean
4. `handle_request_vote` role gate: PRIMARY→deny; STANDBY+fresh→deny; STANDBY+stale→evaluate; ELECTING→evaluate
5. `ha_elect_evaluate_and_signal` enters COMMITTING then PRIMARY after majority ack
6. Election watchdog fires within `election_timeout_ms + 200ms` when no replies
7. `ha_repl_ack_received`: majority acks → `CTRL_PRIMARY`; below majority → stays COMMITTING
8. Committing timeout watchdog: promotes to PRIMARY (degraded) if acks don't arrive

### Integration Tests
9. 3-node startup: ctrl-a wins election → CTRL_PRIMARY_COMMITTING → pushes config to ctrl-b/c → receives replCatchUpDone × 2 → CTRL_PRIMARY → serves jobs
10. ctrl-b/c start with config_tag=0; receive config via primary push (no independent fetch); confirm via replCatchUpDone
11. Primary crash (SIGKILL): ctrl-b heartbeat timeout → re-election → COMMITTING → ack from ctrl-c → PRIMARY; serves traffic from last WAL state
12. Client during COMMITTING receives `XCE_ERR_NOT_READY` and retries successfully after gate opens

### Regression Tests
13. Single-node self-elects; no standbys → committing timeout fires → degraded PRIMARY immediately
14. ASAN clean: no UAF, no stack-use-after-return in election or committing path