# HA Protocol & Async I/O

## Philosophy

The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.

**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).

**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.

## Async I/O Model

Three components in `src/lib/`:

| Component | Thread model | Purpose | Blocking? |
|-----------|-------------|---------|-----------|
| **iomgr** | Single-threaded epoll loop | Inbound packet dispatch, endpoint lifecycle | **Never block** |
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | Executes vtask handlers for dispatched packets | **Return quickly** |
| **delaymgr** | Single timer thread | Periodic/delayed callbacks, connect(), file I/O | **Permitted** |

**Critical rule**: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.

Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.

## Connection Ownership

**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.

**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.

## Raft Election

`src/controller/raft.c` implements standard Raft-style election:

- **RequestVote + RequestVoteReply** over Cap'n Proto
- **Freshness gate**: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
- **Votes persisted atomically** with term increment (raft_cb_persist)
- **Majority wins**: quorum = ceil(n_peers / 2)
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning
- **WAL truncation** on promotion: `persist_truncate_wal_above(safe_committed_lsn)` removes diverging entries above the safe commit point

**Key safety invariants**:
- `currentTerm` is monotonic (never decreases)
- `votedFor` updated atomically with term increment before granting any vote
- `commitIndex` never decreases (monotonic)

## Heartbeat Model

Bidirectional heartbeat with Raft-style AppendEntries semantics:

- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.

## Replication Consistency (Two-Tier)

**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).

**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.

## Dispatch Gate

`ha_dispatch_gate` controls which opcodes are accepted per role:

| Role | Allowed |
|------|---------|
| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |

This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.

## Raft-to-Code Mapping

| TLA+ Variable | XCE Code |
|--------------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
| `votedFor[i]` | `ha->voted_for_name[]` |
| `log[i]` | WAL segments in `persist->wal_path` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |