myxce/.kiro/steering/ha-async-io.md

# HA Protocol & Async I/O

## Philosophy

The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.

**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).

**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.

## Async I/O Model

Three components in `src/lib/` plus a dedicated slow-lane executor:

| Component | Thread model | May do | Must NOT do |
|-----------|-------------|--------|-------------|
| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |

**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).**
No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, commit.c:195-203) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.

> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`) — tracked as the `raft-drive-async` / P2 work in roadmap.md.

Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.

## Connection Ownership

**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.

**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.

## Raft Election

The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.

- **RequestVote + RequestVoteReply** over Cap'n Proto
- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
- **Votes persisted atomically** with term increment (raft_cb_persist)
- **Majority wins**: quorum = ceil(n_peers / 2)
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
- **WAL truncation rule (load-bearing — bug-828)**: the **ONLY** truncates are (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, and (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile — both run *before* the new leader commits anything. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`) — it lags the true commit point, so a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".

**Key safety invariants**:
- `currentTerm` is monotonic (never decreases)
- `votedFor` updated atomically with term increment before granting any vote
- `commitIndex` never decreases (monotonic)

## Heartbeat Model

Bidirectional heartbeat with Raft-style AppendEntries semantics:

- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.

## Replication Consistency (Two-Tier)

**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).

**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.

## Per-Entry Term & `(term,lsn)` Reconcile (ha-term-safety)

Every WAL entry carries the **leader's real Raft term** in `wal_entry_hdr_t.term` (a value fix — the 40-byte header layout is unchanged). The primary stamps `s->term` via `raft_log_append` → `on_persist_entry`; the **live replication carries that term** (the legacy term-0 `repl_broadcast` was retired — a standby persists the *leader's* term, not 0 or a local substitute). Standbys intake every heartbeat/replication through `raft_on_append_entries_request`, so `UpdateTerm` (adopt a higher term + step down) and `(term,lsn)` conflict-truncate happen on every RPC. **Reconcile by `(term, lsn)`, never by LSN alone** — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back `hdr.term` per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the **same unified log** as a `WAL_CONFIG_MARKER` (a `{version_tag, sha256}` reference, no config bytes) that commits + reconciles by `(term,lsn)` exactly like a job entry; the config *bytes* travel out-of-band and are **hash-verified against the committed marker before activation** (mismatch ⇒ loud serve-fence, stay alive + retry).

## Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling)

The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commit.c`, see `persistence.md`) is **authoritative** for advancing `committed_lsn`. The raft library's `commit_index` (fed via `on_commit_advanced` after `raft_advance_commit`) is kept **advisory only** — it is NOT folded in as a hard ceiling (`committed = min(durable-majority, raft_commit_index)` was tried and removed). Rationale: under the single-writer model the durable-majority watermark is *already* a current-term `(term,lsn)`-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a `min()` ceiling **starves commit below the safe durable majority** (jobs acked-durable on a majority but never committed). Keep driving the library (for its own `match_index`/`next_index`/send state) but never let its commit index *gate* `committed_lsn`. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling.

## Dispatch Gate

`ha_dispatch_gate` controls which opcodes are accepted per role:

| Role | Allowed |
|------|---------|
| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |

This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.

## Raft-to-Code Mapping

| TLA+ Variable | XCE Code |
|--------------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
| `votedFor[i]` | `ha->voted_for_name[]` |
| `log[i]` | WAL segments in `persist->wal_path` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |