myxce/.kiro/steering/ha-async-io.md

# HA Protocol & Async I/O

## Philosophy

The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.

**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).

**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.

## Async I/O Model

Three components in `src/lib/` plus a dedicated slow-lane executor:

| Component | Thread model | May do | Must NOT do |
|-----------|-------------|--------|-------------|
| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1743-1745) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |

**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).**
No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, `wait_cap`/`wait_admit_locked`, commit.c:241-271) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.

> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`). **All three are now fixed** — spec `raft-drive-async` (IMPLEMENTED, validate-impl GO 2026-07-18, 21/21 tasks) moved each off its liveness thread: snapshot transfer now runs on a dedicated slow-lane worker (`snapshot_lane_fn`, `ha.c:4021`, kept off the iomgr thread — see the "DEFAULT ON" note at `ha.c:5568`); `push_config_transport_only` now spawns a dedicated pthread per standby (`pthread_create`, `config_repl.c:1589`) instead of busy-polling+joining on iomgr; and `socket_connect` (`netio.c:302`, doc comment `netio.c:287`) is now a bounded, non-blocking TCP connect via `poll()` with `connect_timeout_ms`, so it no longer pins the delaymgr heartbeat thread. See bug-1447 below for a same-day crash this spec's own validation soak surfaced and fixed.

Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.

**Same-batch endpoint lifetime (bug-1447, fixed 2026-07-18).** `io_worker`'s epoll loop processes every fd from ONE `epoll_wait()` batch (`events[MAX_EPOLL_EVENTS]`) in a single pass. A handler for one event in that batch (e.g. `on_standby_connected` evicting a stale duplicate outbound channel via `iomgr_unmonitor()`+`endpoint_unref()`) can free a DIFFERENT endpoint that ALSO has a ready event later in the SAME batch array — `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, it does not invalidate an event already sitting in the current batch. The later event then dereferences a freed `endpoint_t`/`SSL*` (heap-use-after-free). `INVOKE_MSG_HANDLER`'s ref/unref bracket only protects the endpoint a handler is CURRENTLY running on, not a different endpoint torn down as a side effect. Fix: `io_worker_dispatch_batch()` takes `endpoint_ref()` on every endpoint in the batch before dispatching any of them, and `endpoint_unref()`s each only after the whole batch completes — same lock-free bracket idiom as bug-1298, applied to the batch as a unit. When adding any handler that can tear down an endpoint other than its own, this batch-wide ref bracket is why that's safe — do not "optimize" it away as redundant with `INVOKE_MSG_HANDLER`'s per-call ref.

## Connection Ownership

**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.

**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.

## Raft Election

The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.

**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix.

- **RequestVote + RequestVoteReply** over Cap'n Proto
- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
- **Votes persisted atomically** with term increment (raft_cb_persist)
- **Majority wins**: quorum = ceil(n_peers / 2)
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".

**Key safety invariants**:
- `currentTerm` is monotonic (never decreases)
- `votedFor` updated atomically with term increment before granting any vote
- `commitIndex` never decreases (monotonic)

## Heartbeat Model

Bidirectional heartbeat with Raft-style AppendEntries semantics:

- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. **Every return path must re-arm (bug-1562).** The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within `2 * election_timeout_ms` of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (`role == CTRL_ELECTING || pre_vote_in_progress`). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call `ha_arm_heartbeat_timeout_vtask` before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary).

## Replication Consistency (Two-Tier)

**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).

**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.

## Per-Entry Term & `(term,lsn)` Reconcile (ha-term-safety)

Every WAL entry carries the **leader's real Raft term** in `wal_entry_hdr_t.term` (a value fix — the 40-byte header layout is unchanged). The primary stamps `s->term` via `raft_log_append` → `on_persist_entry`; the **live replication carries that term** (the legacy term-0 `repl_broadcast` was retired — a standby persists the *leader's* term, not 0 or a local substitute). Standbys intake every heartbeat/replication through `raft_on_append_entries_request`, so `UpdateTerm` (adopt a higher term + step down) and `(term,lsn)` conflict-truncate happen on every RPC. **Reconcile by `(term, lsn)`, never by LSN alone** — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back `hdr.term` per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the **same unified log** as a `WAL_CONFIG_MARKER` (a `{version_tag, sha256}` reference, no config bytes) that commits + reconciles by `(term,lsn)` exactly like a job entry; the config *bytes* travel out-of-band and are **hash-verified against the committed marker before activation** (mismatch ⇒ loud serve-fence, stay alive + retry).

## Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling)

The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commit.c`, see `persistence.md`) is **authoritative** for advancing `committed_lsn`. The raft library's `commit_index` (fed via `on_commit_advanced` after `raft_advance_commit`) is kept **advisory only** — it is NOT folded in as a hard ceiling (`committed = min(durable-majority, raft_commit_index)` was tried and removed). Rationale: under the single-writer model the durable-majority watermark is *already* a current-term `(term,lsn)`-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a `min()` ceiling **starves commit below the safe durable majority** (jobs acked-durable on a majority but never committed). Keep driving the library (for its own `match_index`/`next_index`/send state) but never let its commit index *gate* `committed_lsn`. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling.

## Dispatch Gate

`ha_dispatch_gate` controls which opcodes are accepted per role:

| Role | Allowed |
|------|---------|
| **CTRL_PRIMARY** | All opcodes, subject to two fences checked before the quorum gate (`ha_dispatch_gate`, ha.c:4525): `config_serve_fenced()` (ha-term-safety task 4.3) and `upgrade_materialization_fenced()` (config-upgrade-snapshot task 6.2) each refuse state-changing ops while set, admitting only replication/read-only/status opcodes. Once unfenced: client ops are subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` (plus read-only/status/replication opcodes) is still admitted |
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, heartbeat, redirect, steppingDown, requestVote, preVoteRequest, probeConfigTag, upgradeConfigAck, ctrlstatus, + all replication opcodes (replEntry, replAck, replSnapshot, replCatchUpDone, replConfig, replConfigPush, walAck, snapshotTransfer, walCatchupEntry) |
| **CTRL_INIT / ELECTING** | requestVote, preVoteRequest, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer, ctrlstatus |

This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.

## Raft-to-Code Mapping

| TLA+ Variable | XCE Code |
|--------------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
| `votedFor[i]` | `ha->voted_for_name[]` |
| `log[i]` | WAL segments in `persist->wal_path` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |