myxce/.kiro/steering/ha-async-io.md

HA Protocol & Async I/O

Philosophy

The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.

Single-writer constraint: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).

Process-lifecycle invariant (applies to ALL controller code): a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via exit()/abort()/tlog_exit() as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + stay alive and keep retrying catch-up (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See src/controller/CLAUDE.md → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.

Async I/O Model

Three components in src/lib/:

Component	Thread model	Purpose	Blocking?
iomgr	Single-threaded epoll loop	Inbound packet dispatch, endpoint lifecycle	Never block
vtaskmgr	N-thread pool (default 8, --vcpus CLI override; cluster.c:1082)	Executes vtask handlers for dispatched packets	Return quickly
delaymgr	Single timer thread	Periodic/delayed callbacks, connect(), file I/O	Permitted

Critical rule: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.

Outgoing sends use xce_request(ep, &req) which enqueues to the endpoint's send buffer — always non-blocking. Replies use xce_reply(ep, host, &reply) — also non-blocking.

Connection Ownership

Inbound vote connections (accepted on the listener): Used exclusively for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

Outbound control channels (ha->standby_endpoints): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by vote_ep_err_handler (promoted vote endpoints) and standby_ep_err_handler_primary (late-joiner outbound connections). Freed on disconnect.

Late joiners: primary_heartbeat_fn scans the peers list and opens a fresh outbound connection to any peer absent from standby_endpoints. On connect, on_standby_connected sends ctrlhello, registers the endpoint, and calls repl_add_standby.

Raft Election

The Raft protocol logic lives in the library src/lib/raft/raft.c, consumed verbatim (election, logOk freshness, (term,index) truncation, current-term commit, UpdateTerm). The controller (ha.c/cluster.c) is a thin adapter over it via five callbacks (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced) and the entry points (raft_log_append, raft_on_append_entries_request/reply, raft_advance_commit). The controller does NOT re-implement Raft protocol decisions — landed by spec ha-term-safety (closes bug-851: silent committed-prefix divergence from term-blind hdr.term==0 logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.

• RequestVote + RequestVoteReply over Cap'n Proto
• Freshness gate ((term,lsn), term-primary): a voter grants iff cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len). The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
• Votes persisted atomically with term increment (raft_cb_persist)
• Majority wins: quorum = ceil(n_peers / 2)
• safe_committed_lsn derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised (tag, durable-floor LSN) MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to self_lsn and truncates committed entries — bug-828).
• WAL truncation rule (load-bearing — bug-828): the ONLY truncates are (a) the votes-derived post-election persist_truncate_wal_above(safe_committed_lsn) in raft_cb_become_leader, and (b) the library on_truncate_from (term,lsn) conflict reconcile — both run before the new leader commits anything. NEVER truncate on the lagging learned-committed watermark (last_known_committed_lsn) — it lags the true commit point, so a pre-election/heuristic truncate on it physically destroys committed entries (clamps durable_lsn down). This generalizes persistence.md's "never truncate on a stale ctrlhello".

Key safety invariants:

• currentTerm is monotonic (never decreases)
• votedFor updated atomically with term increment before granting any vote
• commitIndex never decreases (monotonic)

Heartbeat Model

Bidirectional heartbeat with Raft-style AppendEntries semantics:

• Primary → standby: OPCode_nop every heartbeat_interval_ms (3s) via self-perpetuating delaymgr one-shot (ha_start_primary_heartbeat). Re-arms only while role is PRIMARY.
• Standby → primary: Each heartbeat reply carries (applied_lsn, config_tag) via RequestReply.stateReport, giving the primary continuous visibility into standby convergence.
• Standby liveness watchdog: ha_arm_heartbeat_timeout_vtask armed on entering CTRL_STANDBY. If no message arrives from the primary within heartbeat_timeout_ms (10s), triggers re-election via ha_on_primary_ctrl_fd_rdhup.

Replication Consistency (Two-Tier)

Tier 1 — Majority (service gate): Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes CTRL_PRIMARY directly and, at promotion, appends a WAL_NOOP entry of its current term and arms the commit engine's writable-LSN gate (commit_set_writable_lsn). Confirmation of any operation then waits behind committed_lsn >= writable_lsn — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).

Tier 2 — Full peers (continuous best-effort): Primary continuously pushes config and WAL to all standbys via ha_dispatch_repl_action_for_standby on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.

Per-Entry Term & (term,lsn) Reconcile (ha-term-safety)

Every WAL entry carries the leader's real Raft term in wal_entry_hdr_t.term (a value fix — the 40-byte header layout is unchanged). The primary stamps s->term via raft_log_append → on_persist_entry; the live replication carries that term (the legacy term-0 repl_broadcast was retired — a standby persists the leader's term, not 0 or a local substitute). Standbys intake every heartbeat/replication through raft_on_append_entries_request, so UpdateTerm (adopt a higher term + step down) and (term,lsn) conflict-truncate happen on every RPC. Reconcile by (term, lsn), never by LSN alone — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back hdr.term per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the same unified log as a WAL_CONFIG_MARKER (a {version_tag, sha256} reference, no config bytes) that commits + reconciles by (term,lsn) exactly like a job entry; the config bytes travel out-of-band and are hash-verified against the committed marker before activation (mismatch ⇒ loud serve-fence, stay alive + retry).

Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling)

The durable-majority barrier + the own-term WAL_NOOP writable gate (commit.c, see persistence.md) is authoritative for advancing committed_lsn. The raft library's commit_index (fed via on_commit_advanced after raft_advance_commit) is kept advisory only — it is NOT folded in as a hard ceiling (committed = min(durable-majority, raft_commit_index) was tried and removed). Rationale: under the single-writer model the durable-majority watermark is already a current-term (term,lsn)-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a min() ceiling starves commit below the safe durable majority (jobs acked-durable on a majority but never committed). Keep driving the library (for its own match_index/next_index/send state) but never let its commit index gate committed_lsn. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling.

Dispatch Gate

ha_dispatch_gate controls which opcodes are accepted per role:

Role	Allowed
CTRL_PRIMARY	All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist OPCode_notify/update/setpreason/clrpreason is still admitted)
CTRL_STANDBY	nop, reply, hello, join, shutdown, + all replication opcodes
CTRL_INIT / ELECTING	requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer

This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.

Raft-to-Code Mapping

TLA+ Variable	XCE Code
currentTerm[i]	ha->term (_Atomic uint64_t)
state[i]	ha->role (INIT, ELECTING, PRIMARY, STANDBY)
votedFor[i]	ha->voted_for_name[]
log[i]	WAL segments in persist->wal_path
commitIndex[i]	persist->committed_lsn (_Atomic uint64_t)
lastApplied[i]	ha->applied_lsn (_Atomic uint64_t)