myxce/.kiro/steering/ha-async-io.md

HA Protocol & Async I/O

Philosophy

The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.

Single-writer constraint: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).

Process-lifecycle invariant (applies to ALL controller code): a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via exit()/abort()/tlog_exit() as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + stay alive and keep retrying catch-up (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See src/controller/CLAUDE.md → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.

Async I/O Model

Three components in src/lib/:

Component	Thread model	Purpose	Blocking?
iomgr	Single-threaded epoll loop	Inbound packet dispatch, endpoint lifecycle	Never block
vtaskmgr	N-thread pool (default 8, --vcpus CLI override; cluster.c:1082)	Executes vtask handlers for dispatched packets	Return quickly
delaymgr	Single timer thread	Periodic/delayed callbacks, connect(), file I/O	Permitted

Critical rule: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.

Outgoing sends use xce_request(ep, &req) which enqueues to the endpoint's send buffer — always non-blocking. Replies use xce_reply(ep, host, &reply) — also non-blocking.

Connection Ownership

Inbound vote connections (accepted on the listener): Used exclusively for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

Outbound control channels (ha->standby_endpoints): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by vote_ep_err_handler (promoted vote endpoints) and standby_ep_err_handler_primary (late-joiner outbound connections). Freed on disconnect.

Late joiners: primary_heartbeat_fn scans the peers list and opens a fresh outbound connection to any peer absent from standby_endpoints. On connect, on_standby_connected sends ctrlhello, registers the endpoint, and calls repl_add_standby.

Raft Election

src/controller/raft.c implements standard Raft-style election:

• RequestVote + RequestVoteReply over Cap'n Proto
• Freshness gate: candidate must have lastLogTerm >= voter AND lastLogIndex >= voter
• Votes persisted atomically with term increment (raft_cb_persist)
• Majority wins: quorum = ceil(n_peers / 2)
• safe_committed_lsn derived from voter RequestVoteReply messages — no round-trip needed after winning
• WAL truncation on promotion: persist_truncate_wal_above(safe_committed_lsn) removes diverging entries above the safe commit point

Key safety invariants:

• currentTerm is monotonic (never decreases)
• votedFor updated atomically with term increment before granting any vote
• commitIndex never decreases (monotonic)

Heartbeat Model

Bidirectional heartbeat with Raft-style AppendEntries semantics:

• Primary → standby: OPCode_nop every heartbeat_interval_ms (3s) via self-perpetuating delaymgr one-shot (ha_start_primary_heartbeat). Re-arms only while role is PRIMARY.
• Standby → primary: Each heartbeat reply carries (applied_lsn, config_tag) via RequestReply.stateReport, giving the primary continuous visibility into standby convergence.
• Standby liveness watchdog: ha_arm_heartbeat_timeout_vtask armed on entering CTRL_STANDBY. If no message arrives from the primary within heartbeat_timeout_ms (10s), triggers re-election via ha_on_primary_ctrl_fd_rdhup.

Replication Consistency (Two-Tier)

Tier 1 — Majority (service gate): Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes CTRL_PRIMARY directly and, at promotion, appends a WAL_NOOP entry of its current term and arms the commit engine's writable-LSN gate (commit_set_writable_lsn). Confirmation of any operation then waits behind committed_lsn >= writable_lsn — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).

Tier 2 — Full peers (continuous best-effort): Primary continuously pushes config and WAL to all standbys via ha_dispatch_repl_action_for_standby on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.

Dispatch Gate

ha_dispatch_gate controls which opcodes are accepted per role:

Role	Allowed
CTRL_PRIMARY	All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist OPCode_notify/update/setpreason/clrpreason is still admitted)
CTRL_STANDBY	nop, reply, hello, join, shutdown, + all replication opcodes
CTRL_INIT / ELECTING	requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer

This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.

Raft-to-Code Mapping

TLA+ Variable	XCE Code
currentTerm[i]	ha->term (_Atomic uint64_t)
state[i]	ha->role (INIT, ELECTING, PRIMARY, STANDBY)
votedFor[i]	ha->voted_for_name[]
log[i]	WAL segments in persist->wal_path
commitIndex[i]	persist->committed_lsn (_Atomic uint64_t)
lastApplied[i]	ha->applied_lsn (_Atomic uint64_t)