myxce/.kiro/steering/ha-async-io.md

HA Protocol & Async I/O

Philosophy

The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.

Single-writer constraint: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).

Process-lifecycle invariant (applies to ALL controller code): a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via exit()/abort()/tlog_exit() as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + stay alive and keep retrying catch-up (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See src/controller/CLAUDE.md → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.

Async I/O Model

Three components in src/lib/ plus a dedicated slow-lane executor:

Component	Thread model	May do	Must NOT do
iomgr	Single-threaded epoll loop	non-blocking socket I/O; endpoint lifecycle; dispatch to vtask	any blocking call; any inline message processing (dispatch it)
vtaskmgr	N-thread pool (default 8, --vcpus CLI override; cluster.c:1743-1745)	bounded handler work; bounded blocking ONLY under admission reservation	long/unbounded blocking I/O (snapshot transfer, config-push) — starves the pool
delaymgr	Single timer thread	fire timers; trigger (enqueue) work; return fast	connect(), file I/O, any blocking — it drives heartbeats + election timeouts
slow lane	dedicated executor / admission-capped pool	long blocking I/O: reconnect, snapshot transfer, config-push	— (this is where blocking ops go, NOT delaymgr)

Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler). No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:

1. iomgr does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
2. delaymgr only fires timers and triggers work, then returns. It NEVER blocks (no connect(), no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
3. vtask pool runs bounded handler work. Bounded blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (COMMIT_WAIT_RESERVED, wait_cap/wait_admit_locked, commit.c:241-271) that keeps workers free for liveness-critical messages. Long/unbounded blocking I/O does NOT belong here.
4. Long blocking I/O (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.

SUPERSEDES (2026-06-25): the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is also the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking connect() there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + persist_checkpoint inline on iomgr (ha.c:2702), push_config_transport_only busy-poll+join on iomgr (config_repl.c:1332), and blocking gethostbyname+connect on the delaymgr thread (netio.c:289/302). All three are now fixed — spec raft-drive-async (IMPLEMENTED, validate-impl GO 2026-07-18, 21/21 tasks) moved each off its liveness thread: snapshot transfer now runs on a dedicated slow-lane worker (snapshot_lane_fn, ha.c:4021, kept off the iomgr thread — see the "DEFAULT ON" note at ha.c:5568); push_config_transport_only now spawns a dedicated pthread per standby (pthread_create, config_repl.c:1589) instead of busy-polling+joining on iomgr; and socket_connect (netio.c:302, doc comment netio.c:287) is now a bounded, non-blocking TCP connect via poll() with connect_timeout_ms, so it no longer pins the delaymgr heartbeat thread. See bug-1447 below for a same-day crash this spec's own validation soak surfaced and fixed.

Outgoing sends use xce_request(ep, &req) which enqueues to the endpoint's send buffer — always non-blocking. Replies use xce_reply(ep, host, &reply) — also non-blocking.

Same-batch endpoint lifetime (bug-1447, fixed 2026-07-18). io_worker's epoll loop processes every fd from ONE epoll_wait() batch (events[MAX_EPOLL_EVENTS]) in a single pass. A handler for one event in that batch (e.g. on_standby_connected evicting a stale duplicate outbound channel via iomgr_unmonitor()+endpoint_unref()) can free a DIFFERENT endpoint that ALSO has a ready event later in the SAME batch array — epoll_ctl(DEL) only stops future epoll_wait() calls, it does not invalidate an event already sitting in the current batch. The later event then dereferences a freed endpoint_t/SSL* (heap-use-after-free). INVOKE_MSG_HANDLER's ref/unref bracket only protects the endpoint a handler is CURRENTLY running on, not a different endpoint torn down as a side effect. Fix: io_worker_dispatch_batch() takes endpoint_ref() on every endpoint in the batch before dispatching any of them, and endpoint_unref()s each only after the whole batch completes — same lock-free bracket idiom as bug-1298, applied to the batch as a unit. When adding any handler that can tear down an endpoint other than its own, this batch-wide ref bracket is why that's safe — do not "optimize" it away as redundant with INVOKE_MSG_HANDLER's per-call ref.

Connection Ownership

Inbound vote connections (accepted on the listener): Used exclusively for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).

Outbound control channels (ha->standby_endpoints): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by vote_ep_err_handler (promoted vote endpoints) and standby_ep_err_handler_primary (late-joiner outbound connections). Freed on disconnect.

Late joiners: primary_heartbeat_fn scans the peers list and opens a fresh outbound connection to any peer absent from standby_endpoints. On connect, on_standby_connected sends ctrlhello, registers the endpoint, and calls repl_add_standby.

Raft Election

The Raft protocol logic lives in the library src/lib/raft/raft.c, consumed verbatim (election, logOk freshness, (term,index) truncation, current-term commit, UpdateTerm). The controller (ha.c/cluster.c) is a thin adapter over it via five callbacks (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced) and the entry points (raft_log_append, raft_on_append_entries_request/reply, raft_advance_commit). The controller does NOT re-implement Raft protocol decisions — landed by spec ha-term-safety (closes bug-851: silent committed-prefix divergence from term-blind hdr.term==0 logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.

Vote-request send failure MUST still reach the reply tally (bug-1423/1424). raft_cb_send_vote_request / raft_cb_send_pre_vote_request send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via vote_reply_dispatch/pre_vote_reply_dispatch (→ raft_on_vote_reply/raft_on_pre_vote_reply). A endpoint_connect/iomgr_monitor failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only handle_request_vote_reply. Without this, the library's replies_received == n_peers - 1 fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in CTRL_ELECTING in a 3-node cluster with one node down before this fix.

• RequestVote + RequestVoteReply over Cap'n Proto
• Freshness gate ((term,lsn), term-primary): a voter grants iff cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len). The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
• Votes persisted atomically with term increment (raft_cb_persist)
• Majority wins: quorum = ceil(n_peers / 2)
• safe_committed_lsn derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised (tag, durable-floor LSN) MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to self_lsn and truncates committed entries — bug-828).
• WAL truncation rule (load-bearing — bug-828): every persist_truncate_wal_above call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (last_known_committed_lsn), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps durable_lsn down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election persist_truncate_wal_above(safe_committed_lsn) in raft_cb_become_leader, (b) the library on_truncate_from (term,lsn) conflict reconcile, both running before the new leader commits anything; (c) ha_demotion_convergence_fn (ha_convergence.c), which truncates to the node's own local committed_lsn on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) persist_wal_write_replicated's same-LSN payload-conflict repair (persist.c, truncate_wal_above_locked(p, lsn-1)), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores committed_lsn verbatim afterward — never regresses the committed boundary. NEVER truncate on the lagging learned-committed watermark (last_known_committed_lsn). This generalizes persistence.md's "never truncate on a stale ctrlhello".

Key safety invariants:

• currentTerm is monotonic (never decreases)
• votedFor updated atomically with term increment before granting any vote
• commitIndex never decreases (monotonic)

Heartbeat Model

Bidirectional heartbeat with Raft-style AppendEntries semantics:

• Primary → standby: OPCode_nop every heartbeat_interval_ms (3s) via self-perpetuating delaymgr one-shot (ha_start_primary_heartbeat). Re-arms only while role is PRIMARY.
• Standby → primary: Each heartbeat reply carries (applied_lsn, config_tag, term) via RequestReply.stateReport, giving the primary continuous visibility into standby convergence.
• Heartbeat-reply term-discovery fallback (bug-1421). A stale-term primary does NOT reliably learn a higher term from an AE reject: ha_raft_on_repl_ack only routes AE replies through raft_on_append_entries_reply on success=1, and that function itself drops any reply whose term doesn't match (raft.c stale-term drop) — so an AE reject can never adopt a higher term through the normal path. stateReport.term (above) and a WAL-NACK's carried term are the two fallback vectors: both call raft_on_higher_term_observed(s, observed_term) (strictly-greater-than only; touches term/voted_for/role; on_persist before on_become_follower), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
• Standby liveness watchdog: ha_arm_heartbeat_timeout_vtask armed on entering CTRL_STANDBY. If no message arrives from the primary within heartbeat_timeout_ms (10s), triggers re-election via ha_on_primary_ctrl_fd_rdhup. Every return path must re-arm (bug-1562). The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within 2 * election_timeout_ms of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (role == CTRL_ELECTING || pre_vote_in_progress). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call ha_arm_heartbeat_timeout_vtask before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary).

Replication Consistency (Two-Tier)

Tier 1 — Majority (service gate): Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes CTRL_PRIMARY directly and, at promotion, appends a WAL_NOOP entry of its current term and arms the commit engine's writable-LSN gate (commit_set_writable_lsn). Confirmation of any operation then waits behind committed_lsn >= writable_lsn — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).

Tier 2 — Full peers (continuous best-effort): Primary continuously pushes config and WAL to all standbys via ha_dispatch_repl_action_for_standby on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.

Per-Entry Term & (term,lsn) Reconcile (ha-term-safety)

Every WAL entry carries the leader's real Raft term in wal_entry_hdr_t.term (a value fix — the 40-byte header layout is unchanged). The primary stamps s->term via raft_log_append → on_persist_entry; the live replication carries that term (the legacy term-0 repl_broadcast was retired — a standby persists the leader's term, not 0 or a local substitute). Standbys intake every heartbeat/replication through raft_on_append_entries_request, so UpdateTerm (adopt a higher term + step down) and (term,lsn) conflict-truncate happen on every RPC. Reconcile by (term, lsn), never by LSN alone — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back hdr.term per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the same unified log as a WAL_CONFIG_MARKER (a {version_tag, sha256} reference, no config bytes) that commits + reconciles by (term,lsn) exactly like a job entry; the config bytes travel out-of-band and are hash-verified against the committed marker before activation (mismatch ⇒ loud serve-fence, stay alive + retry).

Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling)

The durable-majority barrier + the own-term WAL_NOOP writable gate (commit.c, see persistence.md) is authoritative for advancing committed_lsn. The raft library's commit_index (fed via on_commit_advanced after raft_advance_commit) is kept advisory only — it is NOT folded in as a hard ceiling (committed = min(durable-majority, raft_commit_index) was tried and removed). Rationale: under the single-writer model the durable-majority watermark is already a current-term (term,lsn)-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a min() ceiling starves commit below the safe durable majority (jobs acked-durable on a majority but never committed). Keep driving the library (for its own match_index/next_index/send state) but never let its commit index gate committed_lsn. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling.

Dispatch Gate

ha_dispatch_gate controls which opcodes are accepted per role:

Role	Allowed
CTRL_PRIMARY	All opcodes, subject to two fences checked before the quorum gate (ha_dispatch_gate, ha.c:4525): config_serve_fenced() (ha-term-safety task 4.3) and upgrade_materialization_fenced() (config-upgrade-snapshot task 6.2) each refuse state-changing ops while set, admitting only replication/read-only/status opcodes. Once unfenced: client ops are subject to write quorum; under no quorum, the fact allowlist OPCode_notify/update/setpreason/clrpreason (plus read-only/status/replication opcodes) is still admitted
CTRL_STANDBY	nop, reply, hello, join, shutdown, heartbeat, redirect, steppingDown, requestVote, preVoteRequest, probeConfigTag, upgradeConfigAck, ctrlstatus, + all replication opcodes (replEntry, replAck, replSnapshot, replCatchUpDone, replConfig, replConfigPush, walAck, snapshotTransfer, walCatchupEntry)
CTRL_INIT / ELECTING	requestVote, preVoteRequest, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer, ctrlstatus

This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.

Raft-to-Code Mapping

TLA+ Variable	XCE Code
currentTerm[i]	ha->term (_Atomic uint64_t)
state[i]	ha->role (INIT, ELECTING, PRIMARY, STANDBY)
votedFor[i]	ha->voted_for_name[]
log[i]	WAL segments in persist->wal_path
commitIndex[i]	persist->committed_lsn (_Atomic uint64_t)
lastApplied[i]	ha->applied_lsn (_Atomic uint64_t)