myxce/.kiro/specs/raft-drive-async/brief.md

Brief: raft-drive-async

Problem

The Raft protocol drive is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals (iomgr and delaymgr). When either stalls, heartbeats/votes/ACKs stop flowing → peers declare the leader dead → spurious election. This is a likely root of the bug-851 chaotic-failover family.

A blocking audit (2026-06-25) found three CRITICAL violations of the (newly corrected) thread-role contract in ha-async-io.md:

1. Snapshot send + persist_checkpoint inline on iomgr — ha_dispatch_repl_action_for_standby (ha.c:2702) runs persist_checkpoint (fdatasync + WAL-segment trim) + ha_send_snapshot_to_standby (fopen/fread chunk loop, config_repl.c:2446-2503) on the iomgr event loop. A large snapshot freezes ALL heartbeat/vote/client processing for its whole duration.
2. Config-push wait inline on iomgr — push_config_transport_only → wait_all_with_timeout busy-polls usleep(5000) + pthread_join (config_repl.c:1332,1340), called from peer_ctrl_msg_handler paths (ha.c:1172/1177/2672/2677/3973), up to CONF_PUSH_TIMEOUT_MS.
3. Blocking gethostbyname + connect on the delaymgr heartbeat thread — endpoint_connect (netio.c:289/302) via ha_connect_to_standby from primary_heartbeat_fn. One slow/unreachable peer stalls ALL heartbeats + election-timeouts for seconds. (set_fd_nonblocking is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)

Root cause: peer_ctrl_msg_handler is registered directly as ep->msg_handler on peer/vote endpoints — it runs inline on iomgr instead of dispatching to the vtask pool like normal cluster messages (handle_msg → vtask_handle_msg). So any work it does, including the heavy I/O above, executes on the event loop.

Plus MEDIUM vtask-pool blockers (pool-exhaustion, not drive-thread): config-ack wait (config_repl.c:1459), a 30-second usleep checkpoint poll (config_repl.c:1661), ha_persist_term fsync on a vote grant (ha.c:3407 — adds election latency).

Current State

• The corrected thread-role contract is now steering (ha-async-io.md, 2026-06-25): iomgr = non-blocking I/O + dispatch only; delaymgr = trigger-only, never blocks; vtask pool = bounded blocking only under admission reservation; long blocking I/O → dedicated slow lane. This spec brings the code into compliance with that contract. (The contract SUPERSEDED the old "delaymgr for blocking ops = Permitted" rule, which was the design flaw behind these findings — delaymgr is both the heartbeat driver and was the documented blocking sink.)
• Existing primitives to reuse: COMMIT_WAIT_RESERVED admission cap (commit.c:195-203) already keeps vtask-pool headroom against blocking waits; the non-blocking-connect + bounded-poll pattern already exists in xce_probe_config_tag (config_repl.c:1753-1761); vtask_handle_msg (vtaskmgr.c:356) is the dispatch path but is endpoint+inbound-packet-bound (vtype=VTASK_IO).
• Confirmed already-correct: RequestVote fan-out, heartbeat fan-out, WAL replication streaming all use non-blocking xce_request() enqueue; standby-side replicated WAL write + fdatasync run on a vtask worker, not iomgr.

Desired Outcome

The protocol drive is non-blocking on both liveness threads, by construction:

• iomgr does only non-blocking socket I/O + dispatch — peer_ctrl_msg_handler's real work is dispatched to the vtask pool (conforming to the existing handle_msg pattern), so no future handler addition can reintroduce inline blocking.
• Inside the vtask, fast protocol work runs inline (decode ack → update match_index → advance commit → send next heartbeat — all bounded); long I/O is triggered to a slow lane (snapshot transfer, config-push), never run in the request-pool worker.
• delaymgr only fires timers and triggers work; the peer connect is bounded (non-blocking + poll timeout) and offloaded off the timer thread.
• Heartbeat liveness is provably protected: a partitioned (unreachable, not merely dead) peer cannot stall heartbeats to healthy peers or trigger a spurious election.

Approach

Bring the code into compliance with the steering thread-role contract, structurally rather than per-op:

1. Dispatch peer_ctrl_msg_handler to the vtask pool (the structural fix). iomgr becomes epoll + enqueue only. Preserve per-connection message ordering (VTASK_IO per-endpoint serialization is a feature here). Marshal any iomgr-owned state mutations (epoll/endpoint lifecycle) back to the iomgr thread (bug-646 discipline).
2. Two-tier execution inside the vtask: fast bounded protocol work inline; trigger long I/O (snapshot send + persist_checkpoint, config-push) to a dedicated slow-lane / admission-capped executor so multi-second tasks cannot starve the request pool. Per-peer dedup (one snapshot in flight per standby); coordinate with ongoing WAL-catchup stream and the primary's appends/WAL-trim.
3. delaymgr connect → bounded + offloaded: reuse the xce_probe_config_tag non-blocking-connect + bounded-poll pattern; trigger the reconnect as an idempotent, refcounted, deduped task off the delaymgr thread; marshal the connected-fd epoll registration back to iomgr. DNS (gethostbyname) is LOW real-world risk (HPC peers are IP//etc/hosts) — defer async DNS unless measured.
4. General (non-IO) task-submit path: add the submit primitive needed for #2/#3 (current vtask_handle_msg requires an inbound packet); or generalize it.
5. Admission discipline: any offloaded blocking task lives under a reservation (extend the COMMIT_WAIT_RESERVED pattern) so liveness-critical messages always have a free worker.

Scope

• In:
  • Dispatch peer_ctrl_msg_handler real work to the vtask pool (root-cause structural fix).
  • Snapshot send + persist_checkpoint off iomgr → slow lane, with per-peer dedup + WAL coordination.
  • Config-push wait off iomgr → slow lane / async ACK.
  • delaymgr peer connect → bounded non-blocking + offloaded; reconnect dedup/refcount/marshal-back.
  • General non-IO vtask submit path + admission reservation for offloaded blocking tasks.
  • Partition (unreachable-peer) regression test; verify no spurious election + heartbeats to survivors continue.
• Out:
  • The commit-barrier async conversion (P1 / repl_wait_majority_ack → resolve) — owned by raft-apply-decoupling as its keystone (different concern: that's the commit drive, this is the transport/timer drive).
  • Apply-after-commit / hook decoupling / facts-through-raft (raft-apply-decoupling).
  • Async DNS (deferred unless measured).
  • The MEDIUM vtask-pool blockers beyond what the admission discipline naturally covers (note them; fix opportunistically).
  • Entity-bearing snapshots / config-upgrade snapshot (separate specs).

Boundary Candidates

• The peer_ctrl_msg_handler → vtask dispatch seam.
• The dedicated slow-lane executor + admission reservation.
• The general non-IO vtask-submit primitive.
• The bounded-connect + offloaded-reconnect path.
• The iomgr-owned-state marshal-back mechanism (epoll/endpoint lifecycle).

Out of Boundary

• Commit-engine internals (P1 owns the commit-drive async).
• Snapshot content format (entity-bearing snapshots is a separate spec) — this spec only moves the transfer off the drive thread.

Upstream / Downstream

• Upstream: the ha-async-io.md thread-role contract (authority); existing COMMIT_WAIT_RESERVED (commit.c) and xce_probe_config_tag bounded-connect pattern (reuse); election-iomgr-refactor (COMPLETED upstream — provides reusable primitives delay_task_ms, on_connected, ha_start_election_async/non-blocking vote fan-out).
• Downstream: raft-apply-decoupling (its async commit drive + apply pump reuse the slow-lane + admission infrastructure this spec establishes; and the drive must be non-blocking before apply is restructured on top of it); safe failover for the whole bug-851 family depends on this landing.

Existing Spec Touchpoints

• Adjacent (RESOLVED standalone 2026-06-25): election-iomgr-refactor is ADJACENT-SAME-AREA — it fixed the startup blocking path and is fully implemented; this spec fixes the runtime drive-thread paths it never touched. Keep separate; consume it as upstream. (Scope-check verdict in research.md.)
• Coordinate: ha-term-safety (touches the same peer_ctrl_msg_handler / repl dispatch region); ha-wal-replication (snapshot/catch-up transport); controller-status (heartbeat/committed propagation on the same drive).
• Verification home: raft-sim-conformance (add the partition/liveness assertion) and /xce-test-ha (partition variant).

Constraints

• The corrected thread-role contract (ha-async-io.md) is authoritative — iomgr and delaymgr never block inline; long I/O on the slow lane; bounded blocking in vtask only under admission reservation.
• "Controllers never self-terminate" — an unreachable peer is a loud retry, never an abort.
• Preserve per-connection message ordering when dispatching peer_ctrl_msg_handler to vtask.
• Must not regress the heartbeat fast path measurably at cluster scale (3–5 peers); per-message vtask overhead accepted in exchange for uniformity + liveness safety.
• Verified by a partition (unreachable, not killed) soak — the case /xce-test-ha's current kill-only soak does not reproduce.

Provenance

Surfaced 2026-06-25 during the raft-apply-decoupling discovery as prerequisite P2 (drive hardening); promoted to its own spec because it threatens liveness independently of the apply refactor. Full analysis: .kiro/specs/raft-apply-decoupling/facts-in-raft.md § "Two distinct prerequisites" + P2 offload mechanism. Thread-role contract: .kiro/steering/ha-async-io.md (corrected 2026-06-25).