# Brief: raft-drive-async

## Problem

The Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals (iomgr and delaymgr). When either stalls, heartbeats/votes/ACKs stop flowing → peers declare the leader dead → spurious election. This is a likely root of the **bug-851 chaotic-failover family**.

A blocking audit (2026-06-25) found three CRITICAL violations of the (newly corrected) thread-role contract in `ha-async-io.md`:

1. **Snapshot send + `persist_checkpoint` inline on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) runs `persist_checkpoint` (fdatasync + WAL-segment trim) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503) on the iomgr event loop. A large snapshot freezes ALL heartbeat/vote/client processing for its whole duration.
2. **Config-push wait inline on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172/1177/2672/2677/3973), up to `CONF_PUSH_TIMEOUT_MS`.
3. **Blocking `gethostbyname` + `connect` on the delaymgr heartbeat thread** — `endpoint_connect` (netio.c:289/302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unreachable peer stalls ALL heartbeats + election-timeouts for seconds. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)

**Root cause:** `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** on peer/vote endpoints — it runs inline on iomgr instead of dispatching to the vtask pool like normal cluster messages (`handle_msg` → `vtask_handle_msg`). So any work it does, including the heavy I/O above, executes on the event loop.

Plus MEDIUM vtask-pool blockers (pool-exhaustion, not drive-thread): config-ack wait (config_repl.c:1459), a 30-second `usleep` checkpoint poll (config_repl.c:1661), `ha_persist_term` fsync on a vote grant (ha.c:3407 — adds election latency).

## Current State

- The corrected **thread-role contract** is now steering (`ha-async-io.md`, 2026-06-25): iomgr = non-blocking I/O + dispatch only; delaymgr = trigger-only, never blocks; vtask pool = bounded blocking only under admission reservation; long blocking I/O → dedicated slow lane. This spec brings the **code into compliance** with that contract. (The contract SUPERSEDED the old "delaymgr for blocking ops = Permitted" rule, which was the design flaw behind these findings — delaymgr is both the heartbeat driver and was the documented blocking sink.)
- Existing primitives to reuse: `COMMIT_WAIT_RESERVED` admission cap (commit.c:195-203) already keeps vtask-pool headroom against blocking waits; the non-blocking-connect + bounded-`poll` pattern already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); `vtask_handle_msg` (vtaskmgr.c:356) is the dispatch path but is endpoint+inbound-packet-bound (`vtype=VTASK_IO`).
- Confirmed already-correct: RequestVote fan-out, heartbeat fan-out, WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + fdatasync run on a vtask worker, not iomgr.

## Desired Outcome

The protocol drive is non-blocking on both liveness threads, by construction:

- **iomgr** does only non-blocking socket I/O + dispatch — `peer_ctrl_msg_handler`'s real work is dispatched to the vtask pool (conforming to the existing `handle_msg` pattern), so no future handler addition can reintroduce inline blocking.
- Inside the vtask, **fast protocol work runs inline** (decode ack → update match_index → advance commit → send next heartbeat — all bounded); **long I/O is triggered to a slow lane** (snapshot transfer, config-push), never run in the request-pool worker.
- **delaymgr** only fires timers and triggers work; the peer connect is **bounded** (non-blocking + `poll` timeout) and **offloaded** off the timer thread.
- Heartbeat liveness is provably protected: a partitioned (unreachable, not merely dead) peer cannot stall heartbeats to healthy peers or trigger a spurious election.

## Approach

Bring the code into compliance with the steering thread-role contract, structurally rather than per-op:

1. **Dispatch `peer_ctrl_msg_handler` to the vtask pool** (the structural fix). iomgr becomes epoll + enqueue only. Preserve per-connection message ordering (VTASK_IO per-endpoint serialization is a feature here). Marshal any iomgr-owned state mutations (epoll/endpoint lifecycle) back to the iomgr thread (bug-646 discipline).
2. **Two-tier execution inside the vtask:** fast bounded protocol work inline; **trigger** long I/O (snapshot send + `persist_checkpoint`, config-push) to a **dedicated slow-lane / admission-capped executor** so multi-second tasks cannot starve the request pool. Per-peer dedup (one snapshot in flight per standby); coordinate with ongoing WAL-catchup stream and the primary's appends/WAL-trim.
3. **delaymgr connect → bounded + offloaded:** reuse the `xce_probe_config_tag` non-blocking-connect + bounded-`poll` pattern; trigger the reconnect as an idempotent, refcounted, deduped task off the delaymgr thread; marshal the connected-fd epoll registration back to iomgr. DNS (`gethostbyname`) is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — defer async DNS unless measured.
4. **General (non-IO) task-submit path:** add the submit primitive needed for #2/#3 (current `vtask_handle_msg` requires an inbound packet); or generalize it.
5. **Admission discipline:** any offloaded blocking task lives under a reservation (extend the `COMMIT_WAIT_RESERVED` pattern) so liveness-critical messages always have a free worker.

## Scope

- **In**:
  - Dispatch `peer_ctrl_msg_handler` real work to the vtask pool (root-cause structural fix).
  - Snapshot send + `persist_checkpoint` off iomgr → slow lane, with per-peer dedup + WAL coordination.
  - Config-push wait off iomgr → slow lane / async ACK.
  - delaymgr peer connect → bounded non-blocking + offloaded; reconnect dedup/refcount/marshal-back.
  - General non-IO vtask submit path + admission reservation for offloaded blocking tasks.
  - Partition (unreachable-peer) regression test; verify no spurious election + heartbeats to survivors continue.
- **Out**:
  - The commit-barrier async conversion (P1 / `repl_wait_majority_ack` → `resolve`) — owned by `raft-apply-decoupling` as its keystone (different concern: that's the *commit* drive, this is the *transport/timer* drive).
  - Apply-after-commit / hook decoupling / facts-through-raft (raft-apply-decoupling).
  - Async DNS (deferred unless measured).
  - The MEDIUM vtask-pool blockers beyond what the admission discipline naturally covers (note them; fix opportunistically).
  - Entity-bearing snapshots / config-upgrade snapshot (separate specs).

## Boundary Candidates

- The `peer_ctrl_msg_handler` → vtask dispatch seam.
- The dedicated slow-lane executor + admission reservation.
- The general non-IO vtask-submit primitive.
- The bounded-connect + offloaded-reconnect path.
- The iomgr-owned-state marshal-back mechanism (epoll/endpoint lifecycle).

## Out of Boundary

- Commit-engine internals (P1 owns the commit-drive async).
- Snapshot *content* format (entity-bearing snapshots is a separate spec) — this spec only moves the *transfer* off the drive thread.

## Upstream / Downstream

- **Upstream**: the `ha-async-io.md` thread-role contract (authority); existing `COMMIT_WAIT_RESERVED` (commit.c) and `xce_probe_config_tag` bounded-connect pattern (reuse); `election-iomgr-refactor` (COMPLETED upstream — provides reusable primitives `delay_task_ms`, `on_connected`, `ha_start_election_async`/non-blocking vote fan-out).
- **Downstream**: `raft-apply-decoupling` (its async commit drive + apply pump reuse the slow-lane + admission infrastructure this spec establishes; and the drive must be non-blocking before apply is restructured on top of it); safe failover for the whole bug-851 family depends on this landing.

## Existing Spec Touchpoints

- **Adjacent (RESOLVED standalone 2026-06-25)**: `election-iomgr-refactor` is ADJACENT-SAME-AREA — it fixed the *startup* blocking path and is fully implemented; this spec fixes the *runtime* drive-thread paths it never touched. Keep separate; consume it as upstream. (Scope-check verdict in research.md.)
- **Coordinate**: `ha-term-safety` (touches the same `peer_ctrl_msg_handler` / repl dispatch region); `ha-wal-replication` (snapshot/catch-up transport); `controller-status` (heartbeat/committed propagation on the same drive).
- **Verification home**: `raft-sim-conformance` (add the partition/liveness assertion) and `/xce-test-ha` (partition variant).

## Constraints

- The corrected thread-role contract (`ha-async-io.md`) is authoritative — iomgr and delaymgr never block inline; long I/O on the slow lane; bounded blocking in vtask only under admission reservation.
- "Controllers never self-terminate" — an unreachable peer is a loud retry, never an abort.
- Preserve per-connection message ordering when dispatching `peer_ctrl_msg_handler` to vtask.
- Must not regress the heartbeat fast path measurably at cluster scale (3–5 peers); per-message vtask overhead accepted in exchange for uniformity + liveness safety.
- Verified by a **partition** (unreachable, not killed) soak — the case `/xce-test-ha`'s current kill-only soak does not reproduce.

## Provenance

Surfaced 2026-06-25 during the `raft-apply-decoupling` discovery as prerequisite **P2** (drive hardening); promoted to its own spec because it threatens liveness independently of the apply refactor. Full analysis: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § "Two distinct prerequisites" + P2 offload mechanism. Thread-role contract: `.kiro/steering/ha-async-io.md` (corrected 2026-06-25).