myxce/.kiro/specs/raft-drive-async/design.md

# Design Document — raft-drive-async

## Overview

**Purpose**: Make the XCE controller's Raft protocol **drive** fully event-driven by removing every blocking operation from the two threads that carry cluster liveness — the iomgr epoll loop and the delaymgr timer — so that a slow/large transfer or an unreachable peer can no longer stall heartbeats and trigger a spurious election.

**Users**: Operators of multi-node controller clusters, who today see false failover under partition or large-snapshot conditions (the likely root of the bug-851 chaotic-failover family).

**Impact**: Changes *how* drive-thread work is scheduled, not *what* the protocol does. `peer_ctrl_msg_handler`'s real work moves from inline-on-iomgr to the vtask pool; long blocking I/O (snapshot transfer, config-push, peer connect) moves to a new dedicated **slow-lane executor**; the delaymgr peer connect becomes bounded and offloaded. All existing consensus semantics, durability (persist-before-ACK), and the synchronous commit barrier are preserved unchanged.

### Goals
- No blocking operation executes inline on the iomgr loop or the delaymgr timer (R1, R2, R5).
- Long blocking I/O runs on a bounded, admission-controlled lane isolated from the request vtask pool (R3).
- Per-connection processing order and consensus correctness preserved when peer control moves to the pool (R4).
- A network-partition (unreachable-peer) regression test proves heartbeat liveness and no spurious election (R7).
- No regression of existing HA features; steady-state heartbeat path unchanged (R6, R8).

### Non-Goals
- Commit-barrier async (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` (P1).
- Apply-after-commit / hook decoupling / facts-through-raft.
- Asynchronous DNS resolution — deferred (HPC peers use IP/`/etc/hosts`); only `connect` is bounded here.
- Snapshot *content*/format (entity-bearing snapshots) — this spec moves the *transfer* off the drive threads only.
- The MEDIUM vtask-pool blockers (config-ack wait, 30 s checkpoint poll, term fsync on vote) beyond what the lane and existing `COMMIT_WAIT_RESERVED` already cover.
- The admin `OP_upgradeConfig` push path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459) — it has a synchronous requester and stays a vtask-blocking operation under `COMMIT_WAIT_RESERVED`; only the requester-less iomgr transport push is offloaded.

## Boundary Commitments

### This Spec Owns
- The scheduling discipline of drive-thread work: dispatching `peer_ctrl_msg_handler` to the vtask pool; the two-tier rule (fast protocol inline in the vtask, long I/O triggered to the slow lane).
- A new **slow-lane executor** component: a bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight deduplication.
- The bounded, offloaded peer-connect path (reorder of `socket_connect` + offloaded reconnect).
- The network-partition regression scenario in the soak harness.

### Out of Boundary
- The commit engine and `repl_wait_majority_ack` (untouched; P1 owns its async conversion).
- WAL/replication protocol content, snapshot format, config-upgrade materialization.
- Election/vote protocol logic (consumed as-is; `election-iomgr-refactor` already made vote fan-out async).
- The durability guarantee and synchronous commit barrier (preserved, not modified).

### Allowed Dependencies
- `ha-async-io.md` thread-role contract (authority).
- Existing primitives reused, not modified in contract: `endpoint_block_tasks`/`endpoint_unblock_tasks` (per-endpoint serialization), lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (thread-safe epoll registration), `vtask_handle_msg`/VTASK_IO dispatch, `delay_task_ms` (non-blocking trigger only), `COMMIT_WAIT_RESERVED` (admission pattern), the `xce_probe_config_tag` bounded-connect pattern, the `fsync_worker`/`push_thread` background-thread precedent.
- `election-iomgr-refactor` (completed upstream): `delay_task_ms`, `on_connected`, non-blocking vote fan-out.

### Revalidation Triggers
- Any change to the slow-lane submit contract or its admission cap.
- Any change that makes a drive thread (iomgr/delaymgr) execute work synchronously again.
- Changes to per-connection ordering semantics of peer-control dispatch (affects consensus correctness assumers: `ha-term-safety`, `raft-apply-decoupling`).
- A change to `socket_connect`'s blocking/timeout behavior (affects all peer connection callers).

## Architecture

### Existing Architecture Analysis

Three drive threads exist (`ha-async-io.md`): **iomgr** (single epoll loop), **vtaskmgr** (N-worker pool, default 8), **delaymgr** (single timer). The corrected thread-role contract (2026-06-25) requires iomgr and delaymgr to never block inline. Current violations:
- `peer_ctrl_msg_handler` is registered directly as `ep->msg_handler` (ha.c:4265/4307/6349) → runs inline on iomgr; it triggers snapshot send (ha.c:2702 → config_repl.c:2446) and config-push (config_repl.c:1332) synchronously on iomgr.
- `socket_connect` (netio.c:289/302) blocks (sets `O_NONBLOCK` only *after* connect, netio.c:716); reached from `primary_heartbeat_fn` on the delaymgr thread → an unreachable peer stalls all heartbeats.
- The codebase's only deferral idiom, `delay_task_ms`, runs the deferred callback *on the delaymgr thread* — a liveness thread — so it cannot host blocking work under the new contract.

### Architecture Pattern & Boundary Map

Selected pattern: **dispatch-and-offload** — iomgr dispatches, the vtask pool does bounded protocol work, a new dedicated slow lane absorbs long blocking I/O; completions re-enter iomgr only through the existing lock-guarded registration call.

```mermaid
graph TB
    subgraph drive[Drive threads - never block]
        iomgr[iomgr epoll loop]
        delaymgr[delaymgr timer]
    end
    subgraph pool[vtask pool - bounded work]
        vtask[peer_ctrl vtask entry]
    end
    subgraph lane[slow-lane executor - dedicated bounded threads]
        slow[blocking IO worker]
    end
    iomgr -->|dispatch VTASK_IO packet| vtask
    vtask -->|fast protocol inline| vtask
    vtask -->|trigger long IO| slow
    delaymgr -->|trigger reconnect| slow
    slow -->|blocking connect snapshot configpush| slow
    slow -->|register connected fd via lock-guarded iomgr_monitor| iomgr
```

**Architecture Integration**:
- Domain boundaries: iomgr = I/O + dispatch; vtask = bounded protocol handling (serialized per endpoint by `endpoint_block_tasks`); slow lane = long blocking I/O only; delaymgr = trigger-only.
- Existing patterns preserved: VTASK_IO per-endpoint serialization, lock-guarded epoll registration, `delay_task_ms` as a non-blocking trigger, `COMMIT_WAIT_RESERVED` admission style.
- New component rationale: the slow lane is the missing *non-liveness execution lane* — the one genuine gap; using dedicated threads (not the vtask pool) keeps long ops from starving request handling (R3 by construction).
- Dependency direction: `netio` (lib) → `slowlane` → `ha`/`config_repl` (controller). Drive threads depend on the lane; the lane depends only on netio + libc.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | C (gnu17), pthreads | slow-lane executor; drive-thread dispatch changes | matches controller stack |
| Messaging / Events | existing iomgr/vtaskmgr/delaymgr + new slow lane | thread-role-compliant scheduling | no new external dep |
| Infrastructure / Runtime | epoll, `poll(2)`, pthread mutex/cond/sem | bounded connect, lane workers, admission | reuse existing patterns |
| Test | soak harness + firewall DROP (iptables/nft) | partition (unreachable-peer) scenario | capability-gated; see Testing |

## File Structure Plan

### New Files
```
src/controller/
├── slowlane.c             # Bounded slow-lane executor: submit, worker loop, admission cap, per-key dedup
└── include/slowlane.h     # slow-lane public API (submit, init, fini)
```

### Modified Files
- `src/lib/vtaskmgr.c` / `src/include/vtaskmgr.h` — **LivenessAdmission**: add `vtask_class_t` + a classified submit (`vtask_enq_classified`); dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus, fast-rejecting `VTASK_REQ_BLOCK` at the cap. Shared lib primitive → Revalidation Trigger.
- `src/controller/commit.c` — fold `inflight_waits`/`COMMIT_WAIT_RESERVED` into the pool reservation: `commit_submit` submits a `VTASK_REQ_BLOCK` instead of self-counting.
- `src/lib/netio.c` — `socket_connect`: set `O_NONBLOCK` before `connect`, bound with `poll(timeout)` (reuse `xce_probe_config_tag` pattern). Add a connect-timeout parameter/config.
- `src/controller/ha.c` — register peer endpoints to dispatch peer-control through the vtask pool (revise `ha.c:4265/4307/6349`); convert `peer_ctrl_msg_handler` into a vtask entry that does fast protocol work inline and `slowlane_submit`s long ops; offload snapshot send (ha.c:2702) and reconnect (`primary_heartbeat_fn`/`ha_connect_to_standby`) to the slow lane with per-peer dedup.
- `src/controller/config_repl.c` — **only the iomgr/promotion path**: run `push_config_transport_only` on the slow lane and replace the `wait_all_with_timeout` drive-thread join (reached from `peer_ctrl`/promotion at ha.c:1172/1177/2672/2677/3973) with a slow-lane-internal wait + non-blocking return. **In scope = the fire-and-forget transport push with NO waiting requester.** The admin `OP_upgradeConfig` path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459, "VTASK-CONTEXT ONLY") is **OUT of scope** — it has a synchronous admin requester awaiting the version/ACK reply and is a MEDIUM vtask-pool blocker bounded by `COMMIT_WAIT_RESERVED`; it must NOT be offloaded (doing so would break the synchronous admin reply).
- `src/controller/cluster.c` (or `myapp.c`) — init/fini the slow-lane executor in the controller lifecycle; pass connect-timeout config.
- `src/include/config.h` / `ha.h` — connect-timeout + slow-lane size/cap constants.
- `.claude/skills/xce-test-ha/run-ha-soak.sh` (+ `xce_test`) — partition (DROP) primitive and the partition scenario.

> Each file keeps one responsibility; `slowlane.c` is the only new logic module. peer-control logic stays in `ha.c` (moved to a vtask entry, not relocated to a new file) to preserve reviewability against existing tests.

## System Flows

### Reconnect to an unreachable peer (R2)
```mermaid
sequenceDiagram
    participant DM as delaymgr timer
    participant SL as slow lane
    participant IO as iomgr
    DM->>DM: heartbeat tick - peer disconnected
    DM->>SL: slowlane_submit reconnect peer if not in-flight
    DM-->>DM: return immediately - other peers still heartbeat
    SL->>SL: nonblocking connect + poll bounded timeout
    alt connect succeeds
        SL->>SL: prepare endpoint set msg_handler
        SL->>IO: iomgr_monitor lock-guarded register fd
    else timeout or refused
        SL->>SL: clear in-flight - retry next tick
    end
```
Key decisions: delaymgr only triggers (idempotent per-peer flag); the connect is bounded so it never approaches the heartbeat timeout; registration reuses the lock-guarded `iomgr_monitor` (no new primitive).

### Snapshot send triggered by a catch-up request (R1)
```mermaid
sequenceDiagram
    participant IO as iomgr
    participant VT as peer_ctrl vtask
    participant SL as slow lane
    IO->>VT: dispatch repl-ack packet VTASK_IO
    VT->>VT: fast - decode, update match_index
    VT->>SL: slowlane_submit snapshot if not in-flight per standby
    VT-->>IO: handler returns - endpoint unblocked, peer messages flow
    SL->>SL: persist_checkpoint + fread chunk loop + send
```
Key decisions: the vtask must **not** hold `ep->busy` across the transfer (R4.3) — it triggers and returns; per-standby dedup prevents concurrent transfers (R1.5).

## Requirements Traceability

| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1–1.4 | liveness during snapshot/config I/O | SlowLane, PeerCtrlDispatch | Snapshot flow |
| 1.5 | no concurrent snapshot per standby | SlowLane (dedup) | Snapshot flow |
| 2.1–2.2 | unreachable peer doesn't stall heartbeats | BoundedConnect, ReconnectOffload | Reconnect flow |
| 2.3 | bounded connect timeout | BoundedConnect | — |
| 2.4–2.5 | retry without self-terminate; dedup | ReconnectOffload, SlowLane | Reconnect flow |
| 2.6 | connect success registers without loss/reorder | ReconnectOffload (iomgr_monitor) | Reconnect flow |
| 3.1–3.4 | admission protection; long IO off pool/drive | SlowLane (cap), LivenessAdmission (pool reservation), PeerCtrlDispatch | — |
| 4.1–4.2 | per-connection processing order | PeerCtrlDispatch (endpoint_block_tasks) | — |
| 4.3 | don't hold block across long IO | PeerCtrlDispatch | Snapshot flow |
| 4.4 | iomgr-state mutation via lock-guarded iface | ReconnectOffload (iomgr_monitor) | Reconnect flow |
| 4.5–4.6 | preserve vote/ack/term + persist-before-vote | PeerCtrlDispatch, LivenessAdmission (ack-progress reservation) | — |
| 5.1–5.4 | thread-role contract compliance | all | — |
| 6.1–6.7 | no regression of existing HA features | all (regression suite) | — |
| 7.1–7.4 | partition verification | PartitionTest | — |
| 8.1–8.2 | heartbeat fast-path unchanged | PeerCtrlDispatch | — |

## Components and Interfaces

| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| SlowLane | controller/infra | bounded executor for long blocking I/O | 1, 3 | pthreads, sem (P0) | Service, Batch |
| LivenessAdmission | lib/vtaskmgr | pool-level task-class reservation: liveness work always has a worker | 3, 4.5 | vtaskmgr dispatcher (P0) | Service |
| PeerCtrlDispatch | controller/ha | move peer-control to vtask; two-tier | 1,4,5,8 | vtaskmgr, LivenessAdmission, endpoint_block_tasks (P0) | Service |
| BoundedConnect | lib/netio | non-blocking + poll-bounded connect | 2 | poll (P0) | Service |
| ReconnectOffload | controller/ha | delaymgr triggers offloaded reconnect | 2 | SlowLane, iomgr_monitor (P0) | Service |
| PartitionTest | test | unreachable-peer regression | 7 | soak harness (P1) | Batch |

### Controller / Infrastructure

#### SlowLane

| Field | Detail |
|-------|--------|
| Intent | Bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight dedup |
| Requirements | 1.1, 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4 |

**Responsibilities & Constraints**
- Owns a fixed set of dedicated worker threads (sized to peer count; small constant) — isolated from the vtask request pool, so long ops cannot starve request handling.
- `slowlane_submit` is non-blocking (enqueue + return); callers on drive threads never wait.
- Enforces an admission cap on concurrent in-flight tasks; excess is queued, never run on a drive thread (R3.3/3.4).
- Per-key dedup: a submit with an in-flight key is a no-op (R1.5 snapshot-per-standby; R2.5 reconnect-per-peer).
- Does not own the work content (connect, snapshot, config-push bodies stay in their modules); it owns scheduling/isolation.

**Dependencies**
- Inbound: PeerCtrlDispatch — submits long ops (P0); ReconnectOffload — submits reconnect (P0).
- Outbound: `iomgr_monitor` (via the submitted task) — register connected fd (P0).
- External: pthreads (threads, mutex, cond, sem) (P0).

**Contracts**: Service [x] / Batch [x]

##### Service Interface
```c
typedef void (*slowlane_fn_t)(void *arg);

int  slowlane_init(slowlane_t *sl, int n_workers, int max_inflight);
/* submit returns 0 (enqueued), -EEXIST (key in flight, deduped), -EAGAIN (cap full, queued/deferred);
   never blocks the caller. key==0 disables dedup. */
int  slowlane_submit(slowlane_t *sl, uint64_t key, slowlane_fn_t fn, void *arg);
void slowlane_fini(slowlane_t *sl);   /* drain + join on shutdown only */
```
- Preconditions: `fn` performs the blocking work and is responsible for clearing any caller-side in-flight state on completion.
- Postconditions: `fn` runs exactly once on a lane worker; never on iomgr/delaymgr.
- Invariants: concurrent running tasks ≤ `max_inflight`; one in-flight task per non-zero key.

**Snapshot/checkpoint concurrency invariant (Phase 3, R1.1/1.3/1.5 — MUST hold before snapshot offload lands)**
Moving `persist_checkpoint` (which runs `trim_wal_segments` — *deletes* WAL segments) + the snapshot `fread` loop onto a lane worker runs them concurrently with live WAL appends, the `fsync_worker` flusher, and the WAL-catchup stream. The lane snapshot path must obey:
- **Frozen boundary**: the snapshot is taken at a fixed `(term, lsn)` checkpoint boundary captured at submit time; the `fread`/send loop reads only data at or below that boundary, never live tail bytes.
- **Trim safety (MECHANISM, not just goal — see bug-1147)**: `trim_wal_segments` (`persist.c:1374`) TODAY bounds deletion ONLY by `active_seg_lsn` (the open segment) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** snapshot boundary, and there is **no** working snapshot backfill (`write_snapshot` emits only a 16-byte P0 sentinel, not materialized state). So a standby lagging by more than ~one open segment already hits a permanent committed-prefix gap (the bug-795/828 family). Moving checkpoint/trim onto a concurrent lane worker makes this latent race *easier* to hit. Therefore this spec MUST add the cursor-bound guard as a stopgap: `trim_wal_segments` computes `floor = min(all standby acked_lsn, all in-flight snapshot boundaries)` under `p->lock` and refuses to delete any segment whose range reaches `floor` or above — never holding `p->lock` across the `fread`/send I/O. This is strategy-A retention discipline that *should* exist regardless of this spec; the full fix for empty/far-behind join (cursor=0) requires entity-bearing snapshots (separate spec) and is explicitly NOT solved here. The design must not assume "a snapshot will backfill the gap" — that path is not built.
- **Single in-flight checkpoint**: at most one checkpoint/trim in flight (lane dedup key = checkpoint), and at most one snapshot per standby (dedup key = standby id, R1.5).
- **No correctness change**: the lane only relocates execution; the checkpoint/trim/snapshot *content* is byte-identical to the current inline path.

**Implementation Notes**
- Integration: init in controller lifecycle (cluster.c/myapp.c); size from peer count.
- Validation: assert submit never called from a context expecting completion; metric on queue depth/age (fail-loud on backlog); a test asserting concurrent append + checkpoint/trim produces no segment-below-boundary deletion and a consistent snapshot.
- Risks: a lane worker blocked indefinitely (e.g., snapshot to a half-open peer) — **bound every in-lane blocking op**: the connect uses the bounded `socket_connect`; the snapshot send must use a bounded socket write (write timeout, mirroring the bounded-connect approach) so a standby going unreachable mid-transfer cannot pin a lane worker. Otherwise stuck transfers exhaust the bounded lane and delay reconnects (which share it). Phase-3 task constraint; size/segment the lane so reconnect and transfer do not starve each other.

#### LivenessAdmission (lib/vtaskmgr)

| Field | Detail |
|-------|--------|
| Intent | Make "keep K workers free for liveness" a property of the vtask **pool**, not of the commit engine — so it covers every blocking op the pool runs once peer-control moves onto it, not just commit-waits |
| Requirements | 3.1, 3.2, 3.3, 4.5 |

**Why this exists (the gap it closes):** Today `vtaskmgr` (`include/vtaskmgr.h`) has a single FIFO `taskq`, N `vcpu` workers, and **no priority/reservation**; a blocking task holds its `vcpu` in `VCPU_BUSY` for the whole block (`vtaskmgr.c:248-275`). The only admission anywhere is `commit.c`'s `inflight_waits` counter (`COMMIT_WAIT_RESERVED=2`, `commit.c:200`) — external to vtaskmgr and counting **only** commit-waits. Once `peer_ctrl` moves onto the pool (PeerCtrlDispatch), the pool also runs the vote-grant term fsync (R4.6) and config-ack waits — blocking ops **invisible** to the commit counter. So "≥2 workers free of commit-waits" no longer means "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. The reservation must therefore become pool-level.

**Responsibilities & Constraints**
- Classify every vtask at enqueue into `VTASK_LIVENESS` (never parks — WAL-ack decode, heartbeat send, vote *receipt/decode*; `commit_on_ack` is documented never-block, `commit.c:16`), `VTASK_LIVE_BLOCK` (bounded mandatory park — vote-grant term fsync, R4.6), or `VTASK_REQ_BLOCK` (may park seconds — commit-wait, config-ack wait). Class is knowable from the message type at the dispatch shim.
- The dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus: a `VTASK_REQ_BLOCK` task is **not admitted** when `count(vcpus running *_BLOCK) >= n_vcpus - VTASK_LIVENESS_RESERVED` — it **fast-rejects with `-XCE_ERR_NOT_READY`** (preserving today's client backpressure, Req 7.1/7.2), it does **not** silently queue or park a worker.
- `VTASK_LIVENESS` tasks are **never** admission-gated → always find a reserved vcpu → acks always advance commit → parked `*_BLOCK` tasks always wake. **Deadlock closed by construction.**
- `VTASK_LIVE_BLOCK` (vote fsync) runs in the reserved headroom and is bounded, so a vote response never queues behind a seconds-long commit-wait (removes a second spurious-election vector).
- `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` is **folded into** this — `commit_submit` becomes "submit a `VTASK_REQ_BLOCK`"; the pool owns the single reservation (`commit_engine_init` no longer needs `n_vcpus`, `cluster.c:1332`).

**Dependencies**: Inbound — PeerCtrlDispatch, commit (P0). Outbound — none (lib-internal). External — pthreads.

**Contracts**: Service [x]
```c
typedef enum { VTASK_LIVENESS=0, VTASK_LIVE_BLOCK, VTASK_REQ_BLOCK } vtask_class_t;
/* submit a classified task; VTASK_REQ_BLOCK returns -XCE_ERR_NOT_READY when the
   reservation would be violated (retriable, never parks a worker). */
int vtask_enq_classified(vtaskmgr_t*, vtask_t*, vtask_class_t);
```
- Invariant: `count(vcpus in *_BLOCK) <= n_vcpus - VTASK_LIVENESS_RESERVED`; ≥`VTASK_LIVENESS_RESERVED` vcpus always available to `VTASK_LIVENESS`.

**Implementation Notes**
- **Sizing (peer-scaled):** `R = VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)`; commit/`REQ_BLOCK` cap = `n_vcpus − R`. Rationale: per-endpoint serialization (`endpoint_block_tasks`) caps in-flight tasks at one per peer connection, so worst-case concurrent liveness work is **N−1** (one ack/vote per other peer) — reserving N−1 means no peer's liveness message ever queues behind a busy reserved worker. **Correctness needs only the floor of 1** (one guaranteed-free worker + liveness-never-gated + sub-ms cycling = deadlock-free regardless of peer count); the peer-scaled middle is a *latency* optimization. **Ceiling `n_vcpus − 1`** keeps ≥1 `REQ_BLOCK` slot so commits can still run (cap never 0). Both `n_peers` and `n_vcpus` are final at init (`cluster.c:1332` already passes both). Config-overridable. On a degenerate small-core box (`n_vcpus < n_peers`) R clamps to `n_vcpus − 1` (full per-peer reservation impossible) — still correct via fast cycling; emit a loud WARN at init so the under-provisioning is operator-visible. Validate the chosen `R` at supported sizes (R8).
- Validation: misclassification is the one failure mode — add a debug assert/loud-WARN if a `VTASK_LIVENESS` task parks beyond a threshold; task 3.4 (acks advance commit under saturation) is the regression oracle.
- Risk: a future blocking path tagged `VTASK_LIVENESS` silently breaks the guarantee — review discipline + the assert + the saturation test.
- **Revalidation Trigger**: any change to the class set, the reservation size, or the dispatch admission rule (consensus correctness + the slow-lane admission contract depend on it).

#### PeerCtrlDispatch

| Field | Detail |
|-------|--------|
| Intent | Dispatch peer-control message handling to the vtask pool; fast work inline, long work to the lane |
| Requirements | 1.4, 4.1, 4.2, 4.3, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2 |

**Responsibilities & Constraints**
- Peer endpoints dispatch through the existing VTASK_IO path (the message is packet-triggered → reuses `vtask_handle_msg`), inheriting `endpoint_block_tasks` per-endpoint serialization (R4.1/4.2).
- The vtask entry performs only bounded protocol work inline (decode, match_index update, commit eval, send next heartbeat) and `slowlane_submit`s any long op; it must not hold `ep->busy` across long I/O (R4.3).
- Preserves existing vote/ack/term semantics and persist-before-vote ordering (R4.5/4.6) — the logic is unchanged, only its execution context moves.
- **Commit-advance preservation (R4.5):** WAL-ack handling — which wakes the commit barrier (`repl_wait_majority_ack` / `commit_set_raft_commit_index`) — moves off iomgr to the vtask pool. The design preserves that a quorum of acks still advances `committed_lsn` promptly: ack handling is bounded fast work (never offloaded to the slow lane), per-endpoint serialization does not delay *cross-endpoint* acks (each peer's acks process independently), and the commit-advance path itself is untouched. This is the only interaction with P1 (`raft-apply-decoupling`, which owns the barrier's async conversion); this spec must not change ack→commit-advance semantics, only the thread the ack is decoded on.

- **Deadlock-freedom argument (R4.5/R3.1 — REQUIRED reasoning, reverses a documented invariant):** Today `commit_on_ack` is explicitly "called from the iomgr read path and never blocks" (`commit.c:16`, `repl.c:220`) — the ack that advances `committed_lsn` and the jobsub vtasks blocked in `commit_submit(COMMIT_MODE_WAIT)` (the synchronous barrier this spec PRESERVES) run on **different** threads. Moving peer-control to the vtask pool **co-locates the wakers (acks) and the waiters (blocked commit-submits) on one pool** — exactly the configuration the iomgr placement was chosen to avoid. Safety rests on a worker reservation — but the existing `commit.c` cap is **insufficient as-is**: `COMMIT_WAIT_RESERVED` (`commit.c:200`) counts **only** commit-waits, while moving `peer_ctrl` onto the pool also puts the vote-grant term fsync (R4.6, `VTASK_LIVE_BLOCK`) and config-ack waits on it — blocking ops the commit counter can't see. "≥2 workers free of commit-waits" therefore does **not** mean "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. **Closing the deadlock requires the LivenessAdmission component** (above) — the reservation must become a pool-level property covering every blocking class, with `VTASK_LIVENESS` (acks) never admission-gated. Required invariants this spec must hold (and test): (1) WAL-ack/commit-advance decode is tagged `VTASK_LIVENESS`, never routed to the slow lane, never gated — admissible independently of any blocking count; (2) the reserved headroom is sized for all concurrent liveness work (N-peer acks + vote responses + heartbeat sends), so the reservation — not luck — guarantees an ack always finds a worker. This co-location is **transient by design**: `raft-apply-decoupling` P1 converts `repl_wait_majority_ack` to event-driven (no parked thread per commit), removing the `VTASK_REQ_BLOCK` waiters from the pool entirely — but until P1 lands, correctness depends on LivenessAdmission holding. **Dependency: PeerCtrlDispatch (3.1) requires LivenessAdmission to land first (Phase 1).** Gate: task 3.4 (quorum-of-acks advances `committed_lsn` with the pool saturated by `*_BLOCK` tasks) is the **admission gate for 3.1** — 3.1 is not "done" until 3.4 is green.

**Dependencies**
- Inbound: iomgr — dispatches the packet (P0).
- Outbound: vtaskmgr (P0); SlowLane — long ops (P0); existing commit/repl/persist logic (P0).

**Contracts**: Service [x]

**Implementation Notes**
- Integration: change peer endpoint `msg_handler` registration (ha.c:4265/4307/6349) to the vtask-dispatching shim; convert `peer_ctrl_msg_handler` body into the vtask entry.
- Validation: existing election/heartbeat/repl tests must pass unchanged (R6); assert no long op runs inline in the entry.
- Risks: a heavy op accidentally left inline → per-peer head-of-line blocking (guard by review + a test asserting the handler returns promptly).

#### BoundedConnect (lib/netio)

| Field | Detail |
|-------|--------|
| Intent | Make `socket_connect` non-blocking + `poll`-bounded | 
| Requirements | 2.3 |

**Responsibilities & Constraints**: set `O_NONBLOCK` before `connect`; on `EINPROGRESS`, `poll` up to a configured timeout; return success/timeout/error. Reuses the `xce_probe_config_tag` pattern.
**Contracts**: Service [x]
```c
/* connect_timeout_ms: bound the attempt; 0 = use default (config). */
int socket_connect(const char *host, int port, int connect_timeout_ms);
```
**Implementation Notes**: callers updated to pass the timeout; default in config.h. This is a behavior change to a shared helper → Revalidation Trigger for all connect callers.

#### ReconnectOffload (controller/ha)

| Field | Detail |
|-------|--------|
| Intent | delaymgr triggers an offloaded, deduped, bounded reconnect | 
| Requirements | 2.1, 2.2, 2.4, 2.5, 2.6 |

**Responsibilities & Constraints**: `primary_heartbeat_fn` (delaymgr) detects a disconnected peer and `slowlane_submit`s the reconnect (per-peer key for dedup); the lane worker does the bounded connect, prepares the endpoint, registers it via lock-guarded `iomgr_monitor`; on failure clears the in-flight flag and lets the next tick retry (never self-terminates).
**Contracts**: Service [x]
**Implementation Notes**: refcount the endpoint across the in-flight window; resolve the inbound-vs-outbound connection race per the existing connection-ownership rules (`ha-async-io.md`).

### Test

#### PartitionTest

| Field | Detail |
|-------|--------|
| Intent | Make one peer unreachable (blackhole), assert liveness + no spurious election + heal | 
| Requirements | 7.1, 7.2, 7.3, 7.4 |

**Contracts**: Batch [x]
**Implementation Notes**: the partition primitive must render one peer **unreachable** (traffic dropped, connect hangs), distinct from `kill -9`. Two mechanisms, **primary = in-process** so R7 actually runs in this project's environment (the sibling specs `ha-term-safety` 5.2/5.4 are already perpetually ENV-BLOCKED on the unprivileged/overlayfs sandbox; a firewall-only R7 would be a permanent skip and the spec's headline guarantee would never be exercised):
- **Primary (portable, no privilege): in-process socket-drop hook in `xce_test`** — a test-only seam (`XCE_TESTING`) that makes one node silently drop/blackhole inbound+outbound peer traffic at the endpoint layer (e.g. stop reading/writing the peer fd, or a drop flag in the I/O path), simulating "unreachable" without any kernel/firewall change. Runs anywhere the unit/soak suite runs.
- **Secondary (infra-level, optional): firewall `iptables`/`nft` DROP** on the node's port for real-network fidelity; capability-gated with a **loud skip** when unprivileged. Complements, does not replace, the in-process path.

Assert (both): heartbeats to survivors continue within the interval, no new-leader election while the leader holds majority, peer rejoins on heal, zero committed-job loss.

## Error Handling

### Error Strategy
- **Connect timeout/refused** → bounded `poll` returns; lane clears in-flight; delaymgr retries next tick; loud WARN with peer + elapsed (fail-loud, never abort — controllers never self-terminate).
- **Lane backlog** → submit returns `-EAGAIN` when the cap is reached; the work is queued, never run on a drive thread; emit a metric/WARN on queue depth + oldest-age so a saturated lane is operator-visible.
- **Snapshot/config-push failure on the lane** → reported through the existing repl/config error paths unchanged; lane only relocates execution, not error semantics.
- **Dedup race** → submit with an in-flight key is a no-op (`-EEXIST`); idempotent.

### Monitoring
- Metrics: lane queue depth, oldest-task age, in-flight count, per-peer reconnect attempts. WARN thresholds make a degraded/partitioned window visible (consistent with fail-loud).

## Testing Strategy

### Unit Tests
- SlowLane: `slowlane_submit` is non-blocking; admission cap bounds concurrent tasks; per-key dedup returns `-EEXIST`; `fn` runs exactly once (3.1–3.4, 1.5).
- BoundedConnect: `socket_connect` returns within the timeout against an unreachable host; succeeds against a reachable one (2.3).
- PeerCtrlDispatch: the vtask entry returns promptly (no long op inline); per-endpoint serialization holds via `endpoint_block_tasks` (4.1–4.3).

### Integration Tests
- Reconnect offload: with one peer unreachable, delaymgr ticks continue and heartbeats to reachable peers are sent within the interval; reconnect is deduped per peer (2.1, 2.2, 2.5).
- Snapshot offload: a catch-up triggering a snapshot does not block iomgr; other peers' heartbeats/votes processed during the transfer; no concurrent transfer per standby (1.1, 1.3, 1.4, 1.5).
- Commit-advance under pool load (R4.5): with the vtask pool near saturation, a quorum of WAL-acks still advances `committed_lsn` promptly; cross-endpoint acks are not serialized behind one another.
- Snapshot/checkpoint concurrency (R1.1/1.3/1.5): concurrent live appends + a lane checkpoint/trim never delete a segment below an in-flight snapshot boundary or a standby catch-up cursor; the produced snapshot is consistent.
- Regression sweep (R6): run the existing suites unchanged — `test_ha_election`, `test_ha_prevote`, `test_ha_liveness`, `test_ha_failover`, `test_ha_rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability`; all pass without behavior changes (6.1–6.7).

### E2E / Soak Tests
- Partition soak (R7): blackhole one peer → assert heartbeats to survivors continue, no spurious election while leader holds majority, peer rejoins on heal, zero committed-job loss (7.1–7.4).
- Failover soak (existing, kill-based) continues to pass (6.3).

### Performance
- Steady-state heartbeat cadence at 3–5 controllers shows no added latency attributable to the new dispatch path; the heartbeat/vote send paths remain non-blocking enqueues (8.1, 8.2).

## Migration Strategy

Phased, each phase independently testable (matches research recommendation):

**Ordering rule:** the slow lane is foundational — every offload (reconnect, snapshot, config-push) submits to it, so it must exist first. And the peer_ctrl→vtask move must land **together with** its long-op offload in the same phase: moving peer_ctrl to a vtask while still running snapshot/config-push inline would hold `endpoint_block_tasks` (`ep->busy`) across multi-second I/O (transient R4.3 violation) and consume a pool worker. The two are therefore coupled.

```mermaid
flowchart TD
    P1[Phase 1: SlowLane executor + BoundedConnect] --> P2[Phase 2: ReconnectOffload sender-side]
    P2 --> P3[Phase 3: PeerCtrlDispatch to vtask PLUS snapshot and configpush offload coupled]
    P3 --> P4[Phase 4: PartitionTest + regression sweep]
```

- **Phase 1** (foundational, lowest risk): build the SlowLane executor (bounded pool + admission + dedup — new component, no behavior change yet, unit-testable in isolation); reorder `socket_connect` to bounded non-blocking + `poll` (independent; already bounds the delaymgr stall even before reconnect is offloaded); **and add LivenessAdmission to vtaskmgr** (task-class reservation; fold in `COMMIT_WAIT_RESERVED`). LivenessAdmission is a **hard prerequisite of Phase 3** — peer_ctrl cannot move onto the pool deadlock-safely without it — so it must land in Phase 1 as dormant infra (no behavior change until peer_ctrl dispatches through it).
- **Phase 2**: offload the delaymgr peer reconnect to the lane (deduped per peer, registered via lock-guarded `iomgr_monitor`) → closes sender-side heartbeat starvation. Shippable.
- **Phase 3** (coupled — the R4.3-safe step): dispatch peer-control to the vtask pool **and** offload the long ops it triggers (snapshot send, config-push transport-only) to the lane in the same change, so no phase ever runs long I/O inline in a blocked vtask. Carries the snapshot/checkpoint concurrency invariant (frozen `(term,lsn)` boundary, trim-vs-catch-up-cursor safety, single in-flight checkpoint) and the bounded in-lane snapshot write.
- **Phase 4**: partition test + full regression sweep.
- Rollback: each phase is behavior-preserving; a phase can be reverted independently since the contract (non-blocking drive) is the only invariant added. P1 adds dormant infra; the first observable change is P2.