myxce/.kiro/specs/raft-drive-async/design.md

Design Document — raft-drive-async

Overview

Purpose: Make the XCE controller's Raft protocol drive fully event-driven by removing every blocking operation from the two threads that carry cluster liveness — the iomgr epoll loop and the delaymgr timer — so that a slow/large transfer or an unreachable peer can no longer stall heartbeats and trigger a spurious election.

Users: Operators of multi-node controller clusters, who today see false failover under partition or large-snapshot conditions (the likely root of the bug-851 chaotic-failover family).

Impact: Changes how drive-thread work is scheduled, not what the protocol does. peer_ctrl_msg_handler's real work moves from inline-on-iomgr to the vtask pool; long blocking I/O (snapshot transfer, config-push, peer connect) moves to a new dedicated slow-lane executor; the delaymgr peer connect becomes bounded and offloaded. All existing consensus semantics, durability (persist-before-ACK), and the synchronous commit barrier are preserved unchanged.

Goals

• No blocking operation executes inline on the iomgr loop or the delaymgr timer (R1, R2, R5).
• Long blocking I/O runs on a bounded, admission-controlled lane isolated from the request vtask pool (R3).
• Per-connection processing order and consensus correctness preserved when peer control moves to the pool (R4).
• A network-partition (unreachable-peer) regression test proves heartbeat liveness and no spurious election (R7).
• No regression of existing HA features; steady-state heartbeat path unchanged (R6, R8).

Non-Goals

• Commit-barrier async (repl_wait_majority_ack → event-driven resolve) — owned by raft-apply-decoupling (P1).
• Apply-after-commit / hook decoupling / facts-through-raft.
• Asynchronous DNS resolution — deferred (HPC peers use IP//etc/hosts); only connect is bounded here.
• Snapshot content/format (entity-bearing snapshots) — this spec moves the transfer off the drive threads only.
• The MEDIUM vtask-pool blockers (config-ack wait, 30 s checkpoint poll, term fsync on vote) beyond what the lane and existing COMMIT_WAIT_RESERVED already cover.
• The admin OP_upgradeConfig push path (push_config_to_standbys → repl_wait_majority_ack, config_repl.c:1459) — it has a synchronous requester and stays a vtask-blocking operation under COMMIT_WAIT_RESERVED; only the requester-less iomgr transport push is offloaded.

Boundary Commitments

This Spec Owns

• The scheduling discipline of drive-thread work: dispatching peer_ctrl_msg_handler to the vtask pool; the two-tier rule (fast protocol inline in the vtask, long I/O triggered to the slow lane).
• A new slow-lane executor component: a bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight deduplication.
• The bounded, offloaded peer-connect path (reorder of socket_connect + offloaded reconnect).
• The network-partition regression scenario in the soak harness.

Out of Boundary

• The commit engine and repl_wait_majority_ack (untouched; P1 owns its async conversion).
• WAL/replication protocol content, snapshot format, config-upgrade materialization.
• Election/vote protocol logic (consumed as-is; election-iomgr-refactor already made vote fan-out async).
• The durability guarantee and synchronous commit barrier (preserved, not modified).

Allowed Dependencies

• ha-async-io.md thread-role contract (authority).
• Existing primitives reused, not modified in contract: endpoint_block_tasks/endpoint_unblock_tasks (per-endpoint serialization), lock-guarded iomgr_monitor/iomgr_unmonitor (thread-safe epoll registration), vtask_handle_msg/VTASK_IO dispatch, delay_task_ms (non-blocking trigger only), COMMIT_WAIT_RESERVED (admission pattern), the xce_probe_config_tag bounded-connect pattern, the fsync_worker/push_thread background-thread precedent.
• election-iomgr-refactor (completed upstream): delay_task_ms, on_connected, non-blocking vote fan-out.

Revalidation Triggers

• Any change to the slow-lane submit contract or its admission cap.
• Any change that makes a drive thread (iomgr/delaymgr) execute work synchronously again.
• Changes to per-connection ordering semantics of peer-control dispatch (affects consensus correctness assumers: ha-term-safety, raft-apply-decoupling).
• A change to socket_connect's blocking/timeout behavior (affects all peer connection callers).

Architecture

Existing Architecture Analysis

Three drive threads exist (ha-async-io.md): iomgr (single epoll loop), vtaskmgr (N-worker pool, default 8), delaymgr (single timer). The corrected thread-role contract (2026-06-25) requires iomgr and delaymgr to never block inline. Current violations:

• peer_ctrl_msg_handler is registered directly as ep->msg_handler (ha.c:4265/4307/6349) → runs inline on iomgr; it triggers snapshot send (ha.c:2702 → config_repl.c:2446) and config-push (config_repl.c:1332) synchronously on iomgr.
• socket_connect (netio.c:289/302) blocks (sets O_NONBLOCK only after connect, netio.c:716); reached from primary_heartbeat_fn on the delaymgr thread → an unreachable peer stalls all heartbeats.
• The codebase's only deferral idiom, delay_task_ms, runs the deferred callback on the delaymgr thread — a liveness thread — so it cannot host blocking work under the new contract.

Architecture Pattern & Boundary Map

Selected pattern: dispatch-and-offload — iomgr dispatches, the vtask pool does bounded protocol work, a new dedicated slow lane absorbs long blocking I/O; completions re-enter iomgr only through the existing lock-guarded registration call.

graph TB
    subgraph drive[Drive threads - never block]
        iomgr[iomgr epoll loop]
        delaymgr[delaymgr timer]
    end
    subgraph pool[vtask pool - bounded work]
        vtask[peer_ctrl vtask entry]
    end
    subgraph lane[slow-lane executor - dedicated bounded threads]
        slow[blocking IO worker]
    end
    iomgr -->|dispatch VTASK_IO packet| vtask
    vtask -->|fast protocol inline| vtask
    vtask -->|trigger long IO| slow
    delaymgr -->|trigger reconnect| slow
    slow -->|blocking connect snapshot configpush| slow
    slow -->|register connected fd via lock-guarded iomgr_monitor| iomgr

Architecture Integration:

• Domain boundaries: iomgr = I/O + dispatch; vtask = bounded protocol handling (serialized per endpoint by endpoint_block_tasks); slow lane = long blocking I/O only; delaymgr = trigger-only.
• Existing patterns preserved: VTASK_IO per-endpoint serialization, lock-guarded epoll registration, delay_task_ms as a non-blocking trigger, COMMIT_WAIT_RESERVED admission style.
• New component rationale: the slow lane is the missing non-liveness execution lane — the one genuine gap; using dedicated threads (not the vtask pool) keeps long ops from starving request handling (R3 by construction).
• Dependency direction: netio (lib) → slowlane → ha/config_repl (controller). Drive threads depend on the lane; the lane depends only on netio + libc.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Backend / Services	C (gnu17), pthreads	slow-lane executor; drive-thread dispatch changes	matches controller stack
Messaging / Events	existing iomgr/vtaskmgr/delaymgr + new slow lane	thread-role-compliant scheduling	no new external dep
Infrastructure / Runtime	epoll, poll(2), pthread mutex/cond/sem	bounded connect, lane workers, admission	reuse existing patterns
Test	soak harness + firewall DROP (iptables/nft)	partition (unreachable-peer) scenario	capability-gated; see Testing

File Structure Plan

New Files

src/controller/
├── slowlane.c             # Bounded slow-lane executor: submit, worker loop, admission cap, per-key dedup
└── include/slowlane.h     # slow-lane public API (submit, init, fini)

Modified Files

• src/lib/vtaskmgr.c / src/include/vtaskmgr.h — LivenessAdmission: add vtask_class_t + a classified submit (vtask_enq_classified); dispatcher reserves VTASK_LIVENESS_RESERVED vcpus, fast-rejecting VTASK_REQ_BLOCK at the cap. Shared lib primitive → Revalidation Trigger.
• src/controller/commit.c — fold inflight_waits/COMMIT_WAIT_RESERVED into the pool reservation: commit_submit submits a VTASK_REQ_BLOCK instead of self-counting.
• src/lib/netio.c — socket_connect: set O_NONBLOCK before connect, bound with poll(timeout) (reuse xce_probe_config_tag pattern). Add a connect-timeout parameter/config.
• src/controller/ha.c — register peer endpoints to dispatch peer-control through the vtask pool (revise ha.c:4265/4307/6349); convert peer_ctrl_msg_handler into a vtask entry that does fast protocol work inline and slowlane_submits long ops; offload snapshot send (ha.c:2702) and reconnect (primary_heartbeat_fn/ha_connect_to_standby) to the slow lane with per-peer dedup.
• src/controller/config_repl.c — only the iomgr/promotion path: run push_config_transport_only on the slow lane and replace the wait_all_with_timeout drive-thread join (reached from peer_ctrl/promotion at ha.c:1172/1177/2672/2677/3973) with a slow-lane-internal wait + non-blocking return. In scope = the fire-and-forget transport push with NO waiting requester. The admin OP_upgradeConfig path (push_config_to_standbys → repl_wait_majority_ack, config_repl.c:1459, "VTASK-CONTEXT ONLY") is OUT of scope — it has a synchronous admin requester awaiting the version/ACK reply and is a MEDIUM vtask-pool blocker bounded by COMMIT_WAIT_RESERVED; it must NOT be offloaded (doing so would break the synchronous admin reply).
• src/controller/cluster.c (or myapp.c) — init/fini the slow-lane executor in the controller lifecycle; pass connect-timeout config.
• src/include/config.h / ha.h — connect-timeout + slow-lane size/cap constants.
• .claude/skills/xce-test-ha/run-ha-soak.sh (+ xce_test) — partition (DROP) primitive and the partition scenario.

Each file keeps one responsibility; slowlane.c is the only new logic module. peer-control logic stays in ha.c (moved to a vtask entry, not relocated to a new file) to preserve reviewability against existing tests.

System Flows

Reconnect to an unreachable peer (R2)

sequenceDiagram
    participant DM as delaymgr timer
    participant SL as slow lane
    participant IO as iomgr
    DM->>DM: heartbeat tick - peer disconnected
    DM->>SL: slowlane_submit reconnect peer if not in-flight
    DM-->>DM: return immediately - other peers still heartbeat
    SL->>SL: nonblocking connect + poll bounded timeout
    alt connect succeeds
        SL->>SL: prepare endpoint set msg_handler
        SL->>IO: iomgr_monitor lock-guarded register fd
    else timeout or refused
        SL->>SL: clear in-flight - retry next tick
    end

Key decisions: delaymgr only triggers (idempotent per-peer flag); the connect is bounded so it never approaches the heartbeat timeout; registration reuses the lock-guarded iomgr_monitor (no new primitive).

Snapshot send triggered by a catch-up request (R1)

sequenceDiagram
    participant IO as iomgr
    participant VT as peer_ctrl vtask
    participant SL as slow lane
    IO->>VT: dispatch repl-ack packet VTASK_IO
    VT->>VT: fast - decode, update match_index
    VT->>SL: slowlane_submit snapshot if not in-flight per standby
    VT-->>IO: handler returns - endpoint unblocked, peer messages flow
    SL->>SL: persist_checkpoint + fread chunk loop + send

Key decisions: the vtask must not hold ep->busy across the transfer (R4.3) — it triggers and returns; per-standby dedup prevents concurrent transfers (R1.5).

Requirements Traceability

Requirement	Summary	Components	Flows
1.1–1.4	liveness during snapshot/config I/O	SlowLane, PeerCtrlDispatch	Snapshot flow
1.5	no concurrent snapshot per standby	SlowLane (dedup)	Snapshot flow
2.1–2.2	unreachable peer doesn't stall heartbeats	BoundedConnect, ReconnectOffload	Reconnect flow
2.3	bounded connect timeout	BoundedConnect	—
2.4–2.5	retry without self-terminate; dedup	ReconnectOffload, SlowLane	Reconnect flow
2.6	connect success registers without loss/reorder	ReconnectOffload (iomgr_monitor)	Reconnect flow
3.1–3.4	admission protection; long IO off pool/drive	SlowLane (cap), LivenessAdmission (pool reservation), PeerCtrlDispatch	—
4.1–4.2	per-connection processing order	PeerCtrlDispatch (endpoint_block_tasks)	—
4.3	don't hold block across long IO	PeerCtrlDispatch	Snapshot flow
4.4	iomgr-state mutation via lock-guarded iface	ReconnectOffload (iomgr_monitor)	Reconnect flow
4.5–4.6	preserve vote/ack/term + persist-before-vote	PeerCtrlDispatch, LivenessAdmission (ack-progress reservation)	—
5.1–5.4	thread-role contract compliance	all	—
6.1–6.7	no regression of existing HA features	all (regression suite)	—
7.1–7.4	partition verification	PartitionTest	—
8.1–8.2	heartbeat fast-path unchanged	PeerCtrlDispatch	—

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
SlowLane	controller/infra	bounded executor for long blocking I/O	1, 3	pthreads, sem (P0)	Service, Batch
LivenessAdmission	lib/vtaskmgr	pool-level task-class reservation: liveness work always has a worker	3, 4.5	vtaskmgr dispatcher (P0)	Service
PeerCtrlDispatch	controller/ha	move peer-control to vtask; two-tier	1,4,5,8	vtaskmgr, LivenessAdmission, endpoint_block_tasks (P0)	Service
BoundedConnect	lib/netio	non-blocking + poll-bounded connect	2	poll (P0)	Service
ReconnectOffload	controller/ha	delaymgr triggers offloaded reconnect	2	SlowLane, iomgr_monitor (P0)	Service
PartitionTest	test	unreachable-peer regression	7	soak harness (P1)	Batch

Controller / Infrastructure

SlowLane

Field	Detail
Intent	Bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight dedup
Requirements	1.1, 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4

Responsibilities & Constraints

• Owns a fixed set of dedicated worker threads (sized to peer count; small constant) — isolated from the vtask request pool, so long ops cannot starve request handling.
• slowlane_submit is non-blocking (enqueue + return); callers on drive threads never wait.
• Enforces an admission cap on concurrent in-flight tasks; excess is queued, never run on a drive thread (R3.3/3.4).
• Per-key dedup: a submit with an in-flight key is a no-op (R1.5 snapshot-per-standby; R2.5 reconnect-per-peer).
• Does not own the work content (connect, snapshot, config-push bodies stay in their modules); it owns scheduling/isolation.

Dependencies

• Inbound: PeerCtrlDispatch — submits long ops (P0); ReconnectOffload — submits reconnect (P0).
• Outbound: iomgr_monitor (via the submitted task) — register connected fd (P0).
• External: pthreads (threads, mutex, cond, sem) (P0).

Contracts: Service [x] / Batch [x]

Service Interface

typedef void (*slowlane_fn_t)(void *arg);

int  slowlane_init(slowlane_t *sl, int n_workers, int max_inflight);
/* submit returns 0 (enqueued), -EEXIST (key in flight, deduped), -EAGAIN (cap full, queued/deferred);
   never blocks the caller. key==0 disables dedup. */
int  slowlane_submit(slowlane_t *sl, uint64_t key, slowlane_fn_t fn, void *arg);
void slowlane_fini(slowlane_t *sl);   /* drain + join on shutdown only */

• Preconditions: fn performs the blocking work and is responsible for clearing any caller-side in-flight state on completion.
• Postconditions: fn runs exactly once on a lane worker; never on iomgr/delaymgr.
• Invariants: concurrent running tasks ≤ max_inflight; one in-flight task per non-zero key.

Snapshot/checkpoint concurrency invariant (Phase 3, R1.1/1.3/1.5 — MUST hold before snapshot offload lands) Moving persist_checkpoint (which runs trim_wal_segments — deletes WAL segments) + the snapshot fread loop onto a lane worker runs them concurrently with live WAL appends, the fsync_worker flusher, and the WAL-catchup stream. The lane snapshot path must obey:

• Frozen boundary: the snapshot is taken at a fixed (term, lsn) checkpoint boundary captured at submit time; the fread/send loop reads only data at or below that boundary, never live tail bytes.
• Trim safety (MECHANISM, not just goal — see bug-1147): trim_wal_segments (persist.c:1374) TODAY bounds deletion ONLY by active_seg_lsn (the open segment) — it consults no standby acked_lsn, no committed_lsn, no snapshot boundary, and there is no working snapshot backfill (write_snapshot emits only a 16-byte P0 sentinel, not materialized state). So a standby lagging by more than ~one open segment already hits a permanent committed-prefix gap (the bug-795/828 family). Moving checkpoint/trim onto a concurrent lane worker makes this latent race easier to hit. Therefore this spec MUST add the cursor-bound guard as a stopgap: trim_wal_segments computes floor = min(all standby acked_lsn, all in-flight snapshot boundaries) under p->lock and refuses to delete any segment whose range reaches floor or above — never holding p->lock across the fread/send I/O. This is strategy-A retention discipline that should exist regardless of this spec; the full fix for empty/far-behind join (cursor=0) requires entity-bearing snapshots (separate spec) and is explicitly NOT solved here. The design must not assume "a snapshot will backfill the gap" — that path is not built.
• Single in-flight checkpoint: at most one checkpoint/trim in flight (lane dedup key = checkpoint), and at most one snapshot per standby (dedup key = standby id, R1.5).
• No correctness change: the lane only relocates execution; the checkpoint/trim/snapshot content is byte-identical to the current inline path.

Implementation Notes

• Integration: init in controller lifecycle (cluster.c/myapp.c); size from peer count.
• Validation: assert submit never called from a context expecting completion; metric on queue depth/age (fail-loud on backlog); a test asserting concurrent append + checkpoint/trim produces no segment-below-boundary deletion and a consistent snapshot.
• Risks: a lane worker blocked indefinitely (e.g., snapshot to a half-open peer) — bound every in-lane blocking op: the connect uses the bounded socket_connect; the snapshot send must use a bounded socket write (write timeout, mirroring the bounded-connect approach) so a standby going unreachable mid-transfer cannot pin a lane worker. Otherwise stuck transfers exhaust the bounded lane and delay reconnects (which share it). Phase-3 task constraint; size/segment the lane so reconnect and transfer do not starve each other.

LivenessAdmission (lib/vtaskmgr)

Field	Detail
Intent	Make "keep K workers free for liveness" a property of the vtask pool, not of the commit engine — so it covers every blocking op the pool runs once peer-control moves onto it, not just commit-waits
Requirements	3.1, 3.2, 3.3, 4.5

Why this exists (the gap it closes): Today vtaskmgr (include/vtaskmgr.h) has a single FIFO taskq, N vcpu workers, and no priority/reservation; a blocking task holds its vcpu in VCPU_BUSY for the whole block (vtaskmgr.c:248-275). The only admission anywhere is commit.c's inflight_waits counter (COMMIT_WAIT_RESERVED=2, commit.c:200) — external to vtaskmgr and counting only commit-waits. Once peer_ctrl moves onto the pool (PeerCtrlDispatch), the pool also runs the vote-grant term fsync (R4.6) and config-ack waits — blocking ops invisible to the commit counter. So "≥2 workers free of commit-waits" no longer means "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. The reservation must therefore become pool-level.

Responsibilities & Constraints

• Classify every vtask at enqueue into VTASK_LIVENESS (never parks — WAL-ack decode, heartbeat send, vote receipt/decode; commit_on_ack is documented never-block, commit.c:16), VTASK_LIVE_BLOCK (bounded mandatory park — vote-grant term fsync, R4.6), or VTASK_REQ_BLOCK (may park seconds — commit-wait, config-ack wait). Class is knowable from the message type at the dispatch shim.
• The dispatcher reserves VTASK_LIVENESS_RESERVED vcpus: a VTASK_REQ_BLOCK task is not admitted when count(vcpus running *_BLOCK) >= n_vcpus - VTASK_LIVENESS_RESERVED — it fast-rejects with -XCE_ERR_NOT_READY (preserving today's client backpressure, Req 7.1/7.2), it does not silently queue or park a worker.
• VTASK_LIVENESS tasks are never admission-gated → always find a reserved vcpu → acks always advance commit → parked *_BLOCK tasks always wake. Deadlock closed by construction.
• VTASK_LIVE_BLOCK (vote fsync) runs in the reserved headroom and is bounded, so a vote response never queues behind a seconds-long commit-wait (removes a second spurious-election vector).
• commit.c's inflight_waits/COMMIT_WAIT_RESERVED is folded into this — commit_submit becomes "submit a VTASK_REQ_BLOCK"; the pool owns the single reservation (commit_engine_init no longer needs n_vcpus, cluster.c:1332).

Dependencies: Inbound — PeerCtrlDispatch, commit (P0). Outbound — none (lib-internal). External — pthreads.

Contracts: Service [x]

typedef enum { VTASK_LIVENESS=0, VTASK_LIVE_BLOCK, VTASK_REQ_BLOCK } vtask_class_t;
/* submit a classified task; VTASK_REQ_BLOCK returns -XCE_ERR_NOT_READY when the
   reservation would be violated (retriable, never parks a worker). */
int vtask_enq_classified(vtaskmgr_t*, vtask_t*, vtask_class_t);

• Invariant: count(vcpus in *_BLOCK) <= n_vcpus - VTASK_LIVENESS_RESERVED; ≥VTASK_LIVENESS_RESERVED vcpus always available to VTASK_LIVENESS.

Implementation Notes

• Sizing (peer-scaled): R = VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1); commit/REQ_BLOCK cap = n_vcpus − R. Rationale: per-endpoint serialization (endpoint_block_tasks) caps in-flight tasks at one per peer connection, so worst-case concurrent liveness work is N−1 (one ack/vote per other peer) — reserving N−1 means no peer's liveness message ever queues behind a busy reserved worker. Correctness needs only the floor of 1 (one guaranteed-free worker + liveness-never-gated + sub-ms cycling = deadlock-free regardless of peer count); the peer-scaled middle is a latency optimization. Ceiling n_vcpus − 1 keeps ≥1 REQ_BLOCK slot so commits can still run (cap never 0). Both n_peers and n_vcpus are final at init (cluster.c:1332 already passes both). Config-overridable. On a degenerate small-core box (n_vcpus < n_peers) R clamps to n_vcpus − 1 (full per-peer reservation impossible) — still correct via fast cycling; emit a loud WARN at init so the under-provisioning is operator-visible. Validate the chosen R at supported sizes (R8).
• Validation: misclassification is the one failure mode — add a debug assert/loud-WARN if a VTASK_LIVENESS task parks beyond a threshold; task 3.4 (acks advance commit under saturation) is the regression oracle.
• Risk: a future blocking path tagged VTASK_LIVENESS silently breaks the guarantee — review discipline + the assert + the saturation test.
• Revalidation Trigger: any change to the class set, the reservation size, or the dispatch admission rule (consensus correctness + the slow-lane admission contract depend on it).

PeerCtrlDispatch

Field	Detail
Intent	Dispatch peer-control message handling to the vtask pool; fast work inline, long work to the lane
Requirements	1.4, 4.1, 4.2, 4.3, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2

Responsibilities & Constraints

• Peer endpoints dispatch through the existing VTASK_IO path (the message is packet-triggered → reuses vtask_handle_msg), inheriting endpoint_block_tasks per-endpoint serialization (R4.1/4.2).

• The vtask entry performs only bounded protocol work inline (decode, match_index update, commit eval, send next heartbeat) and slowlane_submits any long op; it must not hold ep->busy across long I/O (R4.3).

• Preserves existing vote/ack/term semantics and persist-before-vote ordering (R4.5/4.6) — the logic is unchanged, only its execution context moves.

• Commit-advance preservation (R4.5): WAL-ack handling — which wakes the commit barrier (repl_wait_majority_ack / commit_set_raft_commit_index) — moves off iomgr to the vtask pool. The design preserves that a quorum of acks still advances committed_lsn promptly: ack handling is bounded fast work (never offloaded to the slow lane), per-endpoint serialization does not delay cross-endpoint acks (each peer's acks process independently), and the commit-advance path itself is untouched. This is the only interaction with P1 (raft-apply-decoupling, which owns the barrier's async conversion); this spec must not change ack→commit-advance semantics, only the thread the ack is decoded on.

• Deadlock-freedom argument (R4.5/R3.1 — REQUIRED reasoning, reverses a documented invariant): Today commit_on_ack is explicitly "called from the iomgr read path and never blocks" (commit.c:16, repl.c:220) — the ack that advances committed_lsn and the jobsub vtasks blocked in commit_submit(COMMIT_MODE_WAIT) (the synchronous barrier this spec PRESERVES) run on different threads. Moving peer-control to the vtask pool co-locates the wakers (acks) and the waiters (blocked commit-submits) on one pool — exactly the configuration the iomgr placement was chosen to avoid. Safety rests on a worker reservation — but the existing commit.c cap is insufficient as-is: COMMIT_WAIT_RESERVED (commit.c:200) counts only commit-waits, while moving peer_ctrl onto the pool also puts the vote-grant term fsync (R4.6, VTASK_LIVE_BLOCK) and config-ack waits on it — blocking ops the commit counter can't see. "≥2 workers free of commit-waits" therefore does not mean "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. Closing the deadlock requires the LivenessAdmission component (above) — the reservation must become a pool-level property covering every blocking class, with VTASK_LIVENESS (acks) never admission-gated. Required invariants this spec must hold (and test): (1) WAL-ack/commit-advance decode is tagged VTASK_LIVENESS, never routed to the slow lane, never gated — admissible independently of any blocking count; (2) the reserved headroom is sized for all concurrent liveness work (N-peer acks + vote responses + heartbeat sends), so the reservation — not luck — guarantees an ack always finds a worker. This co-location is transient by design: raft-apply-decoupling P1 converts repl_wait_majority_ack to event-driven (no parked thread per commit), removing the VTASK_REQ_BLOCK waiters from the pool entirely — but until P1 lands, correctness depends on LivenessAdmission holding. Dependency: PeerCtrlDispatch (3.1) requires LivenessAdmission to land first (Phase 1). Gate: task 3.4 (quorum-of-acks advances committed_lsn with the pool saturated by *_BLOCK tasks) is the admission gate for 3.1 — 3.1 is not "done" until 3.4 is green.

Dependencies

• Inbound: iomgr — dispatches the packet (P0).
• Outbound: vtaskmgr (P0); SlowLane — long ops (P0); existing commit/repl/persist logic (P0).

Contracts: Service [x]

Implementation Notes

• Integration: change peer endpoint msg_handler registration (ha.c:4265/4307/6349) to the vtask-dispatching shim; convert peer_ctrl_msg_handler body into the vtask entry.
• Validation: existing election/heartbeat/repl tests must pass unchanged (R6); assert no long op runs inline in the entry.
• Risks: a heavy op accidentally left inline → per-peer head-of-line blocking (guard by review + a test asserting the handler returns promptly).

BoundedConnect (lib/netio)

Field	Detail
Intent	Make socket_connect non-blocking + poll-bounded
Requirements	2.3

Responsibilities & Constraints: set O_NONBLOCK before connect; on EINPROGRESS, poll up to a configured timeout; return success/timeout/error. Reuses the xce_probe_config_tag pattern. Contracts: Service [x]

/* connect_timeout_ms: bound the attempt; 0 = use default (config). */
int socket_connect(const char *host, int port, int connect_timeout_ms);

Implementation Notes: callers updated to pass the timeout; default in config.h. This is a behavior change to a shared helper → Revalidation Trigger for all connect callers.

ReconnectOffload (controller/ha)

Field	Detail
Intent	delaymgr triggers an offloaded, deduped, bounded reconnect
Requirements	2.1, 2.2, 2.4, 2.5, 2.6

Responsibilities & Constraints: primary_heartbeat_fn (delaymgr) detects a disconnected peer and slowlane_submits the reconnect (per-peer key for dedup); the lane worker does the bounded connect, prepares the endpoint, registers it via lock-guarded iomgr_monitor; on failure clears the in-flight flag and lets the next tick retry (never self-terminates). Contracts: Service [x] Implementation Notes: refcount the endpoint across the in-flight window; resolve the inbound-vs-outbound connection race per the existing connection-ownership rules (ha-async-io.md).

Test

PartitionTest

Field	Detail
Intent	Make one peer unreachable (blackhole), assert liveness + no spurious election + heal
Requirements	7.1, 7.2, 7.3, 7.4

Contracts: Batch [x] Implementation Notes: the partition primitive must render one peer unreachable (traffic dropped, connect hangs), distinct from kill -9. Two mechanisms, primary = in-process so R7 actually runs in this project's environment (the sibling specs ha-term-safety 5.2/5.4 are already perpetually ENV-BLOCKED on the unprivileged/overlayfs sandbox; a firewall-only R7 would be a permanent skip and the spec's headline guarantee would never be exercised):

• Primary (portable, no privilege): in-process socket-drop hook in xce_test — a test-only seam (XCE_TESTING) that makes one node silently drop/blackhole inbound+outbound peer traffic at the endpoint layer (e.g. stop reading/writing the peer fd, or a drop flag in the I/O path), simulating "unreachable" without any kernel/firewall change. Runs anywhere the unit/soak suite runs.
• Secondary (infra-level, optional): firewall iptables/nft DROP on the node's port for real-network fidelity; capability-gated with a loud skip when unprivileged. Complements, does not replace, the in-process path.

Assert (both): heartbeats to survivors continue within the interval, no new-leader election while the leader holds majority, peer rejoins on heal, zero committed-job loss.

Error Handling

Error Strategy

• Connect timeout/refused → bounded poll returns; lane clears in-flight; delaymgr retries next tick; loud WARN with peer + elapsed (fail-loud, never abort — controllers never self-terminate).
• Lane backlog → submit returns -EAGAIN when the cap is reached; the work is queued, never run on a drive thread; emit a metric/WARN on queue depth + oldest-age so a saturated lane is operator-visible.
• Snapshot/config-push failure on the lane → reported through the existing repl/config error paths unchanged; lane only relocates execution, not error semantics.
• Dedup race → submit with an in-flight key is a no-op (-EEXIST); idempotent.

Monitoring

• Metrics: lane queue depth, oldest-task age, in-flight count, per-peer reconnect attempts. WARN thresholds make a degraded/partitioned window visible (consistent with fail-loud).

Testing Strategy

Unit Tests

• SlowLane: slowlane_submit is non-blocking; admission cap bounds concurrent tasks; per-key dedup returns -EEXIST; fn runs exactly once (3.1–3.4, 1.5).
• BoundedConnect: socket_connect returns within the timeout against an unreachable host; succeeds against a reachable one (2.3).
• PeerCtrlDispatch: the vtask entry returns promptly (no long op inline); per-endpoint serialization holds via endpoint_block_tasks (4.1–4.3).

Integration Tests

• Reconnect offload: with one peer unreachable, delaymgr ticks continue and heartbeats to reachable peers are sent within the interval; reconnect is deduped per peer (2.1, 2.2, 2.5).
• Snapshot offload: a catch-up triggering a snapshot does not block iomgr; other peers' heartbeats/votes processed during the transfer; no concurrent transfer per standby (1.1, 1.3, 1.4, 1.5).
• Commit-advance under pool load (R4.5): with the vtask pool near saturation, a quorum of WAL-acks still advances committed_lsn promptly; cross-endpoint acks are not serialized behind one another.
• Snapshot/checkpoint concurrency (R1.1/1.3/1.5): concurrent live appends + a lane checkpoint/trim never delete a segment below an in-flight snapshot boundary or a standby catch-up cursor; the produced snapshot is consistent.
• Regression sweep (R6): run the existing suites unchanged — test_ha_election, test_ha_prevote, test_ha_liveness, test_ha_failover, test_ha_rdhup_integration, test_repl_majority/apply/catchup, test_config_repl/test_ha_repl_push, test_persist_recovery, test_commit_barrier_rollout/test_commit_durability; all pass without behavior changes (6.1–6.7).

E2E / Soak Tests

• Partition soak (R7): blackhole one peer → assert heartbeats to survivors continue, no spurious election while leader holds majority, peer rejoins on heal, zero committed-job loss (7.1–7.4).
• Failover soak (existing, kill-based) continues to pass (6.3).

Performance

• Steady-state heartbeat cadence at 3–5 controllers shows no added latency attributable to the new dispatch path; the heartbeat/vote send paths remain non-blocking enqueues (8.1, 8.2).

Migration Strategy

Phased, each phase independently testable (matches research recommendation):

Ordering rule: the slow lane is foundational — every offload (reconnect, snapshot, config-push) submits to it, so it must exist first. And the peer_ctrl→vtask move must land together with its long-op offload in the same phase: moving peer_ctrl to a vtask while still running snapshot/config-push inline would hold endpoint_block_tasks (ep->busy) across multi-second I/O (transient R4.3 violation) and consume a pool worker. The two are therefore coupled.

flowchart TD
    P1[Phase 1: SlowLane executor + BoundedConnect] --> P2[Phase 2: ReconnectOffload sender-side]
    P2 --> P3[Phase 3: PeerCtrlDispatch to vtask PLUS snapshot and configpush offload coupled]
    P3 --> P4[Phase 4: PartitionTest + regression sweep]

• Phase 1 (foundational, lowest risk): build the SlowLane executor (bounded pool + admission + dedup — new component, no behavior change yet, unit-testable in isolation); reorder socket_connect to bounded non-blocking + poll (independent; already bounds the delaymgr stall even before reconnect is offloaded); and add LivenessAdmission to vtaskmgr (task-class reservation; fold in COMMIT_WAIT_RESERVED). LivenessAdmission is a hard prerequisite of Phase 3 — peer_ctrl cannot move onto the pool deadlock-safely without it — so it must land in Phase 1 as dormant infra (no behavior change until peer_ctrl dispatches through it).
• Phase 2: offload the delaymgr peer reconnect to the lane (deduped per peer, registered via lock-guarded iomgr_monitor) → closes sender-side heartbeat starvation. Shippable.
• Phase 3 (coupled — the R4.3-safe step): dispatch peer-control to the vtask pool and offload the long ops it triggers (snapshot send, config-push transport-only) to the lane in the same change, so no phase ever runs long I/O inline in a blocked vtask. Carries the snapshot/checkpoint concurrency invariant (frozen (term,lsn) boundary, trim-vs-catch-up-cursor safety, single in-flight checkpoint) and the bounded in-lane snapshot write.
• Phase 4: partition test + full regression sweep.
• Rollback: each phase is behavior-preserving; a phase can be reverted independently since the contract (non-blocking drive) is the only invariant added. P1 adds dormant infra; the first observable change is P2.