myxce/.kiro/specs/raft-drive-async/research.md

Gap Analysis — raft-drive-async (2026-06-25)

Analyzes the gap between the 8 requirements and the existing controller code, to inform the design phase. Evidence from this session's blocking audits + targeted greps.

Current State Investigation

The existing deferral idiom and why it's insufficient

XCE already practices "don't block iomgr": the established pattern is delay_task_ms(&c->delaymgr, ...), which ordq_puts a callback and returns immediately. Its own comments confirm the intent — "run off the iomgr walAck thread … delay_task_ms only enqueues" (config_repl.c:2369, ha_convergence.c:207, config_repl.c:428). But the deferred callback then runs on the delaymgr thread — the heartbeat/election-timeout driver. So the only existing "offload" target is itself a liveness thread. This is the conceptual heart of the gap: the codebase can defer, but its single deferral lane is the thread the corrected ha-async-io.md contract must keep free. The missing capability is a non-liveness execution lane (a "slow lane").

Inline-on-iomgr root cause

peer_ctrl_msg_handler is assigned directly as ep->msg_handler at ha.c:4265 (on_vote_connected), ha.c:4307 (on_pre_vote_connected), ha.c:6349 (standby accept) — so it runs on the iomgr loop. Normal inbound messages instead go handle_msg → vtask_handle_msg → vtask pool (cluster.c:711, registered cluster.c:772). The peer-control path is the anomaly that bypasses the pool.

Reusable building blocks that already exist

• vtask submit primitives: vtask_alloc / vtask_init (vtaskmgr.c:21) / vtask_enq (vtaskmgr.c:342) exist and compose; the only public entry vtask_handle_msg (vtaskmgr.c:356) hardcodes vtype=VTASK_IO and requires ep->fsm_in.pkt. A general non-IO submit is a small composition of the existing primitives, not new infrastructure.
• Per-endpoint serialization (explicit primitive): endpoint_block_tasks() / endpoint_unblock_tasks() (netio.c:896-911) set a per-endpoint ep->busy flag; the VTASK_IO path auto-blocks on task start (vtaskmgr.c:249), unblocks on completion (vtaskmgr.c:66), and the handler path uses it (xcectx.c:440/449), while vtask_eligible (vtaskmgr.c:203) skips a busy endpoint's other tasks. This directly and explicitly satisfies R4.1 (per-connection processing order) when peer control reuses the IO path — a reuse, not new logic. Cross-peer isolation is preserved (block is per-ep, so peer X busy ≠ peer Y blocked). Design trap: the block is held for the whole task duration — so a long op (snapshot send) run inline in a blocked vtask would queue that peer's own subsequent heartbeat-ACKs behind it. This makes the two-tier rule (fast work in vtask, trigger long I/O to the slow lane) mandatory for R4.1 + R1 to both hold.
• Admission cap pattern: COMMIT_WAIT_RESERVED=2 of nvcpus (default 8) caps blocking commit-waits at max(1, n-2) (commit.c:200, commit.h:17) — a reusable pattern for R3, but it is commit-engine-specific today.
• Dedicated-background-thread precedent: fsync_worker (persist.c:624) is a long-lived background thread; push_config_transport_only already spawns per-standby push_threads (config_repl.c:1412) — then blocks by pthread_join/wait_all_with_timeout on the drive thread (config_repl.c:1332). So the threads exist; the defect is the blocking join on the caller, not absent concurrency.
• Bounded-connect pattern: xce_probe_config_tag already does O_NONBLOCK-before-connect + poll(500ms) (config_repl.c:1753-1761). The buggy path (socket_connect, netio.c:289/302) sets non-blocking after connect (netio.c:716) — so R2.3 is largely a reorder + reuse, not new code.

Confirmed absent

• No iomgr cross-thread wake primitive — CORRECTED 2026-06-25 (was overstated). epoll registration is ALREADY thread-safe: iomgr_monitor (EPOLL_CTL_ADD), iomgr_unmonitor (DEL), iomgr_update_poll (MOD) all wrap epoll_ctl in mylock_lock(&iomgr->lock) (netio.c:1046-1082), and epoll_ctl is kernel-thread-safe regardless. The connect→register flow already runs from delaymgr/main threads today (e.g. on_vote_connected). So a worker can do the blocking connect off-thread, fully prepare the endpoint (set ep->msg_handler etc.) BEFORE it is monitored (an unmonitored fd yields no events → no race), then call the lock-guarded iomgr_monitor. No new eventfd/iomgr-post primitive is needed; the marshal-back is "reuse iomgr_monitor." Residual (design-verify, not a primitive): ensure worker-side handling does not concurrently touch an endpoint's read-FSM buffer that iomgr owns — already mitigated by endpoint_block_tasks per-endpoint serialization + lock-guarded poll changes.
• No network-partition test primitive: the soak (run-ha-soak.sh) is process-kill only (kill -9, lines 74/440); unit "unreachable peer" cases use closed ports, not traffic drops. R7 has no existing harness.

Requirement → Asset Map

Req	Existing asset	Gap tag
R1 liveness during snapshot/config I/O	snapshot send ha.c:2702+config_repl.c:2446; config-push threads config_repl.c:1412	Missing off-load lane for snapshot; Constraint config-push blocks on join
R2 unreachable-peer / bounded connect	endpoint_connect/socket_connect netio.c:289/302 (blocking); xce_probe_config_tag pattern (reuse); reconnect via delaymgr primary_heartbeat_fn	Constraint (reorder connect) + Missing (offload reconnect off delaymgr)
R3 admission protection	COMMIT_WAIT_RESERVED pattern (commit-specific)	Missing (generalize to the slow lane)
R4 ordering/correctness	endpoint_block_tasks/endpoint_unblock_tasks per-endpoint sequential primitive (netio.c:896-911, auto on VTASK_IO path); lock-guarded iomgr_monitor/iomgr_unmonitor (netio.c:1046-1082) for thread-safe epoll registration	Reuse (both primitives exist; no new marshal-back primitive needed)
R5 contract compliance	ha-async-io.md contract (steering authority)	verified by R1–R4 (no standalone asset)
R6 no regression	existing suites: test_ha_election/prevote/liveness/failover/rdhup_integration, test_repl_majority/apply/catchup, test_config_repl/test_ha_repl_push, test_persist_recovery, test_commit_barrier_rollout/test_commit_durability	Reuse as regression gate
R7 partition verification	run-ha-soak.sh (kill-only)	Missing (no traffic-drop primitive)
R8 heartbeat perf	heartbeat fan-out already non-blocking enqueue	Constraint (don't regress fast path)

Implementation Approach Options

Option A — Extend in place (minimal)

Dispatch peer_ctrl_msg_handler through the existing handle_msg/vtask_handle_msg path; add a general non-IO vtask_enq wrapper; reorder socket_connect to non-blocking + poll; replace config-push wait_all_with_timeout join with a non-blocking completion check; reuse delay_task_ms only for non-blocking triggers. Touches ha.c, config_repl.c, netio.c, vtaskmgr.c.

• ✅ Least new surface; reuses every existing pattern; fastest path to closing the delaymgr+iomgr inline blocks.
• ❌ Long I/O (snapshot, config-push) still lands in the request vtask pool → relocates the stall to pool exhaustion (R3 only partially met).

Option B — New dedicated slow-lane executor

Build a dedicated bounded slow-lane executor (generalize fsync_worker) for all long blocking I/O. Peer control still dispatched to vtask, but heavy work re-offloaded to the slow lane. (No iomgr-post primitive needed — epoll registration reuses the lock-guarded iomgr_monitor.)

• ✅ Clean separation; R3/R4 fully met by construction; isolates seconds-long I/O from request handling and from delaymgr.
• ❌ One genuinely new component; more interface design than extending in place.

Option C — Hybrid (recommended)

• Extend (reuse): peer_ctrl → vtask via the existing dispatch pattern; general non-IO vtask submit (compose vtask_alloc/init/enq); bounded connect (reorder + reuse xce_probe_config_tag pattern); preserve ordering via VTASK_IO serialization.
• New (the one real gap): a small bounded slow-lane executor for snapshot-transfer + config-push (generalize the fsync_worker/push_thread precedent; per-peer dedup + admission cap parallel to COMMIT_WAIT_RESERVED). Epoll registration of a connected fd reuses the lock-guarded iomgr_monitor (no new primitive).
• Phased:
  • Phase 1 (S–M, highest value/lowest risk): bounded connect + offload reconnect off delaymgr → closes the heartbeat sender-side starvation (R2). Independent, shippable.
  • Phase 2 (M): dispatch peer_ctrl to vtask (epoll registration reuses lock-guarded iomgr_monitor) → closes the iomgr receiver-side inline processing (R4, R5).
  • Phase 3 (L): slow-lane executor + move snapshot-send/config-push to it + admission generalization (R1, R3).
  • Phase 4 (M): partition test harness + regression sweep (R6, R7, R8).
• ✅ Each phase is independently testable and de-risks the next; matches what already exists vs. what's truly missing.
• ❌ Requires phase coordination; must keep ha-async-io.md compliance consistent across phases.

Effort & Risk

Work item	Effort	Risk	Justification
Bounded connect + reconnect offload (Ph1)	S–M	Medium	reorder existing pattern; offload target needs to be non-delaymgr
General non-IO vtask submit	S	Low	compose existing vtask_alloc/init/enq
peer_ctrl → vtask dispatch	M	Medium	correctness-sensitive: ordering (mitigated by endpoint_block_tasks), term/vote semantics; epoll register reuses lock-guarded iomgr_monitor
Slow-lane executor + snapshot-send offload	L	Medium-High	per-peer dedup + coordination with ongoing append/WAL-trim/catch-up stream
Admission generalization (R3)	M	Medium	extend COMMIT_WAIT_RESERVED pattern to slow lane
Partition test harness (R7)	M	Medium	no traffic-drop primitive today; soak infra is kill-only
Overall	L (1–2 wks)	Medium	riskiest: snapshot-offload coordination + moving peer_ctrl off iomgr (no new epoll-core primitive needed)

Recommendations for Design Phase

• Preferred approach: Option C (Hybrid), phased. Ship Phase 1 (bounded connect) early — it independently closes the sender-side heartbeat starvation and is low-risk.
• Key decisions to settle in design:
  1. Connected-fd registration: reuse the lock-guarded iomgr_monitor (netio.c:1058) — a worker prepares the endpoint fully, then registers it; no new primitive. Design-verify only that worker handling does not touch an endpoint's read-FSM buffer concurrently with iomgr (serialized by endpoint_block_tasks).
  2. Slow-lane shape: a shared bounded pool vs. reuse the existing per-standby push_thread model for config-push and a small pool for snapshot. Size to peer count.
  3. Fold-in question — RESOLVED 2026-06-25: keep raft-drive-async STANDALONE. Scope-check verdict ADJACENT-SAME-AREA: election-iomgr-refactor is fully implemented and fixed the startup blocking path (pthread election before cluster_run); it touched none of raft-drive-async's runtime targets (peer_ctrl_msg_handler dispatch, delaymgr connect, snapshot/config-push on iomgr) and explicitly deferred async DNS as a non-goal. It is consumed as completed upstream, providing reusable primitives (delay_task_ms, on_connected fire-once callback, the ha_start_election_async/non-blocking vote-fan-out contract). Folding in would reopen a done spec for net-new infra it never held.

Research Needed (carry to design)

• election-iomgr-refactor scope overlap — RESOLVED 2026-06-25 (scope-check): ADJACENT-SAME-AREA, keep standalone; reuse its primitives (delay_task_ms, on_connected, ha_start_election_async). See Recommendations decision #3.
• Endpoint read-FSM ownership: confirm worker-side peer-control handling never touches an endpoint's read-FSM buffer concurrently with the iomgr read path (expected safe via endpoint_block_tasks + handoff of fsm_in.pkt). (Epoll registration itself is already settled — lock-guarded iomgr_monitor; no new primitive.)
• Partition primitive choice for R7: iptables/tc (root, infra-level) vs. an in-process socket-drop hook in xce_test. Determines test portability.
• Snapshot-send coordination with the concurrent WAL-catch-up stream and the primary's append/trim_wal_segments while the transfer runs on the slow lane.
• Admission interaction between the existing COMMIT_WAIT_RESERVED (commit waits) and the new slow-lane cap — shared budget vs. independent, so the two together can't starve the 8-worker pool.

---

Design Synthesis (2026-06-25, /kiro-spec-design)

Key simplification chosen at design time: the offload target for long blocking I/O is a dedicated slow-lane executor (separate threads), NOT the vtask request pool. Consequences:

1. R3 (pool protection) becomes structural — long ops never run on the vtask pool, so they cannot starve liveness-critical message handling. The existing COMMIT_WAIT_RESERVED continues to cover commit-waits.
2. The "general non-IO vtask submit" research item is SUPERSEDED — not needed. Peer-control handling is packet-triggered, so it reuses the existing VTASK_IO vtask_handle_msg path; the only packet-less trigger (delaymgr reconnect) goes to the slow lane, not the vtask pool. No new vtask submit primitive required.
3. Combined with the already-refuted iomgr-post primitive (epoll registration reuses lock-guarded iomgr_monitor), the only genuinely-new component is the slow-lane executor (slowlane.{c,h}). Everything else is reuse + integration.

Build-vs-adopt: slow lane generalizes the existing fsync_worker/push_thread background-thread precedent (adopt pattern, build the small bounded executor). Bounded connect reuses the xce_probe_config_tag pattern (reorder, not new). Per-endpoint ordering reuses endpoint_block_tasks.

Phased migration (P1 bounded-connect → P2 peer_ctrl→vtask → P3 slow-lane+offload → P4 partition test) keeps each step behavior-preserving and independently revertible.