myxce/.kiro/specs/raft-drive-async/research.md

# Gap Analysis — raft-drive-async (2026-06-25)

Analyzes the gap between the 8 requirements and the existing controller code, to inform the design phase. Evidence from this session's blocking audits + targeted greps.

## Current State Investigation

### The existing deferral idiom and why it's insufficient
XCE already practices "don't block iomgr": the established pattern is `delay_task_ms(&c->delaymgr, ...)`, which `ordq_put`s a callback and returns immediately. Its own comments confirm the intent — "run off the iomgr walAck thread … delay_task_ms only enqueues" (`config_repl.c:2369`, `ha_convergence.c:207`, `config_repl.c:428`). **But the deferred callback then runs on the delaymgr thread — the heartbeat/election-timeout driver.** So the only existing "offload" target is itself a liveness thread. This is the conceptual heart of the gap: the codebase can *defer*, but its single deferral lane is the thread the corrected `ha-async-io.md` contract must keep free. **The missing capability is a non-liveness execution lane (a "slow lane").**

### Inline-on-iomgr root cause
`peer_ctrl_msg_handler` is assigned directly as `ep->msg_handler` at `ha.c:4265` (on_vote_connected), `ha.c:4307` (on_pre_vote_connected), `ha.c:6349` (standby accept) — so it runs on the iomgr loop. Normal inbound messages instead go `handle_msg` → `vtask_handle_msg` → vtask pool (`cluster.c:711`, registered `cluster.c:772`). The peer-control path is the anomaly that bypasses the pool.

### Reusable building blocks that already exist
- **vtask submit primitives**: `vtask_alloc` / `vtask_init` (vtaskmgr.c:21) / `vtask_enq` (vtaskmgr.c:342) exist and compose; the only public entry `vtask_handle_msg` (vtaskmgr.c:356) hardcodes `vtype=VTASK_IO` and *requires* `ep->fsm_in.pkt`. A general non-IO submit is a **small composition** of the existing primitives, not new infrastructure.
- **Per-endpoint serialization (explicit primitive)**: `endpoint_block_tasks()` / `endpoint_unblock_tasks()` (netio.c:896-911) set a per-endpoint `ep->busy` flag; the VTASK_IO path auto-blocks on task start (vtaskmgr.c:249), unblocks on completion (vtaskmgr.c:66), and the handler path uses it (xcectx.c:440/449), while `vtask_eligible` (vtaskmgr.c:203) skips a busy endpoint's other tasks. This **directly and explicitly satisfies R4.1** (per-connection processing order) when peer control reuses the IO path — a reuse, not new logic. Cross-peer isolation is preserved (block is per-`ep`, so peer X busy ≠ peer Y blocked). **Design trap:** the block is held for the whole task duration — so a long op (snapshot send) run *inline* in a blocked vtask would queue that peer's own subsequent heartbeat-ACKs behind it. This makes the two-tier rule (fast work in vtask, **trigger** long I/O to the slow lane) mandatory for R4.1 + R1 to both hold.
- **Admission cap pattern**: `COMMIT_WAIT_RESERVED=2` of `nvcpus` (default 8) caps blocking commit-waits at `max(1, n-2)` (commit.c:200, commit.h:17) — a reusable *pattern* for R3, but it is commit-engine-specific today.
- **Dedicated-background-thread precedent**: `fsync_worker` (persist.c:624) is a long-lived background thread; `push_config_transport_only` **already spawns per-standby `push_thread`s** (config_repl.c:1412) — then blocks by `pthread_join`/`wait_all_with_timeout` on the drive thread (config_repl.c:1332). So the *threads* exist; the defect is the **blocking join on the caller**, not absent concurrency.
- **Bounded-connect pattern**: `xce_probe_config_tag` already does `O_NONBLOCK`-before-connect + `poll(500ms)` (config_repl.c:1753-1761). The buggy path (`socket_connect`, netio.c:289/302) sets non-blocking *after* connect (netio.c:716) — so R2.3 is largely a **reorder + reuse**, not new code.

### Confirmed absent
- ~~No iomgr cross-thread wake primitive~~ — **CORRECTED 2026-06-25 (was overstated).** epoll registration is ALREADY thread-safe: `iomgr_monitor` (EPOLL_CTL_ADD), `iomgr_unmonitor` (DEL), `iomgr_update_poll` (MOD) all wrap `epoll_ctl` in `mylock_lock(&iomgr->lock)` (netio.c:1046-1082), and `epoll_ctl` is kernel-thread-safe regardless. The connect→register flow already runs from delaymgr/main threads today (e.g. `on_vote_connected`). So a worker can do the blocking connect off-thread, fully prepare the endpoint (set `ep->msg_handler` etc.) BEFORE it is monitored (an unmonitored fd yields no events → no race), then call the lock-guarded `iomgr_monitor`. **No new eventfd/iomgr-post primitive is needed; the marshal-back is "reuse `iomgr_monitor`."** Residual (design-verify, not a primitive): ensure worker-side handling does not concurrently touch an endpoint's read-FSM buffer that iomgr owns — already mitigated by `endpoint_block_tasks` per-endpoint serialization + lock-guarded poll changes.
- **No network-partition test primitive**: the soak (`run-ha-soak.sh`) is process-kill only (`kill -9`, lines 74/440); unit "unreachable peer" cases use closed ports, not traffic drops. R7 has **no existing harness**.

## Requirement → Asset Map

| Req | Existing asset | Gap tag |
|-----|----------------|---------|
| R1 liveness during snapshot/config I/O | snapshot send `ha.c:2702`+`config_repl.c:2446`; config-push threads `config_repl.c:1412` | **Missing** off-load lane for snapshot; **Constraint** config-push blocks on join |
| R2 unreachable-peer / bounded connect | `endpoint_connect`/`socket_connect` netio.c:289/302 (blocking); `xce_probe_config_tag` pattern (reuse); reconnect via delaymgr `primary_heartbeat_fn` | **Constraint** (reorder connect) + **Missing** (offload reconnect off delaymgr) |
| R3 admission protection | `COMMIT_WAIT_RESERVED` pattern (commit-specific) | **Missing** (generalize to the slow lane) |
| R4 ordering/correctness | `endpoint_block_tasks`/`endpoint_unblock_tasks` per-endpoint sequential primitive (netio.c:896-911, auto on VTASK_IO path); lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (netio.c:1046-1082) for thread-safe epoll registration | **Reuse** (both primitives exist; no new marshal-back primitive needed) |
| R5 contract compliance | `ha-async-io.md` contract (steering authority) | verified by R1–R4 (no standalone asset) |
| R6 no regression | existing suites: `test_ha_election/prevote/liveness/failover/rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability` | **Reuse** as regression gate |
| R7 partition verification | `run-ha-soak.sh` (kill-only) | **Missing** (no traffic-drop primitive) |
| R8 heartbeat perf | heartbeat fan-out already non-blocking enqueue | **Constraint** (don't regress fast path) |

## Implementation Approach Options

### Option A — Extend in place (minimal)
Dispatch `peer_ctrl_msg_handler` through the existing `handle_msg`/`vtask_handle_msg` path; add a general non-IO `vtask_enq` wrapper; reorder `socket_connect` to non-blocking + `poll`; replace config-push `wait_all_with_timeout` join with a non-blocking completion check; reuse `delay_task_ms` only for *non-blocking triggers*. Touches ha.c, config_repl.c, netio.c, vtaskmgr.c.
- ✅ Least new surface; reuses every existing pattern; fastest path to closing the delaymgr+iomgr inline blocks.
- ❌ Long I/O (snapshot, config-push) still lands in the **request** vtask pool → relocates the stall to pool exhaustion (R3 only partially met).

### Option B — New dedicated slow-lane executor
Build a dedicated bounded **slow-lane executor** (generalize `fsync_worker`) for all long blocking I/O. Peer control still dispatched to vtask, but heavy work re-offloaded to the slow lane. (No iomgr-post primitive needed — epoll registration reuses the lock-guarded `iomgr_monitor`.)
- ✅ Clean separation; R3/R4 fully met by construction; isolates seconds-long I/O from request handling and from delaymgr.
- ❌ One genuinely new component; more interface design than extending in place.

### Option C — Hybrid (recommended)
- **Extend** (reuse): peer_ctrl → vtask via the existing dispatch pattern; general non-IO vtask submit (compose `vtask_alloc/init/enq`); bounded connect (reorder + reuse `xce_probe_config_tag` pattern); preserve ordering via `VTASK_IO` serialization.
- **New** (the one real gap): a small bounded **slow-lane executor** for snapshot-transfer + config-push (generalize the `fsync_worker`/`push_thread` precedent; per-peer dedup + admission cap parallel to `COMMIT_WAIT_RESERVED`). Epoll registration of a connected fd reuses the lock-guarded `iomgr_monitor` (no new primitive).
- **Phased**:
  - *Phase 1 (S–M, highest value/lowest risk):* bounded connect + offload reconnect off delaymgr → closes the heartbeat *sender*-side starvation (R2). Independent, shippable.
  - *Phase 2 (M):* dispatch peer_ctrl to vtask (epoll registration reuses lock-guarded `iomgr_monitor`) → closes the iomgr *receiver*-side inline processing (R4, R5).
  - *Phase 3 (L):* slow-lane executor + move snapshot-send/config-push to it + admission generalization (R1, R3).
  - *Phase 4 (M):* partition test harness + regression sweep (R6, R7, R8).
- ✅ Each phase is independently testable and de-risks the next; matches what already exists vs. what's truly missing.
- ❌ Requires phase coordination; must keep `ha-async-io.md` compliance consistent across phases.

## Effort & Risk

| Work item | Effort | Risk | Justification |
|-----------|--------|------|---------------|
| Bounded connect + reconnect offload (Ph1) | S–M | Medium | reorder existing pattern; offload target needs to be non-delaymgr |
| General non-IO vtask submit | S | Low | compose existing `vtask_alloc/init/enq` |
| peer_ctrl → vtask dispatch | M | Medium | correctness-sensitive: ordering (mitigated by `endpoint_block_tasks`), term/vote semantics; epoll register reuses lock-guarded `iomgr_monitor` |
| Slow-lane executor + snapshot-send offload | L | Medium-High | per-peer dedup + coordination with ongoing append/WAL-trim/catch-up stream |
| Admission generalization (R3) | M | Medium | extend `COMMIT_WAIT_RESERVED` pattern to slow lane |
| Partition test harness (R7) | M | Medium | no traffic-drop primitive today; soak infra is kill-only |
| **Overall** | **L (1–2 wks)** | **Medium** | riskiest: snapshot-offload coordination + moving peer_ctrl off iomgr (no new epoll-core primitive needed) |

## Recommendations for Design Phase

- **Preferred approach:** Option C (Hybrid), phased. Ship Phase 1 (bounded connect) early — it independently closes the sender-side heartbeat starvation and is low-risk.
- **Key decisions to settle in design:**
  1. **Connected-fd registration:** reuse the lock-guarded `iomgr_monitor` (netio.c:1058) — a worker prepares the endpoint fully, then registers it; no new primitive. Design-verify only that worker handling does not touch an endpoint's read-FSM buffer concurrently with iomgr (serialized by `endpoint_block_tasks`).
  2. **Slow-lane shape:** a shared bounded pool vs. reuse the existing per-standby `push_thread` model for config-push and a small pool for snapshot. Size to peer count.
  3. **Fold-in question — RESOLVED 2026-06-25: keep `raft-drive-async` STANDALONE.** Scope-check verdict ADJACENT-SAME-AREA: `election-iomgr-refactor` is fully implemented and fixed the *startup* blocking path (pthread election before `cluster_run`); it touched none of raft-drive-async's runtime targets (`peer_ctrl_msg_handler` dispatch, delaymgr `connect`, snapshot/config-push on iomgr) and explicitly deferred async DNS as a non-goal. It is consumed as completed *upstream*, providing reusable primitives (`delay_task_ms`, `on_connected` fire-once callback, the `ha_start_election_async`/non-blocking vote-fan-out contract). Folding in would reopen a done spec for net-new infra it never held.

## Research Needed (carry to design)

- ~~election-iomgr-refactor scope overlap~~ — **RESOLVED 2026-06-25** (scope-check): ADJACENT-SAME-AREA, keep standalone; reuse its primitives (`delay_task_ms`, `on_connected`, `ha_start_election_async`). See Recommendations decision #3.
- **Endpoint read-FSM ownership**: confirm worker-side peer-control handling never touches an endpoint's read-FSM buffer concurrently with the iomgr read path (expected safe via `endpoint_block_tasks` + handoff of `fsm_in.pkt`). (Epoll registration itself is already settled — lock-guarded `iomgr_monitor`; no new primitive.)
- **Partition primitive choice** for R7: `iptables`/`tc` (root, infra-level) vs. an in-process socket-drop hook in `xce_test`. Determines test portability.
- **Snapshot-send coordination** with the concurrent WAL-catch-up stream and the primary's append/`trim_wal_segments` while the transfer runs on the slow lane.
- **Admission interaction** between the existing `COMMIT_WAIT_RESERVED` (commit waits) and the new slow-lane cap — shared budget vs. independent, so the two together can't starve the 8-worker pool.

---

## Design Synthesis (2026-06-25, /kiro-spec-design)

Key simplification chosen at design time: **the offload target for long blocking I/O is a dedicated slow-lane executor (separate threads), NOT the vtask request pool.** Consequences:

1. **R3 (pool protection) becomes structural** — long ops never run on the vtask pool, so they cannot starve liveness-critical message handling. The existing `COMMIT_WAIT_RESERVED` continues to cover commit-waits.
2. **The "general non-IO vtask submit" research item is SUPERSEDED** — not needed. Peer-control handling is packet-triggered, so it reuses the existing VTASK_IO `vtask_handle_msg` path; the only packet-less trigger (delaymgr reconnect) goes to the slow lane, not the vtask pool. No new vtask submit primitive required.
3. Combined with the already-refuted iomgr-post primitive (epoll registration reuses lock-guarded `iomgr_monitor`), **the only genuinely-new component is the slow-lane executor** (`slowlane.{c,h}`). Everything else is reuse + integration.

Build-vs-adopt: slow lane generalizes the existing `fsync_worker`/`push_thread` background-thread precedent (adopt pattern, build the small bounded executor). Bounded connect reuses the `xce_probe_config_tag` pattern (reorder, not new). Per-endpoint ordering reuses `endpoint_block_tasks`.

Phased migration (P1 bounded-connect → P2 peer_ctrl→vtask → P3 slow-lane+offload → P4 partition test) keeps each step behavior-preserving and independently revertible.