lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: promote raft-apply-decoupling spec family from /kiro-discovery

Browse source 
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.

- raft-drive-async: full spec docs (brief/requirements/research/design)
  committed alongside its already-tracked tasks.md; async-drive prerequisite
  for the apply-decoupling work, sequenced to start first (threatens
  failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
  apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
  decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
  sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
  documenting the current on-receipt fact-reaction deviation this family
  will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
  "implemented" (both were mislabeled tasks-generated despite full 18/18
  and 24/24 task completion + raft_sim GO).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
Rongsong Shen 2026-07-03 02:41:11 +00:00
parent 9318f9977e
commit 5f307e297b
14 changed files with 1075 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

92
.kiro/specs/raft-drive-async/brief.md Normal file
View file

@ -0,0 +1,92 @@
# Brief: raft-drive-async
## Problem
The Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals (iomgr and delaymgr). When either stalls, heartbeats/votes/ACKs stop flowing → peers declare the leader dead → spurious election. This is a likely root of the **bug-851 chaotic-failover family**.
A blocking audit (2026-06-25) found three CRITICAL violations of the (newly corrected) thread-role contract in `ha-async-io.md`:
1. **Snapshot send + `persist_checkpoint` inline on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) runs `persist_checkpoint` (fdatasync + WAL-segment trim) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503) on the iomgr event loop. A large snapshot freezes ALL heartbeat/vote/client processing for its whole duration.
2. **Config-push wait inline on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172/1177/2672/2677/3973), up to `CONF_PUSH_TIMEOUT_MS`.
3. **Blocking `gethostbyname` + `connect` on the delaymgr heartbeat thread** — `endpoint_connect` (netio.c:289/302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unreachable peer stalls ALL heartbeats + election-timeouts for seconds. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)
**Root cause:** `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** on peer/vote endpoints — it runs inline on iomgr instead of dispatching to the vtask pool like normal cluster messages (`handle_msg` → `vtask_handle_msg`). So any work it does, including the heavy I/O above, executes on the event loop.
Plus MEDIUM vtask-pool blockers (pool-exhaustion, not drive-thread): config-ack wait (config_repl.c:1459), a 30-second `usleep` checkpoint poll (config_repl.c:1661), `ha_persist_term` fsync on a vote grant (ha.c:3407 — adds election latency).
## Current State
- The corrected **thread-role contract** is now steering (`ha-async-io.md`, 2026-06-25): iomgr = non-blocking I/O + dispatch only; delaymgr = trigger-only, never blocks; vtask pool = bounded blocking only under admission reservation; long blocking I/O → dedicated slow lane. This spec brings the **code into compliance** with that contract. (The contract SUPERSEDED the old "delaymgr for blocking ops = Permitted" rule, which was the design flaw behind these findings — delaymgr is both the heartbeat driver and was the documented blocking sink.)
- Existing primitives to reuse: `COMMIT_WAIT_RESERVED` admission cap (commit.c:195-203) already keeps vtask-pool headroom against blocking waits; the non-blocking-connect + bounded-`poll` pattern already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); `vtask_handle_msg` (vtaskmgr.c:356) is the dispatch path but is endpoint+inbound-packet-bound (`vtype=VTASK_IO`).
- Confirmed already-correct: RequestVote fan-out, heartbeat fan-out, WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + fdatasync run on a vtask worker, not iomgr.
## Desired Outcome
The protocol drive is non-blocking on both liveness threads, by construction:
- **iomgr** does only non-blocking socket I/O + dispatch — `peer_ctrl_msg_handler`'s real work is dispatched to the vtask pool (conforming to the existing `handle_msg` pattern), so no future handler addition can reintroduce inline blocking.
- Inside the vtask, **fast protocol work runs inline** (decode ack → update match_index → advance commit → send next heartbeat — all bounded); **long I/O is triggered to a slow lane** (snapshot transfer, config-push), never run in the request-pool worker.
- **delaymgr** only fires timers and triggers work; the peer connect is **bounded** (non-blocking + `poll` timeout) and **offloaded** off the timer thread.
- Heartbeat liveness is provably protected: a partitioned (unreachable, not merely dead) peer cannot stall heartbeats to healthy peers or trigger a spurious election.
## Approach
Bring the code into compliance with the steering thread-role contract, structurally rather than per-op:
1. **Dispatch `peer_ctrl_msg_handler` to the vtask pool** (the structural fix). iomgr becomes epoll + enqueue only. Preserve per-connection message ordering (VTASK_IO per-endpoint serialization is a feature here). Marshal any iomgr-owned state mutations (epoll/endpoint lifecycle) back to the iomgr thread (bug-646 discipline).
2. **Two-tier execution inside the vtask:** fast bounded protocol work inline; **trigger** long I/O (snapshot send + `persist_checkpoint`, config-push) to a **dedicated slow-lane / admission-capped executor** so multi-second tasks cannot starve the request pool. Per-peer dedup (one snapshot in flight per standby); coordinate with ongoing WAL-catchup stream and the primary's appends/WAL-trim.
3. **delaymgr connect → bounded + offloaded:** reuse the `xce_probe_config_tag` non-blocking-connect + bounded-`poll` pattern; trigger the reconnect as an idempotent, refcounted, deduped task off the delaymgr thread; marshal the connected-fd epoll registration back to iomgr. DNS (`gethostbyname`) is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — defer async DNS unless measured.
4. **General (non-IO) task-submit path:** add the submit primitive needed for #2/#3 (current `vtask_handle_msg` requires an inbound packet); or generalize it.
5. **Admission discipline:** any offloaded blocking task lives under a reservation (extend the `COMMIT_WAIT_RESERVED` pattern) so liveness-critical messages always have a free worker.
## Scope
- **In**:
- Dispatch `peer_ctrl_msg_handler` real work to the vtask pool (root-cause structural fix).
- Snapshot send + `persist_checkpoint` off iomgr → slow lane, with per-peer dedup + WAL coordination.
- Config-push wait off iomgr → slow lane / async ACK.
- delaymgr peer connect → bounded non-blocking + offloaded; reconnect dedup/refcount/marshal-back.
- General non-IO vtask submit path + admission reservation for offloaded blocking tasks.
- Partition (unreachable-peer) regression test; verify no spurious election + heartbeats to survivors continue.
- **Out**:
- The commit-barrier async conversion (P1 / `repl_wait_majority_ack` → `resolve`) — owned by `raft-apply-decoupling` as its keystone (different concern: that's the *commit* drive, this is the *transport/timer* drive).
- Apply-after-commit / hook decoupling / facts-through-raft (raft-apply-decoupling).
- Async DNS (deferred unless measured).
- The MEDIUM vtask-pool blockers beyond what the admission discipline naturally covers (note them; fix opportunistically).
- Entity-bearing snapshots / config-upgrade snapshot (separate specs).
## Boundary Candidates
- The `peer_ctrl_msg_handler` → vtask dispatch seam.
- The dedicated slow-lane executor + admission reservation.
- The general non-IO vtask-submit primitive.
- The bounded-connect + offloaded-reconnect path.
- The iomgr-owned-state marshal-back mechanism (epoll/endpoint lifecycle).
## Out of Boundary
- Commit-engine internals (P1 owns the commit-drive async).
- Snapshot *content* format (entity-bearing snapshots is a separate spec) — this spec only moves the *transfer* off the drive thread.
## Upstream / Downstream
- **Upstream**: the `ha-async-io.md` thread-role contract (authority); existing `COMMIT_WAIT_RESERVED` (commit.c) and `xce_probe_config_tag` bounded-connect pattern (reuse); `election-iomgr-refactor` (COMPLETED upstream — provides reusable primitives `delay_task_ms`, `on_connected`, `ha_start_election_async`/non-blocking vote fan-out).
- **Downstream**: `raft-apply-decoupling` (its async commit drive + apply pump reuse the slow-lane + admission infrastructure this spec establishes; and the drive must be non-blocking before apply is restructured on top of it); safe failover for the whole bug-851 family depends on this landing.
## Existing Spec Touchpoints
- **Adjacent (RESOLVED standalone 2026-06-25)**: `election-iomgr-refactor` is ADJACENT-SAME-AREA — it fixed the *startup* blocking path and is fully implemented; this spec fixes the *runtime* drive-thread paths it never touched. Keep separate; consume it as upstream. (Scope-check verdict in research.md.)
- **Coordinate**: `ha-term-safety` (touches the same `peer_ctrl_msg_handler` / repl dispatch region); `ha-wal-replication` (snapshot/catch-up transport); `controller-status` (heartbeat/committed propagation on the same drive).
- **Verification home**: `raft-sim-conformance` (add the partition/liveness assertion) and `/xce-test-ha` (partition variant).
## Constraints
- The corrected thread-role contract (`ha-async-io.md`) is authoritative — iomgr and delaymgr never block inline; long I/O on the slow lane; bounded blocking in vtask only under admission reservation.
- "Controllers never self-terminate" — an unreachable peer is a loud retry, never an abort.
- Preserve per-connection message ordering when dispatching `peer_ctrl_msg_handler` to vtask.
- Must not regress the heartbeat fast path measurably at cluster scale (3–5 peers); per-message vtask overhead accepted in exchange for uniformity + liveness safety.
- Verified by a **partition** (unreachable, not killed) soak — the case `/xce-test-ha`'s current kill-only soak does not reproduce.
## Provenance
Surfaced 2026-06-25 during the `raft-apply-decoupling` discovery as prerequisite **P2** (drive hardening); promoted to its own spec because it threatens liveness independently of the apply refactor. Full analysis: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § "Two distinct prerequisites" + P2 offload mechanism. Thread-role contract: `.kiro/steering/ha-async-io.md` (corrected 2026-06-25).

385
.kiro/specs/raft-drive-async/design.md Normal file
View file

@ -0,0 +1,385 @@
# Design Document — raft-drive-async
## Overview
**Purpose**: Make the XCE controller's Raft protocol **drive** fully event-driven by removing every blocking operation from the two threads that carry cluster liveness — the iomgr epoll loop and the delaymgr timer — so that a slow/large transfer or an unreachable peer can no longer stall heartbeats and trigger a spurious election.
**Users**: Operators of multi-node controller clusters, who today see false failover under partition or large-snapshot conditions (the likely root of the bug-851 chaotic-failover family).
**Impact**: Changes *how* drive-thread work is scheduled, not *what* the protocol does. `peer_ctrl_msg_handler`'s real work moves from inline-on-iomgr to the vtask pool; long blocking I/O (snapshot transfer, config-push, peer connect) moves to a new dedicated **slow-lane executor**; the delaymgr peer connect becomes bounded and offloaded. All existing consensus semantics, durability (persist-before-ACK), and the synchronous commit barrier are preserved unchanged.
### Goals
- No blocking operation executes inline on the iomgr loop or the delaymgr timer (R1, R2, R5).
- Long blocking I/O runs on a bounded, admission-controlled lane isolated from the request vtask pool (R3).
- Per-connection processing order and consensus correctness preserved when peer control moves to the pool (R4).
- A network-partition (unreachable-peer) regression test proves heartbeat liveness and no spurious election (R7).
- No regression of existing HA features; steady-state heartbeat path unchanged (R6, R8).
### Non-Goals
- Commit-barrier async (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` (P1).
- Apply-after-commit / hook decoupling / facts-through-raft.
- Asynchronous DNS resolution — deferred (HPC peers use IP/`/etc/hosts`); only `connect` is bounded here.
- Snapshot *content*/format (entity-bearing snapshots) — this spec moves the *transfer* off the drive threads only.
- The MEDIUM vtask-pool blockers (config-ack wait, 30 s checkpoint poll, term fsync on vote) beyond what the lane and existing `COMMIT_WAIT_RESERVED` already cover.
- The admin `OP_upgradeConfig` push path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459) — it has a synchronous requester and stays a vtask-blocking operation under `COMMIT_WAIT_RESERVED`; only the requester-less iomgr transport push is offloaded.
## Boundary Commitments
### This Spec Owns
- The scheduling discipline of drive-thread work: dispatching `peer_ctrl_msg_handler` to the vtask pool; the two-tier rule (fast protocol inline in the vtask, long I/O triggered to the slow lane).
- A new **slow-lane executor** component: a bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight deduplication.
- The bounded, offloaded peer-connect path (reorder of `socket_connect` + offloaded reconnect).
- The network-partition regression scenario in the soak harness.
### Out of Boundary
- The commit engine and `repl_wait_majority_ack` (untouched; P1 owns its async conversion).
- WAL/replication protocol content, snapshot format, config-upgrade materialization.
- Election/vote protocol logic (consumed as-is; `election-iomgr-refactor` already made vote fan-out async).
- The durability guarantee and synchronous commit barrier (preserved, not modified).
### Allowed Dependencies
- `ha-async-io.md` thread-role contract (authority).
- Existing primitives reused, not modified in contract: `endpoint_block_tasks`/`endpoint_unblock_tasks` (per-endpoint serialization), lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (thread-safe epoll registration), `vtask_handle_msg`/VTASK_IO dispatch, `delay_task_ms` (non-blocking trigger only), `COMMIT_WAIT_RESERVED` (admission pattern), the `xce_probe_config_tag` bounded-connect pattern, the `fsync_worker`/`push_thread` background-thread precedent.
- `election-iomgr-refactor` (completed upstream): `delay_task_ms`, `on_connected`, non-blocking vote fan-out.
### Revalidation Triggers
- Any change to the slow-lane submit contract or its admission cap.
- Any change that makes a drive thread (iomgr/delaymgr) execute work synchronously again.
- Changes to per-connection ordering semantics of peer-control dispatch (affects consensus correctness assumers: `ha-term-safety`, `raft-apply-decoupling`).
- A change to `socket_connect`'s blocking/timeout behavior (affects all peer connection callers).
## Architecture
### Existing Architecture Analysis
Three drive threads exist (`ha-async-io.md`): **iomgr** (single epoll loop), **vtaskmgr** (N-worker pool, default 8), **delaymgr** (single timer). The corrected thread-role contract (2026-06-25) requires iomgr and delaymgr to never block inline. Current violations:
- `peer_ctrl_msg_handler` is registered directly as `ep->msg_handler` (ha.c:4265/4307/6349) → runs inline on iomgr; it triggers snapshot send (ha.c:2702 → config_repl.c:2446) and config-push (config_repl.c:1332) synchronously on iomgr.
- `socket_connect` (netio.c:289/302) blocks (sets `O_NONBLOCK` only *after* connect, netio.c:716); reached from `primary_heartbeat_fn` on the delaymgr thread → an unreachable peer stalls all heartbeats.
- The codebase's only deferral idiom, `delay_task_ms`, runs the deferred callback *on the delaymgr thread* — a liveness thread — so it cannot host blocking work under the new contract.
### Architecture Pattern & Boundary Map
Selected pattern: **dispatch-and-offload** — iomgr dispatches, the vtask pool does bounded protocol work, a new dedicated slow lane absorbs long blocking I/O; completions re-enter iomgr only through the existing lock-guarded registration call.
```mermaid
graph TB
subgraph drive[Drive threads - never block]
iomgr[iomgr epoll loop]
delaymgr[delaymgr timer]
end
subgraph pool[vtask pool - bounded work]
vtask[peer_ctrl vtask entry]
end
subgraph lane[slow-lane executor - dedicated bounded threads]
slow[blocking IO worker]
end
iomgr -->|dispatch VTASK_IO packet| vtask
vtask -->|fast protocol inline| vtask
vtask -->|trigger long IO| slow
delaymgr -->|trigger reconnect| slow
slow -->|blocking connect snapshot configpush| slow
slow -->|register connected fd via lock-guarded iomgr_monitor| iomgr
```
**Architecture Integration**:
- Domain boundaries: iomgr = I/O + dispatch; vtask = bounded protocol handling (serialized per endpoint by `endpoint_block_tasks`); slow lane = long blocking I/O only; delaymgr = trigger-only.
- Existing patterns preserved: VTASK_IO per-endpoint serialization, lock-guarded epoll registration, `delay_task_ms` as a non-blocking trigger, `COMMIT_WAIT_RESERVED` admission style.
- New component rationale: the slow lane is the missing *non-liveness execution lane* — the one genuine gap; using dedicated threads (not the vtask pool) keeps long ops from starving request handling (R3 by construction).
- Dependency direction: `netio` (lib) → `slowlane` → `ha`/`config_repl` (controller). Drive threads depend on the lane; the lane depends only on netio + libc.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | C (gnu17), pthreads | slow-lane executor; drive-thread dispatch changes | matches controller stack |
| Messaging / Events | existing iomgr/vtaskmgr/delaymgr + new slow lane | thread-role-compliant scheduling | no new external dep |
| Infrastructure / Runtime | epoll, `poll(2)`, pthread mutex/cond/sem | bounded connect, lane workers, admission | reuse existing patterns |
| Test | soak harness + firewall DROP (iptables/nft) | partition (unreachable-peer) scenario | capability-gated; see Testing |
## File Structure Plan
### New Files
```
src/controller/
├── slowlane.c # Bounded slow-lane executor: submit, worker loop, admission cap, per-key dedup
└── include/slowlane.h # slow-lane public API (submit, init, fini)
```
### Modified Files
- `src/lib/vtaskmgr.c` / `src/include/vtaskmgr.h` — **LivenessAdmission**: add `vtask_class_t` + a classified submit (`vtask_enq_classified`); dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus, fast-rejecting `VTASK_REQ_BLOCK` at the cap. Shared lib primitive → Revalidation Trigger.
- `src/controller/commit.c` — fold `inflight_waits`/`COMMIT_WAIT_RESERVED` into the pool reservation: `commit_submit` submits a `VTASK_REQ_BLOCK` instead of self-counting.
- `src/lib/netio.c` — `socket_connect`: set `O_NONBLOCK` before `connect`, bound with `poll(timeout)` (reuse `xce_probe_config_tag` pattern). Add a connect-timeout parameter/config.
- `src/controller/ha.c` — register peer endpoints to dispatch peer-control through the vtask pool (revise `ha.c:4265/4307/6349`); convert `peer_ctrl_msg_handler` into a vtask entry that does fast protocol work inline and `slowlane_submit`s long ops; offload snapshot send (ha.c:2702) and reconnect (`primary_heartbeat_fn`/`ha_connect_to_standby`) to the slow lane with per-peer dedup.
- `src/controller/config_repl.c` — **only the iomgr/promotion path**: run `push_config_transport_only` on the slow lane and replace the `wait_all_with_timeout` drive-thread join (reached from `peer_ctrl`/promotion at ha.c:1172/1177/2672/2677/3973) with a slow-lane-internal wait + non-blocking return. **In scope = the fire-and-forget transport push with NO waiting requester.** The admin `OP_upgradeConfig` path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459, "VTASK-CONTEXT ONLY") is **OUT of scope** — it has a synchronous admin requester awaiting the version/ACK reply and is a MEDIUM vtask-pool blocker bounded by `COMMIT_WAIT_RESERVED`; it must NOT be offloaded (doing so would break the synchronous admin reply).
- `src/controller/cluster.c` (or `myapp.c`) — init/fini the slow-lane executor in the controller lifecycle; pass connect-timeout config.
- `src/include/config.h` / `ha.h` — connect-timeout + slow-lane size/cap constants.
- `.claude/skills/xce-test-ha/run-ha-soak.sh` (+ `xce_test`) — partition (DROP) primitive and the partition scenario.
> Each file keeps one responsibility; `slowlane.c` is the only new logic module. peer-control logic stays in `ha.c` (moved to a vtask entry, not relocated to a new file) to preserve reviewability against existing tests.
## System Flows
### Reconnect to an unreachable peer (R2)
```mermaid
sequenceDiagram
participant DM as delaymgr timer
participant SL as slow lane
participant IO as iomgr
DM->>DM: heartbeat tick - peer disconnected
DM->>SL: slowlane_submit reconnect peer if not in-flight
DM-->>DM: return immediately - other peers still heartbeat
SL->>SL: nonblocking connect + poll bounded timeout
alt connect succeeds
SL->>SL: prepare endpoint set msg_handler
SL->>IO: iomgr_monitor lock-guarded register fd
else timeout or refused
SL->>SL: clear in-flight - retry next tick
end
```
Key decisions: delaymgr only triggers (idempotent per-peer flag); the connect is bounded so it never approaches the heartbeat timeout; registration reuses the lock-guarded `iomgr_monitor` (no new primitive).
### Snapshot send triggered by a catch-up request (R1)
```mermaid
sequenceDiagram
participant IO as iomgr
participant VT as peer_ctrl vtask
participant SL as slow lane
IO->>VT: dispatch repl-ack packet VTASK_IO
VT->>VT: fast - decode, update match_index
VT->>SL: slowlane_submit snapshot if not in-flight per standby
VT-->>IO: handler returns - endpoint unblocked, peer messages flow
SL->>SL: persist_checkpoint + fread chunk loop + send
```
Key decisions: the vtask must **not** hold `ep->busy` across the transfer (R4.3) — it triggers and returns; per-standby dedup prevents concurrent transfers (R1.5).
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1–1.4 | liveness during snapshot/config I/O | SlowLane, PeerCtrlDispatch | Snapshot flow |
| 1.5 | no concurrent snapshot per standby | SlowLane (dedup) | Snapshot flow |
| 2.1–2.2 | unreachable peer doesn't stall heartbeats | BoundedConnect, ReconnectOffload | Reconnect flow |
| 2.3 | bounded connect timeout | BoundedConnect | — |
| 2.4–2.5 | retry without self-terminate; dedup | ReconnectOffload, SlowLane | Reconnect flow |
| 2.6 | connect success registers without loss/reorder | ReconnectOffload (iomgr_monitor) | Reconnect flow |
| 3.1–3.4 | admission protection; long IO off pool/drive | SlowLane (cap), LivenessAdmission (pool reservation), PeerCtrlDispatch | — |
| 4.1–4.2 | per-connection processing order | PeerCtrlDispatch (endpoint_block_tasks) | — |
| 4.3 | don't hold block across long IO | PeerCtrlDispatch | Snapshot flow |
| 4.4 | iomgr-state mutation via lock-guarded iface | ReconnectOffload (iomgr_monitor) | Reconnect flow |
| 4.5–4.6 | preserve vote/ack/term + persist-before-vote | PeerCtrlDispatch, LivenessAdmission (ack-progress reservation) | — |
| 5.1–5.4 | thread-role contract compliance | all | — |
| 6.1–6.7 | no regression of existing HA features | all (regression suite) | — |
| 7.1–7.4 | partition verification | PartitionTest | — |
| 8.1–8.2 | heartbeat fast-path unchanged | PeerCtrlDispatch | — |
## Components and Interfaces
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| SlowLane | controller/infra | bounded executor for long blocking I/O | 1, 3 | pthreads, sem (P0) | Service, Batch |
| LivenessAdmission | lib/vtaskmgr | pool-level task-class reservation: liveness work always has a worker | 3, 4.5 | vtaskmgr dispatcher (P0) | Service |
| PeerCtrlDispatch | controller/ha | move peer-control to vtask; two-tier | 1,4,5,8 | vtaskmgr, LivenessAdmission, endpoint_block_tasks (P0) | Service |
| BoundedConnect | lib/netio | non-blocking + poll-bounded connect | 2 | poll (P0) | Service |
| ReconnectOffload | controller/ha | delaymgr triggers offloaded reconnect | 2 | SlowLane, iomgr_monitor (P0) | Service |
| PartitionTest | test | unreachable-peer regression | 7 | soak harness (P1) | Batch |
### Controller / Infrastructure
#### SlowLane
| Field | Detail |
|-------|--------|
| Intent | Bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight dedup |
| Requirements | 1.1, 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4 |
**Responsibilities & Constraints**
- Owns a fixed set of dedicated worker threads (sized to peer count; small constant) — isolated from the vtask request pool, so long ops cannot starve request handling.
- `slowlane_submit` is non-blocking (enqueue + return); callers on drive threads never wait.
- Enforces an admission cap on concurrent in-flight tasks; excess is queued, never run on a drive thread (R3.3/3.4).
- Per-key dedup: a submit with an in-flight key is a no-op (R1.5 snapshot-per-standby; R2.5 reconnect-per-peer).
- Does not own the work content (connect, snapshot, config-push bodies stay in their modules); it owns scheduling/isolation.
**Dependencies**
- Inbound: PeerCtrlDispatch — submits long ops (P0); ReconnectOffload — submits reconnect (P0).
- Outbound: `iomgr_monitor` (via the submitted task) — register connected fd (P0).
- External: pthreads (threads, mutex, cond, sem) (P0).
**Contracts**: Service [x] / Batch [x]
##### Service Interface
```c
typedef void (*slowlane_fn_t)(void *arg);
int slowlane_init(slowlane_t *sl, int n_workers, int max_inflight);
/* submit returns 0 (enqueued), -EEXIST (key in flight, deduped), -EAGAIN (cap full, queued/deferred);
never blocks the caller. key==0 disables dedup. */
int slowlane_submit(slowlane_t *sl, uint64_t key, slowlane_fn_t fn, void *arg);
void slowlane_fini(slowlane_t *sl); /* drain + join on shutdown only */
```
- Preconditions: `fn` performs the blocking work and is responsible for clearing any caller-side in-flight state on completion.
- Postconditions: `fn` runs exactly once on a lane worker; never on iomgr/delaymgr.
- Invariants: concurrent running tasks ≤ `max_inflight`; one in-flight task per non-zero key.
**Snapshot/checkpoint concurrency invariant (Phase 3, R1.1/1.3/1.5 — MUST hold before snapshot offload lands)**
Moving `persist_checkpoint` (which runs `trim_wal_segments` — *deletes* WAL segments) + the snapshot `fread` loop onto a lane worker runs them concurrently with live WAL appends, the `fsync_worker` flusher, and the WAL-catchup stream. The lane snapshot path must obey:
- **Frozen boundary**: the snapshot is taken at a fixed `(term, lsn)` checkpoint boundary captured at submit time; the `fread`/send loop reads only data at or below that boundary, never live tail bytes.
- **Trim safety (MECHANISM, not just goal — see bug-1147)**: `trim_wal_segments` (`persist.c:1374`) TODAY bounds deletion ONLY by `active_seg_lsn` (the open segment) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** snapshot boundary, and there is **no** working snapshot backfill (`write_snapshot` emits only a 16-byte P0 sentinel, not materialized state). So a standby lagging by more than ~one open segment already hits a permanent committed-prefix gap (the bug-795/828 family). Moving checkpoint/trim onto a concurrent lane worker makes this latent race *easier* to hit. Therefore this spec MUST add the cursor-bound guard as a stopgap: `trim_wal_segments` computes `floor = min(all standby acked_lsn, all in-flight snapshot boundaries)` under `p->lock` and refuses to delete any segment whose range reaches `floor` or above — never holding `p->lock` across the `fread`/send I/O. This is strategy-A retention discipline that *should* exist regardless of this spec; the full fix for empty/far-behind join (cursor=0) requires entity-bearing snapshots (separate spec) and is explicitly NOT solved here. The design must not assume "a snapshot will backfill the gap" — that path is not built.
- **Single in-flight checkpoint**: at most one checkpoint/trim in flight (lane dedup key = checkpoint), and at most one snapshot per standby (dedup key = standby id, R1.5).
- **No correctness change**: the lane only relocates execution; the checkpoint/trim/snapshot *content* is byte-identical to the current inline path.
**Implementation Notes**
- Integration: init in controller lifecycle (cluster.c/myapp.c); size from peer count.
- Validation: assert submit never called from a context expecting completion; metric on queue depth/age (fail-loud on backlog); a test asserting concurrent append + checkpoint/trim produces no segment-below-boundary deletion and a consistent snapshot.
- Risks: a lane worker blocked indefinitely (e.g., snapshot to a half-open peer) — **bound every in-lane blocking op**: the connect uses the bounded `socket_connect`; the snapshot send must use a bounded socket write (write timeout, mirroring the bounded-connect approach) so a standby going unreachable mid-transfer cannot pin a lane worker. Otherwise stuck transfers exhaust the bounded lane and delay reconnects (which share it). Phase-3 task constraint; size/segment the lane so reconnect and transfer do not starve each other.
#### LivenessAdmission (lib/vtaskmgr)
| Field | Detail |
|-------|--------|
| Intent | Make "keep K workers free for liveness" a property of the vtask **pool**, not of the commit engine — so it covers every blocking op the pool runs once peer-control moves onto it, not just commit-waits |
| Requirements | 3.1, 3.2, 3.3, 4.5 |
**Why this exists (the gap it closes):** Today `vtaskmgr` (`include/vtaskmgr.h`) has a single FIFO `taskq`, N `vcpu` workers, and **no priority/reservation**; a blocking task holds its `vcpu` in `VCPU_BUSY` for the whole block (`vtaskmgr.c:248-275`). The only admission anywhere is `commit.c`'s `inflight_waits` counter (`COMMIT_WAIT_RESERVED=2`, `commit.c:200`) — external to vtaskmgr and counting **only** commit-waits. Once `peer_ctrl` moves onto the pool (PeerCtrlDispatch), the pool also runs the vote-grant term fsync (R4.6) and config-ack waits — blocking ops **invisible** to the commit counter. So "≥2 workers free of commit-waits" no longer means "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. The reservation must therefore become pool-level.
**Responsibilities & Constraints**
- Classify every vtask at enqueue into `VTASK_LIVENESS` (never parks — WAL-ack decode, heartbeat send, vote *receipt/decode*; `commit_on_ack` is documented never-block, `commit.c:16`), `VTASK_LIVE_BLOCK` (bounded mandatory park — vote-grant term fsync, R4.6), or `VTASK_REQ_BLOCK` (may park seconds — commit-wait, config-ack wait). Class is knowable from the message type at the dispatch shim.
- The dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus: a `VTASK_REQ_BLOCK` task is **not admitted** when `count(vcpus running *_BLOCK) >= n_vcpus - VTASK_LIVENESS_RESERVED` — it **fast-rejects with `-XCE_ERR_NOT_READY`** (preserving today's client backpressure, Req 7.1/7.2), it does **not** silently queue or park a worker.
- `VTASK_LIVENESS` tasks are **never** admission-gated → always find a reserved vcpu → acks always advance commit → parked `*_BLOCK` tasks always wake. **Deadlock closed by construction.**
- `VTASK_LIVE_BLOCK` (vote fsync) runs in the reserved headroom and is bounded, so a vote response never queues behind a seconds-long commit-wait (removes a second spurious-election vector).
- `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` is **folded into** this — `commit_submit` becomes "submit a `VTASK_REQ_BLOCK`"; the pool owns the single reservation (`commit_engine_init` no longer needs `n_vcpus`, `cluster.c:1332`).
**Dependencies**: Inbound — PeerCtrlDispatch, commit (P0). Outbound — none (lib-internal). External — pthreads.
**Contracts**: Service [x]
```c
typedef enum { VTASK_LIVENESS=0, VTASK_LIVE_BLOCK, VTASK_REQ_BLOCK } vtask_class_t;
/* submit a classified task; VTASK_REQ_BLOCK returns -XCE_ERR_NOT_READY when the
reservation would be violated (retriable, never parks a worker). */
int vtask_enq_classified(vtaskmgr_t*, vtask_t*, vtask_class_t);
```
- Invariant: `count(vcpus in *_BLOCK) <= n_vcpus - VTASK_LIVENESS_RESERVED`; ≥`VTASK_LIVENESS_RESERVED` vcpus always available to `VTASK_LIVENESS`.
**Implementation Notes**
- **Sizing (peer-scaled):** `R = VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)`; commit/`REQ_BLOCK` cap = `n_vcpus − R`. Rationale: per-endpoint serialization (`endpoint_block_tasks`) caps in-flight tasks at one per peer connection, so worst-case concurrent liveness work is **N−1** (one ack/vote per other peer) — reserving N−1 means no peer's liveness message ever queues behind a busy reserved worker. **Correctness needs only the floor of 1** (one guaranteed-free worker + liveness-never-gated + sub-ms cycling = deadlock-free regardless of peer count); the peer-scaled middle is a *latency* optimization. **Ceiling `n_vcpus − 1`** keeps ≥1 `REQ_BLOCK` slot so commits can still run (cap never 0). Both `n_peers` and `n_vcpus` are final at init (`cluster.c:1332` already passes both). Config-overridable. On a degenerate small-core box (`n_vcpus < n_peers`) R clamps to `n_vcpus − 1` (full per-peer reservation impossible) — still correct via fast cycling; emit a loud WARN at init so the under-provisioning is operator-visible. Validate the chosen `R` at supported sizes (R8).
- Validation: misclassification is the one failure mode — add a debug assert/loud-WARN if a `VTASK_LIVENESS` task parks beyond a threshold; task 3.4 (acks advance commit under saturation) is the regression oracle.
- Risk: a future blocking path tagged `VTASK_LIVENESS` silently breaks the guarantee — review discipline + the assert + the saturation test.
- **Revalidation Trigger**: any change to the class set, the reservation size, or the dispatch admission rule (consensus correctness + the slow-lane admission contract depend on it).
#### PeerCtrlDispatch
| Field | Detail |
|-------|--------|
| Intent | Dispatch peer-control message handling to the vtask pool; fast work inline, long work to the lane |
| Requirements | 1.4, 4.1, 4.2, 4.3, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2 |
**Responsibilities & Constraints**
- Peer endpoints dispatch through the existing VTASK_IO path (the message is packet-triggered → reuses `vtask_handle_msg`), inheriting `endpoint_block_tasks` per-endpoint serialization (R4.1/4.2).
- The vtask entry performs only bounded protocol work inline (decode, match_index update, commit eval, send next heartbeat) and `slowlane_submit`s any long op; it must not hold `ep->busy` across long I/O (R4.3).
- Preserves existing vote/ack/term semantics and persist-before-vote ordering (R4.5/4.6) — the logic is unchanged, only its execution context moves.
- **Commit-advance preservation (R4.5):** WAL-ack handling — which wakes the commit barrier (`repl_wait_majority_ack` / `commit_set_raft_commit_index`) — moves off iomgr to the vtask pool. The design preserves that a quorum of acks still advances `committed_lsn` promptly: ack handling is bounded fast work (never offloaded to the slow lane), per-endpoint serialization does not delay *cross-endpoint* acks (each peer's acks process independently), and the commit-advance path itself is untouched. This is the only interaction with P1 (`raft-apply-decoupling`, which owns the barrier's async conversion); this spec must not change ack→commit-advance semantics, only the thread the ack is decoded on.
- **Deadlock-freedom argument (R4.5/R3.1 — REQUIRED reasoning, reverses a documented invariant):** Today `commit_on_ack` is explicitly "called from the iomgr read path and never blocks" (`commit.c:16`, `repl.c:220`) — the ack that advances `committed_lsn` and the jobsub vtasks blocked in `commit_submit(COMMIT_MODE_WAIT)` (the synchronous barrier this spec PRESERVES) run on **different** threads. Moving peer-control to the vtask pool **co-locates the wakers (acks) and the waiters (blocked commit-submits) on one pool** — exactly the configuration the iomgr placement was chosen to avoid. Safety rests on a worker reservation — but the existing `commit.c` cap is **insufficient as-is**: `COMMIT_WAIT_RESERVED` (`commit.c:200`) counts **only** commit-waits, while moving `peer_ctrl` onto the pool also puts the vote-grant term fsync (R4.6, `VTASK_LIVE_BLOCK`) and config-ack waits on it — blocking ops the commit counter can't see. "≥2 workers free of commit-waits" therefore does **not** mean "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. **Closing the deadlock requires the LivenessAdmission component** (above) — the reservation must become a pool-level property covering every blocking class, with `VTASK_LIVENESS` (acks) never admission-gated. Required invariants this spec must hold (and test): (1) WAL-ack/commit-advance decode is tagged `VTASK_LIVENESS`, never routed to the slow lane, never gated — admissible independently of any blocking count; (2) the reserved headroom is sized for all concurrent liveness work (N-peer acks + vote responses + heartbeat sends), so the reservation — not luck — guarantees an ack always finds a worker. This co-location is **transient by design**: `raft-apply-decoupling` P1 converts `repl_wait_majority_ack` to event-driven (no parked thread per commit), removing the `VTASK_REQ_BLOCK` waiters from the pool entirely — but until P1 lands, correctness depends on LivenessAdmission holding. **Dependency: PeerCtrlDispatch (3.1) requires LivenessAdmission to land first (Phase 1).** Gate: task 3.4 (quorum-of-acks advances `committed_lsn` with the pool saturated by `*_BLOCK` tasks) is the **admission gate for 3.1** — 3.1 is not "done" until 3.4 is green.
**Dependencies**
- Inbound: iomgr — dispatches the packet (P0).
- Outbound: vtaskmgr (P0); SlowLane — long ops (P0); existing commit/repl/persist logic (P0).
**Contracts**: Service [x]
**Implementation Notes**
- Integration: change peer endpoint `msg_handler` registration (ha.c:4265/4307/6349) to the vtask-dispatching shim; convert `peer_ctrl_msg_handler` body into the vtask entry.
- Validation: existing election/heartbeat/repl tests must pass unchanged (R6); assert no long op runs inline in the entry.
- Risks: a heavy op accidentally left inline → per-peer head-of-line blocking (guard by review + a test asserting the handler returns promptly).
#### BoundedConnect (lib/netio)
| Field | Detail |
|-------|--------|
| Intent | Make `socket_connect` non-blocking + `poll`-bounded |
| Requirements | 2.3 |
**Responsibilities & Constraints**: set `O_NONBLOCK` before `connect`; on `EINPROGRESS`, `poll` up to a configured timeout; return success/timeout/error. Reuses the `xce_probe_config_tag` pattern.
**Contracts**: Service [x]
```c
/* connect_timeout_ms: bound the attempt; 0 = use default (config). */
int socket_connect(const char *host, int port, int connect_timeout_ms);
```
**Implementation Notes**: callers updated to pass the timeout; default in config.h. This is a behavior change to a shared helper → Revalidation Trigger for all connect callers.
#### ReconnectOffload (controller/ha)
| Field | Detail |
|-------|--------|
| Intent | delaymgr triggers an offloaded, deduped, bounded reconnect |
| Requirements | 2.1, 2.2, 2.4, 2.5, 2.6 |
**Responsibilities & Constraints**: `primary_heartbeat_fn` (delaymgr) detects a disconnected peer and `slowlane_submit`s the reconnect (per-peer key for dedup); the lane worker does the bounded connect, prepares the endpoint, registers it via lock-guarded `iomgr_monitor`; on failure clears the in-flight flag and lets the next tick retry (never self-terminates).
**Contracts**: Service [x]
**Implementation Notes**: refcount the endpoint across the in-flight window; resolve the inbound-vs-outbound connection race per the existing connection-ownership rules (`ha-async-io.md`).
### Test
#### PartitionTest
| Field | Detail |
|-------|--------|
| Intent | Make one peer unreachable (blackhole), assert liveness + no spurious election + heal |
| Requirements | 7.1, 7.2, 7.3, 7.4 |
**Contracts**: Batch [x]
**Implementation Notes**: the partition primitive must render one peer **unreachable** (traffic dropped, connect hangs), distinct from `kill -9`. Two mechanisms, **primary = in-process** so R7 actually runs in this project's environment (the sibling specs `ha-term-safety` 5.2/5.4 are already perpetually ENV-BLOCKED on the unprivileged/overlayfs sandbox; a firewall-only R7 would be a permanent skip and the spec's headline guarantee would never be exercised):
- **Primary (portable, no privilege): in-process socket-drop hook in `xce_test`** — a test-only seam (`XCE_TESTING`) that makes one node silently drop/blackhole inbound+outbound peer traffic at the endpoint layer (e.g. stop reading/writing the peer fd, or a drop flag in the I/O path), simulating "unreachable" without any kernel/firewall change. Runs anywhere the unit/soak suite runs.
- **Secondary (infra-level, optional): firewall `iptables`/`nft` DROP** on the node's port for real-network fidelity; capability-gated with a **loud skip** when unprivileged. Complements, does not replace, the in-process path.
Assert (both): heartbeats to survivors continue within the interval, no new-leader election while the leader holds majority, peer rejoins on heal, zero committed-job loss.
## Error Handling
### Error Strategy
- **Connect timeout/refused** → bounded `poll` returns; lane clears in-flight; delaymgr retries next tick; loud WARN with peer + elapsed (fail-loud, never abort — controllers never self-terminate).
- **Lane backlog** → submit returns `-EAGAIN` when the cap is reached; the work is queued, never run on a drive thread; emit a metric/WARN on queue depth + oldest-age so a saturated lane is operator-visible.
- **Snapshot/config-push failure on the lane** → reported through the existing repl/config error paths unchanged; lane only relocates execution, not error semantics.
- **Dedup race** → submit with an in-flight key is a no-op (`-EEXIST`); idempotent.
### Monitoring
- Metrics: lane queue depth, oldest-task age, in-flight count, per-peer reconnect attempts. WARN thresholds make a degraded/partitioned window visible (consistent with fail-loud).
## Testing Strategy
### Unit Tests
- SlowLane: `slowlane_submit` is non-blocking; admission cap bounds concurrent tasks; per-key dedup returns `-EEXIST`; `fn` runs exactly once (3.1–3.4, 1.5).
- BoundedConnect: `socket_connect` returns within the timeout against an unreachable host; succeeds against a reachable one (2.3).
- PeerCtrlDispatch: the vtask entry returns promptly (no long op inline); per-endpoint serialization holds via `endpoint_block_tasks` (4.1–4.3).
### Integration Tests
- Reconnect offload: with one peer unreachable, delaymgr ticks continue and heartbeats to reachable peers are sent within the interval; reconnect is deduped per peer (2.1, 2.2, 2.5).
- Snapshot offload: a catch-up triggering a snapshot does not block iomgr; other peers' heartbeats/votes processed during the transfer; no concurrent transfer per standby (1.1, 1.3, 1.4, 1.5).
- Commit-advance under pool load (R4.5): with the vtask pool near saturation, a quorum of WAL-acks still advances `committed_lsn` promptly; cross-endpoint acks are not serialized behind one another.
- Snapshot/checkpoint concurrency (R1.1/1.3/1.5): concurrent live appends + a lane checkpoint/trim never delete a segment below an in-flight snapshot boundary or a standby catch-up cursor; the produced snapshot is consistent.
- Regression sweep (R6): run the existing suites unchanged — `test_ha_election`, `test_ha_prevote`, `test_ha_liveness`, `test_ha_failover`, `test_ha_rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability`; all pass without behavior changes (6.1–6.7).
### E2E / Soak Tests
- Partition soak (R7): blackhole one peer → assert heartbeats to survivors continue, no spurious election while leader holds majority, peer rejoins on heal, zero committed-job loss (7.1–7.4).
- Failover soak (existing, kill-based) continues to pass (6.3).
### Performance
- Steady-state heartbeat cadence at 3–5 controllers shows no added latency attributable to the new dispatch path; the heartbeat/vote send paths remain non-blocking enqueues (8.1, 8.2).
## Migration Strategy
Phased, each phase independently testable (matches research recommendation):
**Ordering rule:** the slow lane is foundational — every offload (reconnect, snapshot, config-push) submits to it, so it must exist first. And the peer_ctrl→vtask move must land **together with** its long-op offload in the same phase: moving peer_ctrl to a vtask while still running snapshot/config-push inline would hold `endpoint_block_tasks` (`ep->busy`) across multi-second I/O (transient R4.3 violation) and consume a pool worker. The two are therefore coupled.
```mermaid
flowchart TD
P1[Phase 1: SlowLane executor + BoundedConnect] --> P2[Phase 2: ReconnectOffload sender-side]
P2 --> P3[Phase 3: PeerCtrlDispatch to vtask PLUS snapshot and configpush offload coupled]
P3 --> P4[Phase 4: PartitionTest + regression sweep]
```
- **Phase 1** (foundational, lowest risk): build the SlowLane executor (bounded pool + admission + dedup — new component, no behavior change yet, unit-testable in isolation); reorder `socket_connect` to bounded non-blocking + `poll` (independent; already bounds the delaymgr stall even before reconnect is offloaded); **and add LivenessAdmission to vtaskmgr** (task-class reservation; fold in `COMMIT_WAIT_RESERVED`). LivenessAdmission is a **hard prerequisite of Phase 3** — peer_ctrl cannot move onto the pool deadlock-safely without it — so it must land in Phase 1 as dormant infra (no behavior change until peer_ctrl dispatches through it).
- **Phase 2**: offload the delaymgr peer reconnect to the lane (deduped per peer, registered via lock-guarded `iomgr_monitor`) → closes sender-side heartbeat starvation. Shippable.
- **Phase 3** (coupled — the R4.3-safe step): dispatch peer-control to the vtask pool **and** offload the long ops it triggers (snapshot send, config-push transport-only) to the lane in the same change, so no phase ever runs long I/O inline in a blocked vtask. Carries the snapshot/checkpoint concurrency invariant (frozen `(term,lsn)` boundary, trim-vs-catch-up-cursor safety, single in-flight checkpoint) and the bounded in-lane snapshot write.
- **Phase 4**: partition test + full regression sweep.
- Rollback: each phase is behavior-preserving; a phase can be reverted independently since the contract (non-blocking drive) is the only invariant added. P1 adds dormant infra; the first observable change is P2.

111
.kiro/specs/raft-drive-async/requirements.md Normal file
View file

@ -0,0 +1,111 @@
# Requirements Document
## Introduction
The XCE controller's Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals — the iomgr epoll loop and the delaymgr timer thread. A blocking audit (2026-06-25) found three CRITICAL violations of the corrected `ha-async-io.md` thread-role contract: snapshot send + checkpoint inline on iomgr (`ha.c:2702`, `config_repl.c:2446`), config-push busy-poll + join inline on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr heartbeat thread (`netio.c:289/302`). Root cause: `peer_ctrl_msg_handler` is registered directly as the iomgr message handler (`ha.c:4265/4307/6349`) instead of dispatching to the vtask pool as normal cluster messages do (`cluster.c:711`). When a drive thread stalls, heartbeats/votes/ACKs stop flowing, peers declare the leader dead, and a spurious election follows — the likely root of the bug-851 chaotic-failover family.
This feature brings the controller into compliance with the steering thread-role contract (`ha-async-io.md`, corrected 2026-06-25) so that no liveness-carrying thread blocks inline, while preserving every existing HA behavior and the persist-before-ACK / synchronous-commit policies. It is verified by a new network-partition (peer unreachable, not merely killed) regression scenario that the current process-kill-only soak cannot reproduce.
The authoritative subject for the acceptance criteria below is **the XCE controller** and its named drive threads (iomgr loop, delaymgr timer, vtask pool).
## Boundary Context
- **In scope**: keeping the iomgr loop and delaymgr timer non-blocking; moving long replication/config I/O (snapshot transfer, config-push) off the drive threads to an admission-controlled execution path; bounding and offloading the peer connect; preserving per-connection message ordering and processing correctness; admission protection so liveness-critical message handling is never starved; a network-partition regression test; no regression of existing HA features.
- **Out of scope**: the commit-barrier async conversion (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` as prerequisite P1; the apply-after-commit / hook-decoupling refactor; asynchronous DNS resolution (deferred unless measured — HPC peers use IP/`/etc/hosts`); the snapshot content/format (entity-bearing snapshots is a separate spec) — this feature only moves the snapshot *transfer* off the drive threads; the MEDIUM vtask-pool blockers beyond what admission control covers.
- **Adjacent expectations**: relies on the corrected `ha-async-io.md` thread-role contract as the authority; reuses the existing `COMMIT_WAIT_RESERVED` admission cap and the `xce_probe_config_tag` bounded-connect pattern; consumes `election-iomgr-refactor` (completed) as upstream for its primitives (`delay_task_ms`, `on_connected`, non-blocking vote fan-out) and stays a standalone spec (scope-check 2026-06-25: ADJACENT-SAME-AREA — that spec fixed startup blocking, this fixes runtime drive-thread blocking); touches the same `peer_ctrl_msg_handler`/repl-dispatch region as `ha-term-safety`; preserves the durability and synchronous-commit guarantees owned by `ha-commit-durability` and `ha-commit-engine`.
## Requirements
### Requirement 1: Liveness During Long-Running Replication and Configuration I/O
**Objective:** As a cluster operator, I want the controller to keep heartbeating and processing protocol messages while it transfers a snapshot or pushes a configuration to a standby, so that a large or slow transfer to one peer does not trigger a false failover of the cluster.
#### Acceptance Criteria
1. While the controller is transferring a snapshot to one standby, the controller shall continue to send heartbeats to every other peer within the configured heartbeat interval (default 3000 ms).
2. While the controller is pushing a configuration to standbys, the controller shall continue to send heartbeats to every peer within the configured heartbeat interval and continue to process inbound votes and acknowledgments.
3. When the controller initiates a snapshot transfer, checkpoint creation, or configuration push, the iomgr loop shall remain available to dispatch other inbound messages without waiting for that operation to complete.
4. While a long-running replication or configuration operation is executing, the controller shall continue to process inbound RequestVote messages and respond within the configured election timeout (default 5000 ms).
5. If a snapshot transfer to one standby is already in progress, then the controller shall not start a second concurrent transfer to that same standby.
### Requirement 2: Liveness and Bounded Behavior When a Peer Is Unreachable
**Objective:** As a cluster operator, I want an unreachable peer to never stall heartbeats to the healthy peers, so that one partitioned node cannot cause the surviving majority to elect a new leader unnecessarily.
#### Acceptance Criteria
1. While one peer is unreachable (its host accepts no connection or silently drops traffic), the controller shall continue to send heartbeats to all reachable peers within the configured heartbeat interval.
2. While one peer is unreachable, the delaymgr timer thread shall continue to fire periodic and election-timeout callbacks without being delayed by the connection attempt to the unreachable peer.
3. When the controller attempts to connect to a peer, the controller shall bound the connection attempt to a configured timeout and report failure on expiry rather than waiting for the operating-system connect timeout.
4. If a connection attempt to a peer fails or times out, then the controller shall retry the connection later without terminating, consistent with the "controllers never self-terminate" invariant.
5. When the controller is already attempting to (re)connect to a given peer, the controller shall not enqueue a duplicate concurrent connection attempt for that same peer.
6. When a connection attempt to a peer succeeds, the controller shall register and begin using the new connection without losing or reordering messages already queued for that peer.
### Requirement 3: Worker-Pool Admission Protection
**Objective:** As a cluster operator, I want offloaded blocking work to never consume all worker capacity, so that liveness-critical message handling always has a worker available even under replication or reconnection load.
#### Acceptance Criteria
1. The controller shall reserve worker capacity such that liveness-critical message handling (heartbeats, votes, acknowledgments) always has at least one available worker even when blocking offloaded tasks are in flight.
2. While the maximum number of admitted blocking offloaded tasks is in flight, the controller shall continue to admit and process non-blocking liveness-critical messages.
3. When the number of concurrently requested blocking offloaded tasks exceeds the admission limit, the controller shall queue or defer the excess rather than block a liveness-carrying thread.
4. The controller shall route long/unbounded blocking I/O (snapshot transfer, configuration push, peer reconnect) to an execution path that is subject to the admission limit, not to the iomgr loop or the delaymgr timer thread.
### Requirement 4: Message Ordering and Processing Correctness Preserved
**Objective:** As a cluster operator, I want peer control messages to be processed with the same ordering and correctness as today after they are moved off the iomgr loop, so that consensus correctness is unaffected by the threading change.
#### Acceptance Criteria
1. When peer control messages arrive on a single connection, the controller shall handle (process/apply) them in arrival order, even though handling is dispatched to the worker pool rather than run inline on the iomgr loop. (This concerns inbound *processing* order, distinct from the already-guaranteed outbound *send* order of the endpoint output queue.)
2. While handling a peer control message for a given connection, the controller shall not begin handling another message from that same connection until the first completes, so per-connection processing remains strictly sequential.
3. While handling a peer control message, the controller shall not hold its connection's sequential-processing exclusivity across a long-running offloaded operation (snapshot transfer, configuration push); such operations shall be triggered and the handler shall return, so the connection's subsequent messages (including its heartbeat acknowledgments) are not queued behind the long operation.
4. When peer control message processing mutates iomgr-owned connection or epoll state, the controller shall do so only through the thread-safe (lock-guarded) registration interface, without data races against the iomgr event loop.
5. The controller shall preserve the existing semantics of vote handling, acknowledgment handling, and term updates after relocating peer control message processing off the iomgr loop.
6. The controller shall continue to persist term and vote state before responding to a vote, preserving the existing durability ordering on the vote path.
### Requirement 5: Thread-Role Contract Compliance
**Objective:** As a maintainer, I want the controller's drive threads to obey the steering thread-role contract by construction, so that future changes cannot reintroduce inline blocking on a liveness thread.
#### Acceptance Criteria
1. The iomgr loop shall perform only non-blocking I/O and message dispatch; it shall not execute blocking calls or inline message processing.
2. The delaymgr timer thread shall only fire timers and trigger work; it shall not execute blocking calls (connect, file I/O, or busy-wait sleeps).
3. The controller shall execute bounded blocking work in the worker pool only under the admission reservation, and long/unbounded blocking I/O only on the dedicated offload path.
4. The controller shall conform to the thread-role contract defined in `ha-async-io.md` for all drive-thread code paths brought into scope by this feature.
### Requirement 6: No Regression of Existing HA Features
**Objective:** As a cluster operator, I want all existing HA behaviors to keep working unchanged after the drive is made non-blocking, so that the hardening introduces no functional regression.
#### Acceptance Criteria
1. When the existing controller test suites for election/voting, heartbeat/liveness, failover, snapshot recovery, configuration push/upgrade, WAL replication/catch-up, and the commit barrier are run, the controller shall pass them without modification to their asserted behavior.
2. The controller shall continue to elect a single primary on quorum, increment terms, and reject stale-term votes exactly as the existing election tests assert.
3. The controller shall continue to detect a dead primary and fail over within the configured heartbeat timeout (default 10000 ms) exactly as the existing liveness/failover tests assert.
4. The controller shall continue to deliver snapshots and replicated WAL entries to standbys and recover them on restart exactly as the existing replication and recovery tests assert.
5. The controller shall continue to push and apply configuration changes through majority acknowledgment exactly as the existing config-replication tests assert.
6. The controller shall preserve the persist-before-ACK durability guarantee and the synchronous commit-barrier behavior, which this feature does not modify.
7. If any existing HA regression test fails after the change, then the feature shall be considered incomplete until the behavior is restored.
### Requirement 7: Network-Partition Verification
**Objective:** As a cluster operator, I want a regression test that makes a peer unreachable (not merely killed), so that the heartbeat-starvation failure mode this feature fixes is provably closed and stays closed.
#### Acceptance Criteria
1. The verification suite shall include a scenario that renders one peer unreachable by dropping its traffic (network partition / blackhole), distinct from process termination.
2. When one peer is made unreachable during the scenario, the verification shall assert that heartbeats to the surviving peers continue and that no spurious election of a new leader occurs while the existing leader remains alive and reachable to the majority.
3. When the partition is healed, the verification shall assert that the previously unreachable peer rejoins and catches up without operator intervention.
4. While the partition scenario runs, the verification shall assert that no committed job is lost, consistent with the existing failover soak guarantees.
### Requirement 8: Heartbeat Fast-Path Performance
**Objective:** As a cluster operator, I want the steady-state heartbeat and vote path to remain responsive after the threading change, so that making the drive non-blocking does not degrade normal operation at supported cluster sizes.
#### Acceptance Criteria
1. While the cluster is healthy and all peers are connected, the controller shall send heartbeats at the configured interval without added latency attributable to the new dispatch path at supported cluster sizes (3–5 controllers).
2. The controller shall not introduce additional blocking or unbounded queuing on the steady-state heartbeat and vote send paths, which are already non-blocking enqueues.

100
.kiro/specs/raft-drive-async/research.md Normal file
View file

@ -0,0 +1,100 @@
# Gap Analysis — raft-drive-async (2026-06-25)
Analyzes the gap between the 8 requirements and the existing controller code, to inform the design phase. Evidence from this session's blocking audits + targeted greps.
## Current State Investigation
### The existing deferral idiom and why it's insufficient
XCE already practices "don't block iomgr": the established pattern is `delay_task_ms(&c->delaymgr, ...)`, which `ordq_put`s a callback and returns immediately. Its own comments confirm the intent — "run off the iomgr walAck thread … delay_task_ms only enqueues" (`config_repl.c:2369`, `ha_convergence.c:207`, `config_repl.c:428`). **But the deferred callback then runs on the delaymgr thread — the heartbeat/election-timeout driver.** So the only existing "offload" target is itself a liveness thread. This is the conceptual heart of the gap: the codebase can *defer*, but its single deferral lane is the thread the corrected `ha-async-io.md` contract must keep free. **The missing capability is a non-liveness execution lane (a "slow lane").**
### Inline-on-iomgr root cause
`peer_ctrl_msg_handler` is assigned directly as `ep->msg_handler` at `ha.c:4265` (on_vote_connected), `ha.c:4307` (on_pre_vote_connected), `ha.c:6349` (standby accept) — so it runs on the iomgr loop. Normal inbound messages instead go `handle_msg` → `vtask_handle_msg` → vtask pool (`cluster.c:711`, registered `cluster.c:772`). The peer-control path is the anomaly that bypasses the pool.
### Reusable building blocks that already exist
- **vtask submit primitives**: `vtask_alloc` / `vtask_init` (vtaskmgr.c:21) / `vtask_enq` (vtaskmgr.c:342) exist and compose; the only public entry `vtask_handle_msg` (vtaskmgr.c:356) hardcodes `vtype=VTASK_IO` and *requires* `ep->fsm_in.pkt`. A general non-IO submit is a **small composition** of the existing primitives, not new infrastructure.
- **Per-endpoint serialization (explicit primitive)**: `endpoint_block_tasks()` / `endpoint_unblock_tasks()` (netio.c:896-911) set a per-endpoint `ep->busy` flag; the VTASK_IO path auto-blocks on task start (vtaskmgr.c:249), unblocks on completion (vtaskmgr.c:66), and the handler path uses it (xcectx.c:440/449), while `vtask_eligible` (vtaskmgr.c:203) skips a busy endpoint's other tasks. This **directly and explicitly satisfies R4.1** (per-connection processing order) when peer control reuses the IO path — a reuse, not new logic. Cross-peer isolation is preserved (block is per-`ep`, so peer X busy ≠ peer Y blocked). **Design trap:** the block is held for the whole task duration — so a long op (snapshot send) run *inline* in a blocked vtask would queue that peer's own subsequent heartbeat-ACKs behind it. This makes the two-tier rule (fast work in vtask, **trigger** long I/O to the slow lane) mandatory for R4.1 + R1 to both hold.
- **Admission cap pattern**: `COMMIT_WAIT_RESERVED=2` of `nvcpus` (default 8) caps blocking commit-waits at `max(1, n-2)` (commit.c:200, commit.h:17) — a reusable *pattern* for R3, but it is commit-engine-specific today.
- **Dedicated-background-thread precedent**: `fsync_worker` (persist.c:624) is a long-lived background thread; `push_config_transport_only` **already spawns per-standby `push_thread`s** (config_repl.c:1412) — then blocks by `pthread_join`/`wait_all_with_timeout` on the drive thread (config_repl.c:1332). So the *threads* exist; the defect is the **blocking join on the caller**, not absent concurrency.
- **Bounded-connect pattern**: `xce_probe_config_tag` already does `O_NONBLOCK`-before-connect + `poll(500ms)` (config_repl.c:1753-1761). The buggy path (`socket_connect`, netio.c:289/302) sets non-blocking *after* connect (netio.c:716) — so R2.3 is largely a **reorder + reuse**, not new code.
### Confirmed absent
- ~~No iomgr cross-thread wake primitive~~ — **CORRECTED 2026-06-25 (was overstated).** epoll registration is ALREADY thread-safe: `iomgr_monitor` (EPOLL_CTL_ADD), `iomgr_unmonitor` (DEL), `iomgr_update_poll` (MOD) all wrap `epoll_ctl` in `mylock_lock(&iomgr->lock)` (netio.c:1046-1082), and `epoll_ctl` is kernel-thread-safe regardless. The connect→register flow already runs from delaymgr/main threads today (e.g. `on_vote_connected`). So a worker can do the blocking connect off-thread, fully prepare the endpoint (set `ep->msg_handler` etc.) BEFORE it is monitored (an unmonitored fd yields no events → no race), then call the lock-guarded `iomgr_monitor`. **No new eventfd/iomgr-post primitive is needed; the marshal-back is "reuse `iomgr_monitor`."** Residual (design-verify, not a primitive): ensure worker-side handling does not concurrently touch an endpoint's read-FSM buffer that iomgr owns — already mitigated by `endpoint_block_tasks` per-endpoint serialization + lock-guarded poll changes.
- **No network-partition test primitive**: the soak (`run-ha-soak.sh`) is process-kill only (`kill -9`, lines 74/440); unit "unreachable peer" cases use closed ports, not traffic drops. R7 has **no existing harness**.
## Requirement → Asset Map
| Req | Existing asset | Gap tag |
|-----|----------------|---------|
| R1 liveness during snapshot/config I/O | snapshot send `ha.c:2702`+`config_repl.c:2446`; config-push threads `config_repl.c:1412` | **Missing** off-load lane for snapshot; **Constraint** config-push blocks on join |
| R2 unreachable-peer / bounded connect | `endpoint_connect`/`socket_connect` netio.c:289/302 (blocking); `xce_probe_config_tag` pattern (reuse); reconnect via delaymgr `primary_heartbeat_fn` | **Constraint** (reorder connect) + **Missing** (offload reconnect off delaymgr) |
| R3 admission protection | `COMMIT_WAIT_RESERVED` pattern (commit-specific) | **Missing** (generalize to the slow lane) |
| R4 ordering/correctness | `endpoint_block_tasks`/`endpoint_unblock_tasks` per-endpoint sequential primitive (netio.c:896-911, auto on VTASK_IO path); lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (netio.c:1046-1082) for thread-safe epoll registration | **Reuse** (both primitives exist; no new marshal-back primitive needed) |
| R5 contract compliance | `ha-async-io.md` contract (steering authority) | verified by R1–R4 (no standalone asset) |
| R6 no regression | existing suites: `test_ha_election/prevote/liveness/failover/rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability` | **Reuse** as regression gate |
| R7 partition verification | `run-ha-soak.sh` (kill-only) | **Missing** (no traffic-drop primitive) |
| R8 heartbeat perf | heartbeat fan-out already non-blocking enqueue | **Constraint** (don't regress fast path) |
## Implementation Approach Options
### Option A — Extend in place (minimal)
Dispatch `peer_ctrl_msg_handler` through the existing `handle_msg`/`vtask_handle_msg` path; add a general non-IO `vtask_enq` wrapper; reorder `socket_connect` to non-blocking + `poll`; replace config-push `wait_all_with_timeout` join with a non-blocking completion check; reuse `delay_task_ms` only for *non-blocking triggers*. Touches ha.c, config_repl.c, netio.c, vtaskmgr.c.
- ✅ Least new surface; reuses every existing pattern; fastest path to closing the delaymgr+iomgr inline blocks.
- ❌ Long I/O (snapshot, config-push) still lands in the **request** vtask pool → relocates the stall to pool exhaustion (R3 only partially met).
### Option B — New dedicated slow-lane executor
Build a dedicated bounded **slow-lane executor** (generalize `fsync_worker`) for all long blocking I/O. Peer control still dispatched to vtask, but heavy work re-offloaded to the slow lane. (No iomgr-post primitive needed — epoll registration reuses the lock-guarded `iomgr_monitor`.)
- ✅ Clean separation; R3/R4 fully met by construction; isolates seconds-long I/O from request handling and from delaymgr.
- ❌ One genuinely new component; more interface design than extending in place.
### Option C — Hybrid (recommended)
- **Extend** (reuse): peer_ctrl → vtask via the existing dispatch pattern; general non-IO vtask submit (compose `vtask_alloc/init/enq`); bounded connect (reorder + reuse `xce_probe_config_tag` pattern); preserve ordering via `VTASK_IO` serialization.
- **New** (the one real gap): a small bounded **slow-lane executor** for snapshot-transfer + config-push (generalize the `fsync_worker`/`push_thread` precedent; per-peer dedup + admission cap parallel to `COMMIT_WAIT_RESERVED`). Epoll registration of a connected fd reuses the lock-guarded `iomgr_monitor` (no new primitive).
- **Phased**:
- *Phase 1 (S–M, highest value/lowest risk):* bounded connect + offload reconnect off delaymgr → closes the heartbeat *sender*-side starvation (R2). Independent, shippable.
- *Phase 2 (M):* dispatch peer_ctrl to vtask (epoll registration reuses lock-guarded `iomgr_monitor`) → closes the iomgr *receiver*-side inline processing (R4, R5).
- *Phase 3 (L):* slow-lane executor + move snapshot-send/config-push to it + admission generalization (R1, R3).
- *Phase 4 (M):* partition test harness + regression sweep (R6, R7, R8).
- ✅ Each phase is independently testable and de-risks the next; matches what already exists vs. what's truly missing.
- ❌ Requires phase coordination; must keep `ha-async-io.md` compliance consistent across phases.
## Effort & Risk
| Work item | Effort | Risk | Justification |
|-----------|--------|------|---------------|
| Bounded connect + reconnect offload (Ph1) | S–M | Medium | reorder existing pattern; offload target needs to be non-delaymgr |
| General non-IO vtask submit | S | Low | compose existing `vtask_alloc/init/enq` |
| peer_ctrl → vtask dispatch | M | Medium | correctness-sensitive: ordering (mitigated by `endpoint_block_tasks`), term/vote semantics; epoll register reuses lock-guarded `iomgr_monitor` |
| Slow-lane executor + snapshot-send offload | L | Medium-High | per-peer dedup + coordination with ongoing append/WAL-trim/catch-up stream |
| Admission generalization (R3) | M | Medium | extend `COMMIT_WAIT_RESERVED` pattern to slow lane |
| Partition test harness (R7) | M | Medium | no traffic-drop primitive today; soak infra is kill-only |
| **Overall** | **L (1–2 wks)** | **Medium** | riskiest: snapshot-offload coordination + moving peer_ctrl off iomgr (no new epoll-core primitive needed) |
## Recommendations for Design Phase
- **Preferred approach:** Option C (Hybrid), phased. Ship Phase 1 (bounded connect) early — it independently closes the sender-side heartbeat starvation and is low-risk.
- **Key decisions to settle in design:**
1. **Connected-fd registration:** reuse the lock-guarded `iomgr_monitor` (netio.c:1058) — a worker prepares the endpoint fully, then registers it; no new primitive. Design-verify only that worker handling does not touch an endpoint's read-FSM buffer concurrently with iomgr (serialized by `endpoint_block_tasks`).
2. **Slow-lane shape:** a shared bounded pool vs. reuse the existing per-standby `push_thread` model for config-push and a small pool for snapshot. Size to peer count.
3. **Fold-in question — RESOLVED 2026-06-25: keep `raft-drive-async` STANDALONE.** Scope-check verdict ADJACENT-SAME-AREA: `election-iomgr-refactor` is fully implemented and fixed the *startup* blocking path (pthread election before `cluster_run`); it touched none of raft-drive-async's runtime targets (`peer_ctrl_msg_handler` dispatch, delaymgr `connect`, snapshot/config-push on iomgr) and explicitly deferred async DNS as a non-goal. It is consumed as completed *upstream*, providing reusable primitives (`delay_task_ms`, `on_connected` fire-once callback, the `ha_start_election_async`/non-blocking vote-fan-out contract). Folding in would reopen a done spec for net-new infra it never held.
## Research Needed (carry to design)
- ~~election-iomgr-refactor scope overlap~~ — **RESOLVED 2026-06-25** (scope-check): ADJACENT-SAME-AREA, keep standalone; reuse its primitives (`delay_task_ms`, `on_connected`, `ha_start_election_async`). See Recommendations decision #3.
- **Endpoint read-FSM ownership**: confirm worker-side peer-control handling never touches an endpoint's read-FSM buffer concurrently with the iomgr read path (expected safe via `endpoint_block_tasks` + handoff of `fsm_in.pkt`). (Epoll registration itself is already settled — lock-guarded `iomgr_monitor`; no new primitive.)
- **Partition primitive choice** for R7: `iptables`/`tc` (root, infra-level) vs. an in-process socket-drop hook in `xce_test`. Determines test portability.
- **Snapshot-send coordination** with the concurrent WAL-catch-up stream and the primary's append/`trim_wal_segments` while the transfer runs on the slow lane.
- **Admission interaction** between the existing `COMMIT_WAIT_RESERVED` (commit waits) and the new slow-lane cap — shared budget vs. independent, so the two together can't starve the 8-worker pool.
---
## Design Synthesis (2026-06-25, /kiro-spec-design)
Key simplification chosen at design time: **the offload target for long blocking I/O is a dedicated slow-lane executor (separate threads), NOT the vtask request pool.** Consequences:
1. **R3 (pool protection) becomes structural** — long ops never run on the vtask pool, so they cannot starve liveness-critical message handling. The existing `COMMIT_WAIT_RESERVED` continues to cover commit-waits.
2. **The "general non-IO vtask submit" research item is SUPERSEDED** — not needed. Peer-control handling is packet-triggered, so it reuses the existing VTASK_IO `vtask_handle_msg` path; the only packet-less trigger (delaymgr reconnect) goes to the slow lane, not the vtask pool. No new vtask submit primitive required.
3. Combined with the already-refuted iomgr-post primitive (epoll registration reuses lock-guarded `iomgr_monitor`), **the only genuinely-new component is the slow-lane executor** (`slowlane.{c,h}`). Everything else is reuse + integration.
Build-vs-adopt: slow lane generalizes the existing `fsync_worker`/`push_thread` background-thread precedent (adopt pattern, build the small bounded executor). Bounded connect reuses the `xce_probe_config_tag` pattern (reorder, not new). Per-endpoint ordering reuses `endpoint_block_tasks`.
Phased migration (P1 bounded-connect → P2 peer_ctrl→vtask → P3 slow-lane+offload → P4 partition test) keeps each step behavior-preserving and independently revertible.

22
.kiro/specs/raft-drive-async/spec.json Normal file
View file

@ -0,0 +1,22 @@
{
"feature_name": "raft-drive-async",
"created_at": "2026-06-25T11:37:13Z",
"updated_at": "2026-06-26T04:43:19Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

6
.kiro/specs/raft-drive-async/tasks.md
View file

File diff suppressed because one or more lines are too long