docs: promote raft-apply-decoupling spec family from /kiro-discovery

Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol step lifted out of the notification hook fan-out (root of the bug-795/828/ 851/1060 divergence family). Decision: apply-after-commit via a single shared apply_committed_entry() seam reused by online apply and restart replay. - raft-drive-async: full spec docs (brief/requirements/research/design) committed alongside its already-tracked tasks.md; async-drive prerequisite for the apply-decoupling work, sequenced to start first (threatens failover liveness today). - raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open decision D1). - job-array-single-wal-record: promoted from backlog to a tracked spec, sequenced after raft-apply-decoupling. - ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant), documenting the current on-receipt fact-reaction deviation this family will reconcile. - raft-commit-safety / raft-log-replication: reconcile spec.json phase to "implemented" (both were mislabeled tasks-generated despite full 18/18 and 24/24 task completion + raft_sim GO). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 02:41:11 +00:00 · 2026-07-03 02:41:11 +00:00 · 5f307e297b
commit 5f307e297b
parent 9318f9977e
14 changed files with 1075 additions and 16 deletions
--- a/.kiro/specs/ha-consistency-invariants/requirements.md
+++ b/.kiro/specs/ha-consistency-invariants/requirements.md
@ -42,6 +42,7 @@ The XCE controller cluster shall provide a uniform consistency contract governed
 6. The controller shall advance its commit index monotonically; the commit index shall never decrease.
 7. The controller shall be able to reconstruct its in-memory state from its persisted committed state (snapshot plus committed log entries), discarding any provisional state not present in the committed log.
 8. When the primary receives a reported fact with no requester awaiting confirmation (such as a worker-reported job state transition), the primary shall retain the fact and continue attempting commitment — including across quorum-loss periods — until the fact's log entry is committed; while the primary remains running and holds leadership, it shall not discard an uncommitted reported fact.
+9. **(Facts-Apply invariant — added 2026-06-25 via `/kiro-discovery`.)** When the primary applies a committed reported fact, the state-machine mutation and any externally-consequential reaction it triggers (worker slot release, scheduler dispatch, client/apiserver notification) shall occur only after the fact's log entry is committed, through the same post-commit apply seam used for requester-driven operations; the primary shall not perform a fact's externally-consequential reactions on receipt ahead of commit. This refines criterion 8 (which governs the *durability/retention* of facts) by governing the *timing of their apply*, and aligns facts with the Introduction principle that "anything externally consequential … is gated on commit, not on the in-memory view." **Documented deviation (per the requirements authority's deviation rule):** the current implementation applies `WAL_MODE_RETAIN` facts and fires their reactions on receipt (before commit). Reconciling the implementation with this criterion is owned by the `raft-apply-decoupling` spec — candidate plan and step list in `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` — with the quorum-loss reaction policy tracked there as open decision **D1** (block-but-durable vs a local-apply allowlist).

 ### Requirement 3: Post-Election Write Safety

--- a/.kiro/specs/job-array-single-wal-record/brief.md
+++ b/.kiro/specs/job-array-single-wal-record/brief.md
@ -0,0 +1,74 @@
+# Brief: job-array-single-wal-record
+
+## Problem
+
+A job-array submission fans out into **one Raft log record per array element**. Confirmed in the controller:
+
+- `job_submit()` loops `for (i = 0; i < submit->array_size; i++)` calling `create_job()` once per element (`job.c:74`).
+- `persist_job_hook_fun` then loops over the resulting job list calling `persist_wal_sync_append(p, WAL_JOB_SUBMIT, ...)` once per element (`persist.c:2556`), and **each** call runs its own `repl_wait_majority_ack` commit barrier (`persist.c:2248`).
+- The scheduler receives one push per committed element (`sched.c`).
+
+So a 10k-element array becomes 10k WAL appends + 10k sequential majority-commit barriers + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, scheduler push volume, and failover replay time **linearly in array size**, while holding the submit thread across every per-element barrier.
+
+## Current State
+
+- No batched/atomic array record exists; the array is a loop over the single-job path.
+- The commit engine (`ha-commit-engine`) and barrier rollout (`ha-commit-barrier-rollout`) provide per-LSN majority commit and idempotency guards — the primitives needed to commit one larger record atomically.
+- Per-element identity (jobid = base_id + i) and per-element addressing (status / cancel / completion) are assumed throughout the controller and scheduler.
+
+## Desired Outcome
+
+A job-array submission persists, replicates, and pushes as a **single** Raft log record:
+
+- The controller writes **one** WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through one commit barrier.
+- Replication carries the single record; standbys reconstruct the whole array from one committed LSN.
+- The scheduler receives the array as one push and expands locally.
+- Per-element identity and state addressing are preserved after expansion (status, cancel, completion still operate per task id).
+
+## Approach
+
+Single atomic array record on top of the post-decoupling apply seam:
+
+- Define a `WAL_JOB_ARRAY_SUBMIT` record encoding base job spec + index range/count + sparse per-index overrides.
+- Command handler appends + commits this one record through a single barrier; on commit, the shared `apply_committed_entry()` seam expands it into N in-memory `job_t` entries (reusing `create_job` per element) under one apply step.
+- Restart replay decodes the single record and expands identically (same apply seam → no online/replay divergence).
+- Scheduler push delivers one array descriptor; scheduler expands locally.
+- Preserve idempotency: re-applying a committed array record (replay or retry) must not double-create elements.
+
+## Scope
+
+- **In**: single WAL record format for arrays; atomic single-barrier commit; standby reconstruction from one LSN; single scheduler push + local expansion; preserved per-element identity/addressing; replay expansion; idempotency on re-apply; requirement + regression tests (large-array commit count, failover replay time, per-element status/cancel still correct).
+- **Out**: the apply-ordering/hook-decoupling refactor itself (sibling spec `raft-apply-decoupling`); non-array single-job path changes; scheduler policy changes.
+
+## Boundary Candidates
+
+- The `WAL_JOB_ARRAY_SUBMIT` record format + encode/decode.
+- The single-barrier commit of the array record.
+- The expand-on-apply step (online + replay, shared).
+- The single scheduler push + scheduler-side expansion.
+
+## Out of Boundary
+
+- Per-element override semantics beyond what single-job submit already supports.
+- The commit engine / barrier internals (consumed unchanged).
+
+## Upstream / Downstream
+
+- **Upstream**: `raft-apply-decoupling` (HARD — builds the array expand step on the clean `apply_committed_entry()` seam rather than the old optimistic-before-commit path); `ha-commit-engine` + `ha-commit-barrier-rollout` (single atomic commit + idempotency); `ha-consistency-invariants` (replicated scheduling state).
+- **Downstream**: failover replay time and replication bandwidth improve for array-heavy workloads; lightweight rejoin re-attach optimization benefits.
+
+## Existing Spec Touchpoints
+
+- **Extends**: none (new record type).
+- **Adjacent**: `raft-apply-decoupling` (shared apply seam — must land first); `ha-commit-barrier-rollout` (idempotency guards reused for re-apply safety); `scheduler-controller-msg-gap` (scheduler push path).
+
+## Constraints
+
+- Must preserve per-element identity/state addressing after expansion (status, cancel, completion per task id).
+- Atomic commit: either the whole array commits or none of it (no partial array on a standby).
+- Idempotent re-apply (replay/retry must not duplicate elements).
+- Bounded record size / encoding for very large arrays (avoid a single pathological WAL entry; define a max or chunking policy in design).
+
+## Provenance
+
+Promoted 2026-06-24 from the roadmap backlog item `job-array-single-wal-record` (recorded 2026-06-16) via `/kiro-discovery`.
--- a/.kiro/specs/raft-apply-decoupling/brief.md
+++ b/.kiro/specs/raft-apply-decoupling/brief.md
@ -0,0 +1,86 @@
+# Brief: raft-apply-decoupling
+
+## Problem
+
+The controller's Raft consensus protocol is **embedded inside the notification hook fan-out**, and in-memory state is mutated **before** the entry is committed. This couples three concerns that canonical Raft keeps separate (log → commit → apply → notify):
+
+- `persist_job_hook_fun` is registered as an `AFTER_EVENT` hook, and *inside* that callback it performs the WAL append (`ha_primary_log_append` → `raft_log_append`) **and** blocks on the synchronous majority-commit barrier (`repl_wait_majority_ack`). The raft protocol step is therefore literally a side effect of running a hook (`persist.c:2532`, `persist.c:2198`, `ha.c:3080`).
+- `handle_jobsubmit` mutates the in-memory job table via `create_job()` **before** the WAL/commit step (`job.c:558` runs before the AFTER hook at `job.c:593`). A commit that fails or times out does **not** cleanly revert that optimistic mutation, leaving a window where in-memory state diverges from the committed log.
+- Because apply happens before commit, every downstream consumer (scheduler push, client reply) must defensively re-gate on a commit watermark (`sched.c:38` `job_submit_committed`) to avoid acting on not-yet-committed state — scattering commit-awareness across the codebase instead of centralizing it at one apply point.
+
+This is the structural root behind the bug-795/828/851/1060 divergence family: the controller has no single, ordered "apply committed entry to the state machine" seam.
+
+## Current State
+
+- Hooks (`src/lib/hook.c`) are a typed pub/sub fan-out (`xce_run_hooks`) with `HOOK_BEFORE_EVENT` / `HOOK_AFTER_EVENT` situations. Subscribers: `persist_job_hook_fun` (durability — does WAL+commit), `sched_job_hook_fun` (scheduler push), apiserver SSE callbacks.
+- Order today: `BEFORE hook → create_job() (state mutation) → AFTER hook { WAL append + majority commit barrier } → client reply → scheduler push (commit-gated)`.
+- The raft library (raft-log-replication, raft-commit-safety) already OWNS the replicated log and exposes pure callbacks (`on_persist_entry`, `on_commit_advanced`, etc.). The library half is the right protocol core; the controller half is what is mis-layered.
+- Restart replay (`persist_recover` → `replay_wal` → `apply_wal_entry`, `persist.c:1950/1735/1606`) validates only the CRC chain. There is **no** post-replay check that materialized in-memory state equals the committed log.
+
+## Desired Outcome
+
+The controller follows canonical Raft ordering with one explicit apply seam:
+
+1. **pre-hooks** (validation / preparation only — no authoritative state change)
+2. **raft step** — append entry → replicate → majority commit (an explicit protocol step, NOT a hook callback)
+3. **apply** — a single `apply_committed_entry()` seam mutates the in-memory state machine *after* commit
+4. **after-hooks** = pure notification delivery (scheduler push, client reply/SSE, apiserver) — they observe already-committed, already-applied state and no longer carry WAL/commit responsibility
+
+A commit that never reaches majority therefore never applies, eliminating the optimistic-divergence window. On restart, the same `apply_committed_entry()` seam is reused by replay, and a verification step proves materialized state == committed log.
+
+## Approach
+
+Apply-after-commit (canonical), chosen 2026-06-24. Invert the submit path so the raft commit gates the apply:
+
+- **KEYSTONE PRE-REQUIREMENT (must land first):** convert the protocol drive from blocking to event-driven. `repl_wait_majority_ack()` today blocks on `pthread_cond_timedwait` (commit.c:359) — a parked thread per in-flight commit. Build its async counterpart: append → register `pending(LSN)` → resolve on majority-ACK/timeout/demotion → drive apply + fire reply continuation. B (apply self-deadlock), C (reply lifecycle), and vtask-pool exhaustion are all downstream of this one conversion. See `facts-in-raft.md` § Keystone. The rest of this approach assumes the drive no longer blocks.
+
+- Extract WAL append + commit barrier out of `persist_job_hook_fun` into an explicit controller-side raft step invoked directly by command handlers (not via the hook system).
+- Introduce a single `apply_committed_entry()` function that the commit path calls **after** `on_commit_advanced`, and that WAL replay reuses on restart — guaranteeing online-apply and replay-apply are the same code (no divergence-by-two-implementations).
+- Demote the hook fan-out to notification-only: after-hooks fire from the apply step, observing committed state. Remove the now-redundant `job_submit_committed` watermark gating where the new ordering makes it unnecessary.
+- **Commands vs facts policy:** make commands (`WAL_MODE_WAIT`: job submit, config) strictly apply-after-commit. Facts (`WAL_MODE_RETAIN`: worker-reported start/finish/fail/rusage) are applied on receipt today because the controller reacts immediately. **Direction set 2026-06-25: route facts through commit-then-apply as well** — candidate plan and 15-step list in `facts-in-raft.md` (this spec dir), formalized as the **Facts-Apply invariant** (criterion 2.9) in `ha-consistency-invariants/requirements.md`. After the change, commands and facts share the one `apply_committed_entry()` seam and differ only in ingress + commit-wait/failure policy (see the comparison table in `facts-in-raft.md`). The one remaining open decision is **D1** (quorum-loss reaction policy: block-but-durable vs local-apply allowlist) — resolve before Phase C.
+- **Restart raft-log-applied validation (item #4, in-scope):** add a post-replay verification that the materialized entity set matches the committed log tip (e.g., count/hash of applied entities vs WAL-derived expectation), surfacing a CLEAN/DIVERGENT verdict alongside the existing CRC verdict.
+
+## Scope
+
+- **In**:
+  - Decouple the raft protocol step from the hook fan-out (item #1).
+  - Canonical ordering: pre-hooks → commit → apply → after-hooks-as-notification (item #2).
+  - Single shared `apply_committed_entry()` seam used by both online commit and restart replay.
+  - Post-restart validation that committed raft log was correctly applied to materialized state (item #4).
+  - Requirement + regression test design for the above (item #5): unit tests for ordering/rollback, plus conformance assertions extended in `raft-sim-conformance`.
+- **Out**:
+  - Job-array single-record collapse — owned by the sibling `job-array-single-wal-record` spec.
+  - Changes to the raft library protocol core (owned by raft-log-replication / raft-commit-safety).
+  - Entity-bearing snapshots / config-upgrade snapshot (separate specs).
+  - Read-consistency / leader-lease optimizations.
+
+## Boundary Candidates
+
+- The explicit controller raft step (append + commit barrier) extracted from `persist_job_hook_fun`.
+- The single `apply_committed_entry()` apply seam (shared online + replay).
+- The hook system's demotion to notification-only (after-hook subscribers re-pointed at the apply step).
+- The commands-vs-facts apply policy.
+- The post-restart apply-correctness verification.
+
+## Out of Boundary
+
+- The raft library callbacks/protocol (consumed, not modified here).
+- Array-submission batching.
+- Scheduler framework policy changes.
+
+## Upstream / Downstream
+
+- **Upstream**: raft-log-replication + raft-commit-safety (library log/commit core providing `on_persist_entry` / `on_commit_advanced`); ha-term-safety (controller-side raft integration umbrella — this spec restructures the same submit→commit path ha-term-safety wires, so it must land in coordination with it); ha-commit-durability (durable-commit semantics, unchanged).
+- **Downstream**: `job-array-single-wal-record` (builds the single atomic array record on top of the clean apply seam); worker-rejoin-resync and any future state-machine change benefit from the centralized apply point.
+
+## Existing Spec Touchpoints
+
+- **Extends / sequences with**: `ha-term-safety` (controller raft integration umbrella — disjoint-but-adjacent edits to the submit/commit path; coordinate ordering to avoid conflicting refactors).
+- **Adjacent**: `raft-sim-conformance` (item #5 regression assertions land here — apply-after-commit ordering + no-apply-on-failed-commit invariants); `ha-commit-barrier-rollout` (commit classification table — the WAIT vs RETAIN distinction this spec must respect); `raft-log-replication` / `raft-commit-safety` (library callbacks).
+
+## Constraints
+
+- Must not block the iomgr thread (async-I/O invariant in `ha-async-io.md`): the explicit commit barrier must run off the iomgr thread, as the current `persist_wal_sync_append` already arranges.
+- "Controllers never self-terminate" — a divergent restart verdict is a loud, non-promotable fence, never an `exit`/`abort`.
+- Preserve persist-before-ACK durability and the synchronous commit barrier as policy (do not canonicalize them away).
+- The shared apply seam must be deterministic so online-apply and replay-apply produce identical materialized state.
--- a/.kiro/specs/raft-apply-decoupling/facts-in-raft.md
+++ b/.kiro/specs/raft-apply-decoupling/facts-in-raft.md
@ -0,0 +1,150 @@
+# Design Note: Routing Facts Through Raft (apply-after-commit for RETAIN events)
+
+**Status:** candidate plan (2026-06-25). Resolves the "commands vs facts apply policy" open question in `brief.md`. **D1 resolved 2026-06-25: block-but-durable (retain-only) under quorum loss.** Design-risk triage complete — see `## Design Risks & Tricky Cases`.
+
+## Premise
+
+Today XCE splits state changes into two paths:
+
+- **Commands** (`WAL_MODE_WAIT`: job submit, config) — client-initiated; synchronous commit before reply.
+- **Facts** (`WAL_MODE_RETAIN`: worker-reported job start/finish/fail/rusage) — worker-initiated; **applied + reacted to on receipt**, WAL-appended non-blocking, committed later via a covering standby ACK (`wal_hook_append_classified`, `persist.c:2488`).
+
+Facts were made apply-on-receipt for liveness: a worker fact (job finished) must promptly free the slot and trigger the next dispatch, and the worker has already moved on (fire-and-forget, zero redelivery — bug-396).
+
+**"Raft handling facts" means:** a fact becomes a normal committed log entry. Its *state-machine mutation* (job RUN/DONE/FAIL) and its *reactions* (slot free, dispatch trigger, client/apiserver notify) move to the post-commit `apply_committed_entry()` seam. Only the **durable append** stays on-receipt (so the fact is never lost). This makes online-apply ≡ replay-apply for facts too, and eliminates phantom reactions on uncommitted facts.
+
+## Do we still handle requests and facts differently? (yes — but only at the edges)
+
+Once both requests (commands) and facts route through commit-then-apply, **the apply step unifies** — both go through the same `apply_committed_entry()` seam. What stays type-specific is the *ingress* and the *commit-wait / failure* policy, not the apply:
+
+| Aspect | Request / command (was `WAL_MODE_WAIT`) | Fact (`WAL_MODE_RETAIN`) |
+|---|---|---|
+| Origin / ingress | client; non-leader arrivals redirected via the `libxce-failover` taxonomy | worker; non-leader arrivals rejected/redirected, lost-window recovered by `worker-rejoin-resync` |
+| Commit wait | **synchronous** — handler blocks on `repl_wait_majority_ack`; client reply gated on commit (criteria 2.1/2.2) | **async** — append, return immediately; commit via the group-commit batch; never a per-fact blocking barrier (high volume) |
+| If commit fails / times out | **report FAILED** to the waiting requester (criterion 2.2) | **nobody to tell** — retain + retry while leader (criterion 2.8) |
+| Quorum loss | block/fail (CP) | retained-and-retried; decision **D1** decides any local-react escape hatch |
+| State mutation + reaction (apply) | **shared** `apply_committed_entry()`, post-commit | **shared** `apply_committed_entry()`, post-commit |
+
+**The key shift:** the `WAIT` vs `RETAIN` classification's *meaning narrows*. Today it conflates two things — "apply-after-commit vs apply-on-receipt" **and** "synchronous-confirm vs async". After this change the apply timing is the **same** for both (after commit); the classification governs **only** "does a requester block and get a failure reply" vs "is it async-retained". So we still *route* them differently at the edges, but they *converge* at the one place correctness depends on: the apply seam.
+
+## Why this is correct, not just possible
+
+- A fact is an external input to the replicated state machine. Raft already commits external inputs (that's what `ClientRequest` is). Nothing about a fact requires apply-on-receipt at the *log* level.
+- Reactions are state-machine **outputs**. Canonical Raft produces outputs at apply time (post-commit), on the leader, idempotently. Today's on-receipt reaction violates this — it can fire on a leader that then loses the entry.
+- The latency cost (one commit round-trip before slot-free/dispatch) is the same round-trip commands already pay, and the group-commit flusher (`ha-commit-durability`) already amortizes the high fact volume. The iomgr thread is never blocked: `on_commit_advanced` drives apply asynchronously.
+
+## Decisions to lock before building
+
+- **D1 — Quorum-loss fact policy — RESOLVED 2026-06-25: (a) block-but-durable, retain-only.** When the primary cannot reach a majority, a fact is appended durably but NOT applied (no state mutation, no reaction) until its entry commits. No local-apply allowlist. Rationale: option (b) (local-apply allowlist) directly contradicts criterion 2.4 (provisional state must not drive irrevocable exports such as dispatch) and reopens the bug-795/828/851 divergence family. Cost of (a) is bounded and self-healing — only *new* dispatch stalls during the (already-degraded) quorum-loss window; running jobs continue (worker autonomy); retained facts commit+apply in order on quorum restore; primary-death-in-window recovered by `worker-rejoin-resync`. Consistent with CP + "controllers never self-terminate". Refinements: no carve-outs (allowlists rot); accept the documented stale-read deviation; emit a metric/WARN on retained-fact backlog age; facts inherit Requirement 7's quorum-loss posture.
+  - Rejected: **(b) local-apply allowlist** — buys availability the operator should fix at the quorum level, at the cost of the core safety invariant.
+- **D2 — Reaction timing.** Confirm ALL fact reactions (slot free, scheduler dispatch, client/apiserver notify) are safe to defer to post-commit. Expectation: yes — the worker is fire-and-forget; nothing external blocks on the controller's reaction.
+- **D3 — Idempotency key.** Fact apply must be idempotent (replay, worker-rejoin-resync re-report, retry). Key on `(jobid, monotonic job-state)` so a re-applied or out-of-order fact is a no-op, never a regression of state. Reuse the `mark_job_*` idempotency guards (shared need with `ha-consistency-invariants`).
+
+## The list — follow in order
+
+### Phase A — Preconditions (shared with raft-apply-decoupling core)
+1. Land the single `apply_committed_entry()` seam; route BOTH commands and facts through it.
+2. Add/confirm `mark_job_*` idempotency guards (D3) so apply is replay/resync/retry safe.
+3. Ensure the async group-commit flusher (`ha-commit-durability` `fsync_worker`) wakes a post-commit apply pump via `on_commit_advanced(lsn)` — no iomgr blocking.
+
+### Phase B — Reclassify facts onto the committed-apply path
+4. Change fact ingestion (the `WAL_MODE_RETAIN` path in `wal_hook_append_classified`, `persist.c:2488`) so on receipt it ONLY: validate → `ha_primary_log_append` (append entry) → return. **Remove the on-receipt state mutation and reactions.**
+5. Move the fact state mutation (`restore_job_state`-equivalent: RUN/DONE/FAIL/rusage) into `apply_committed_entry()`, keyed on committed LSN.
+6. Move the reactions (slot free, scheduler dispatch trigger, client/apiserver notify) to fire from the apply step, after the mutation, leader-only.
+7. Implement the apply pump: `on_commit_advanced(lsn)` applies all newly-committed entries (facts + commands) in LSN order exactly once.
+
+### Phase C — Failover / liveness correctness
+8. Reject/redirect facts arriving at a non-leader (workers must target the leader; reuse the `libxce-failover` redirect taxonomy `E_XCE_REDIRECT`/`E_XCE_NOMASTER`/...).
+9. Wire `worker-rejoin-resync` as the backstop for facts lost in the appended-but-not-committed or never-arrived window.
+10. Implement D1 (RESOLVED: block-but-durable — append durably, no apply/react until commit, no allowlist); gate it, emit the retained-fact backlog metric, and test the quorum-loss-then-restore path.
+
+### Phase D — Restart / replay parity
+11. Restart replay applies fact entries through the SAME `apply_committed_entry()` → online-apply ≡ replay-apply.
+12. Extend the brief's post-restart apply-validation (item #4) to cover fact-derived final states, not just submits.
+
+### Phase E — Verification / regression (item #5)
+13. `raft-sim-conformance`: assert no fact reaction (slot-free/dispatch/notify) occurs before that fact's entry commits; assert idempotent re-apply; assert State-Machine-Safety across interleaved fact+command logs.
+14. Live soak (`/xce-test-ha`): kill the primary mid-fact-stream; assert no phantom slot-frees, no divergence, no final-state loss beyond what `worker-rejoin-resync` recovers.
+15. Performance regression: high worker-churn fact stream must not regress — verify group-commit keeps fact commit amortized (NO per-fact synchronous barrier; facts never use the command `repl_wait_majority_ack` blocking path).
+
+## Keystone Pre-requirement: async, non-blocking commit drive *(must land FIRST)*
+
+Everything in this spec is downstream of one conversion. The protocol **drive** is currently **blocking**: `repl_wait_majority_ack()` → `pthread_cond_timedwait(&ce->cond, &ce->lock, &dl)` (commit.c:359, bounded by `DEFAULT_SYNC_TIMEOUT_MS`=500ms, config.h:93) **parks a thread until a majority of WalAcks arrive**. This is *blocking-offloaded* (the wait was moved off the iomgr thread) but NOT *event-driven* — it honors the letter of "never block iomgr" while still parking one worker thread per in-flight commit.
+
+**Pre-requirement — convert the commit engine from `wait` to `resolve`** (do NOT build a parallel registry; reuse the commit engine's existing per-LSN majority tracking — today it `cond_broadcast`s, replace that with `resolve()`):
+- append → register `pending(LSN)` on the existing per-LSN commit structure, return immediately (no park).
+- on each WalAck event: if majority reached → `resolve(lsn, COMMITTED)`; on timeout/demotion/truncation → `resolve(lsn, FAILED)`.
+- `resolve(lsn, outcome)` → advance commit_index → drive apply (off the ACK thread) → fire the reply continuation (C) and any apply-time proposal callbacks (B).
+- the `cond_timedwait` is **deleted, not wrapped**.
+
+**Why it is the keystone:** B (apply→propose self-deadlock), C (reply continuation), and pool-exhaustion (parked workers starving the bounded vtask pool under a slow/degraded majority) are **all downstream of the blocking drive**. One conversion dissolves all three: with nothing parked, there is no thread to self-deadlock, replies become natural callbacks, and the pool can't be starved by commits. The apply-after-commit work sits on top of this — so the async drive is a **hard prerequisite**, sequenced before Phase A.
+
+The blocking call is simple and obviously correct; the event-driven version is a state machine across async events (append-complete, each ACK, timeout, demotion, disconnect) carrying the same exactly-once + completeness burden as C, now on the protocol critical path. That cost is why "driving raft" — not the raft algorithm (already library-verified) — is the hardest part of this effort.
+
+### Two distinct prerequisites — the commit barrier is NOT the only blocking violation
+
+A broad async-violation audit (2026-06-25) found the commit barrier is actually the *least* severe blocking class (it blocks a pooled vtask worker — bounded, degrades gracefully). **Three CRITICAL violations block the iomgr thread itself** — the single epoll loop driving ALL heartbeats/votes/ACKs/client I/O. Root cause: `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** (NOT handed to the vtask pool like normal cluster messages), so anything it does synchronously runs on the iomgr event loop. When iomgr stalls → heartbeats stop → peers declare the leader dead → election storm. This is a *separate defect class* from the apply refactor: it threatens protocol **liveness today**, independent of apply-after-commit, and is a likely root of the bug-851 chaotic-failover family.
+
+**CRITICAL (block iomgr / the heartbeat-driver thread):**
+1. **Snapshot send on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) → `persist_checkpoint` (fdatasync + dir scan + segment unlinks) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503), all on iomgr. A large snapshot freezes every heartbeat/vote/client request for its whole duration. Worse than the commit barrier.
+2. **Config push wait on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172,1177,2672,2677,3973), up to `CONF_PUSH_TIMEOUT_MS`. Code comments (ha.c:75-77) acknowledge the safe-vs-unsafe caller split but unsafe paths exist.
+3. **Blocking DNS + connect on the delaymgr (heartbeat) thread** — `endpoint_connect` → `gethostbyname` + blocking `connect()` (netio.c:289,302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unresolvable peer stalls ALL heartbeats/election-timeouts for seconds → false failover. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)
+
+**MEDIUM (block a vtask pool worker — same class as the commit barrier):**
+- `repl_wait_majority_ack` for `OPCode_upgradeConfigAck` (config_repl.c:1459); 30-second `usleep` checkpoint-wait poll in `do_upgrade_config_with_bytes` (config_repl.c:1661); `ha_persist_term` fsync on a vote grant (ha.c:3407).
+
+**Correctly event-driven (confirmed):** RequestVote fan-out, heartbeat fan-out, and WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + its fdatasync run on a vtask worker, not iomgr.
+
+**Verdict: the protocol drive is NOT fully event-driven.** Two separable prerequisites result:
+- **P1 — commit-barrier async** (this doc's keystone): unblocks the apply/reply/facts work (B, C, pool-exhaustion). Prerequisite for `raft-apply-decoupling`.
+- **P2 — drive hardening** (move snapshot send, config-push wait, and peer DNS/connect OFF the iomgr/delaymgr threads): independently valuable, independently testable, threatens liveness today. **Recommended home: its own spec `raft-drive-async` (or fold into `election-iomgr-refactor`, which already touches this thread model) — NOT buried inside `raft-apply-decoupling`.**
+
+  **Offload mechanism — "trigger, don't execute" (delaymgr/iomgr only schedule; blocking work runs in a pool with admission headroom):**
+  - The timer/event thread enqueues a task and returns immediately; the blocking work runs on a worker. The danger is relocating the stall: dumping blocking work into the bounded vtask pool can starve request handling. **XCE already solved this for the commit barrier — `COMMIT_WAIT_RESERVED` caps concurrent blocking waits at `max(1, n_vcpus − RESERVED)`, reserving headroom (commit.c:195-203).** Any offloaded blocking task MUST live under the same admission discipline.
+  - **API gap:** the existing enqueue `vtask_handle_msg` (vtaskmgr.c:356) is endpoint+inbound-packet-bound (`vtype=VTASK_IO`, requires `ep->fsm_in.pkt`). A reconnect has no inbound packet → P2 needs a **general (non-IO) task-submit path** (`vtask_enq` with custom data), or generalize the API.
+  - **Lane by duration:** short bounded tasks (**reconnect**, ≤200ms connect cap) → vtask pool **under an admission cap** (COMMIT_WAIT_RESERVED-style). Long tasks (**snapshot send, config-push**, seconds) → a **dedicated slow-lane executor**, NOT the request pool (a few concurrent multi-second tasks head-of-line-block request handling even with reservation).
+  - **Coordination per offloaded task:** (1) idempotent in-flight dedup (`pending[peer]` atomic) so periodic timers don't pile up duplicates; (2) refcount the endpoint at enqueue, release at completion (arg outlives the tick); (3) marshal completion (epoll fd registration) back to the **iomgr** owning thread (bug-646); (4) resolve the outbound-vs-inbound connection race.
+  - **DNS:** `gethostbyname` is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — **defer unless measured**. The `connect` fix is mostly reordering an existing pattern: non-blocking connect + bounded `poll` timeout already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); the buggy path just sets non-blocking AFTER connect (netio.c:716).
+
+  **Heartbeat-problem completeness (does the offload fix it? — partially):** Heartbeat liveness spans BOTH threads — delaymgr *sends* heartbeats + fires election timeouts; iomgr *processes* incoming ACKs/votes/replies (`peer_ctrl_msg_handler`). Connect-offload protects the delaymgr (sender) side and election-timeout firing — a real win — but is **necessary-not-sufficient**: the snapshot-send-on-iomgr blocker (#1) freezes the *receiver* loop, so the leader stops hearing its peers regardless of how cleanly it sends. **Heartbeat is safe only when BOTH delaymgr AND iomgr never block → P2 must land all three CRITICALs, not just connect.** Residual: delaymgr ticks at 200ms precision (delay.c) — a timer-granularity floor offload doesn't touch; set the failover-detection timeout with margin above 200ms (tuning, not blocking).
+
+  **Missing regression test:** `/xce-test-ha` only *kills* the primary (process-dead → connect fails fast with RST; does NOT reproduce this). The real test is a **partition variant** — make one peer *unreachable* (blackhole packets so connect hangs and snapshot-send stalls) and assert (a) heartbeats to survivors keep flowing and (b) no spurious election. This specifically exercises the *unreachable*-not-*dead* case the blocking findings care about — the bug-851 chaotic-failover family.
+
+## Design Risks & Tricky Cases
+
+Triaged 2026-06-25. After triage, only **A** and **B** are genuinely architectural; **C** was reclassified to request-handling correctness; **D–G** are correctness/cost refinements.
+
+### A — Reactions are not replayable → re-derive from committed state on recovery *(architectural; RESOLVED by existing resync)*
+A committed log perfectly records *what state should be*, but is a bad script for *what side effects to re-emit*: replaying a job-finish must rebuild DONE state but must NOT re-notify the client or re-dispatch. So reactions originate from **only two places**: (1) **live-leader apply** of newly-committed entries, and (2) the **full-state resync on consumer reconnect**. Recovery never replays reactions — it rebuilds committed state, and the inevitable reconnect (failover / primary-restart / promotion all look identical to a consumer) re-derives the world. Three apply contexts: leader-live = state + react; standby = state only; recovery/replay = state only, then re-derive via resync.
+
+**This is solved by XCE's existing reconnect-and-resync** (scheduler/apiserver full-state push on reconnect = the re-derivation step). Conditions for it to hold:
+1. Resync must push **committed** state only (post-replay), never provisional (criterion 2.7).
+2. **Full reconcile**, not delta (consumer discards old view) — guard against the "lightweight re-attach" optimization eroding this.
+3. Resync re-reports flow through **commit-then-apply** too (a rejoining worker's re-reported fact is appended→committed→applied idempotently, never apply-on-receipt — same D1/2.9 gate).
+4. Order resync **after** replay (re-reports above the recovered tip → append+commit; already-committed → idempotent no-op).
+
+**HARD prerequisite:** the worker→controller half depends on `worker-rejoin-resync`, which is **not yet implemented** (workers send only `hostattrs` today, no job state — bug-396). Without it, a fact lost in a failover window has no redelivery path and A reopens for facts. → upgrade the `worker-rejoin-resync` dependency in `brief.md`/roadmap from "designated backstop" to **HARD**.
+
+### B — Apply → propose reentrancy (deadlock risk) *(architectural; OPEN)*
+Apply runs inside the commit-advance callback. A reaction that produces *new cluster state* must **enqueue a fresh proposal**, never mutate inline and never block waiting for its commit (you'd deadlock inside the commit path). Rule: **apply may enqueue proposals; apply may never block on consensus.** The vtask-continuation discipline below (register-don't-block) is the same mechanism and covers B.
+
+### C — Reply ownership *(RECLASSIFIED: request-handling correctness, NOT architecture; low risk)*
+"One request → exactly one reply" is a **pre-existing** request-handling invariant. Apply-decoupling only **defers** the reply to commit; it adds no new architecture. The genuinely architectural sub-concerns are already solved by the existing design:
+- **Send race / socket-epoll ownership (bug-646 class):** solved — `xce_reply` is non-blocking (it *enqueues* to the endpoint's output queue; the io worker owns the wire). Reply is generated by the vtask handler-continuation (completion marshaled to the iomgr thread), enqueued, and delivered by the io worker.
+- **Endpoint lifetime (bug-1077 class):** solved — the io socket is owned by the endpoint, which is **refcounted**; a continuation holds a ref, so a mid-flight disconnect discards the reply against a dead endpoint rather than a UAF.
+
+**Residual (request-handling correctness, single-threaded if produced on the iomgr thread):**
+1. **Exactly-once deferred-reply resolution** per request across `{commit-success, commit-timeout, leadership-loss, uncommitted-tail truncation}` — an atomic "claimed" flag on the obligation; centralize all exits through one `resolve(lsn, outcome)`.
+2. **Bounded in-flight** (the one system-level item, not per-request): the async model loses the blocking model's free thread-pool backpressure, so add admission control on the accept path.
+
+### D — Which facts need consensus? *(refinement)*
+Job *lifecycle* (start/finish/fail) → RETAIN-through-commit (authoritative). `rusage`/telemetry → consider **LOCAL** (soft, re-derivable, highest-volume — don't commit-amplify it). Re-audit the `wal_classify` table.
+
+### E — Idempotency must be state-monotonic, not LSN-dedupe *(refinement; mandatory)*
+Compare job-state rank (PEND<RUN<DONE) and no-op regressions/duplicates — not LSN dedupe. **Resync amplifies this** (it floods re-reported facts), so E is mandatory, not optional. (Implements D3.)
+
+### F — Strict LSN-ordered apply *(refinement)*
+Serial, in-LSN-order apply pump, never reordered/parallelized — this is what makes "fact for a not-yet-created job" races vanish by construction. Lock it as a `raft-sim-conformance` assertion.
+
+### G — Notify reliability is unchanged *(note, not a regression)*
+Moving after-hooks to notify-only does NOT make delivery reliable (still best-effort, bug-396). Resync backstops loss **across a reconnect**, not per-message. Don't let the cleaner architecture imply guaranteed delivery.
--- a/.kiro/specs/raft-commit-safety/spec.json
+++ b/.kiro/specs/raft-commit-safety/spec.json
@ -1,9 +1,9 @@
 {
  "feature_name": "raft-commit-safety",
  "created_at": "2026-06-21T12:00:00Z",
-  "updated_at": "2026-06-21T12:00:00Z",
+  "updated_at": "2026-06-25T00:00:00Z",
  "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implemented",
  "approvals": {
    "requirements": {
      "generated": true,
@ -19,5 +19,6 @@
    }
  },
  "ready_for_implementation": true,
+  "implementation_status": "implemented — commit-safety + vote-freshness half landed + validated (18/18 tasks, raft_sim GO per memory project_raft_commit_safety_done). Phase reconciled 2026-06-25 (was mislabeled tasks-generated).",
  "note": "Auto-approved (-y mode). Adds the commit-safety + election-freshness half of raft.tla on top of raft-log-replication's library log. raft_advance_commit (AdvanceCommitIndex, raft.tla:259) advances commit_index to Max(agreeIndexes from match_index[]) ONLY when on_get_entry_term(Max)==currentTerm (Figure-8 current-term rule, G4); new callback on_commit_advanced(ctx, commit_index) TRIGGERS the existing synchronous commit barrier / persist-before-ACK durability (does NOT replace it). Vote/PreVote freshness upgraded from LSN-only to (lastLogTerm,lastLogIndex) lexicographic compare (HandleRequestVoteRequest logOk, raft.tla:285, G6) carried via last_log_term in raft_on_vote_request + on_vote_eligible/vote-request payload. Simulator asserts Figure-8 refusal + Leader Completeness. Boundary: owns commit-index advance + vote freshness in src/lib/raft ONLY; NOT the log model/AppendEntries (raft-log-replication), the barrier/durability mechanism (controller), config markers/snapshot (ha-term-safety), or the conformance harness (raft-sim-conformance). Depends: raft-log-replication (HARD). Closes bug-851 G4/G6."
 }
--- a/.kiro/specs/raft-drive-async/brief.md
+++ b/.kiro/specs/raft-drive-async/brief.md
@ -0,0 +1,92 @@
+# Brief: raft-drive-async
+
+## Problem
+
+The Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals (iomgr and delaymgr). When either stalls, heartbeats/votes/ACKs stop flowing → peers declare the leader dead → spurious election. This is a likely root of the **bug-851 chaotic-failover family**.
+
+A blocking audit (2026-06-25) found three CRITICAL violations of the (newly corrected) thread-role contract in `ha-async-io.md`:
+
+1. **Snapshot send + `persist_checkpoint` inline on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) runs `persist_checkpoint` (fdatasync + WAL-segment trim) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503) on the iomgr event loop. A large snapshot freezes ALL heartbeat/vote/client processing for its whole duration.
+2. **Config-push wait inline on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172/1177/2672/2677/3973), up to `CONF_PUSH_TIMEOUT_MS`.
+3. **Blocking `gethostbyname` + `connect` on the delaymgr heartbeat thread** — `endpoint_connect` (netio.c:289/302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unreachable peer stalls ALL heartbeats + election-timeouts for seconds. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)
+
+**Root cause:** `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** on peer/vote endpoints — it runs inline on iomgr instead of dispatching to the vtask pool like normal cluster messages (`handle_msg` → `vtask_handle_msg`). So any work it does, including the heavy I/O above, executes on the event loop.
+
+Plus MEDIUM vtask-pool blockers (pool-exhaustion, not drive-thread): config-ack wait (config_repl.c:1459), a 30-second `usleep` checkpoint poll (config_repl.c:1661), `ha_persist_term` fsync on a vote grant (ha.c:3407 — adds election latency).
+
+## Current State
+
+- The corrected **thread-role contract** is now steering (`ha-async-io.md`, 2026-06-25): iomgr = non-blocking I/O + dispatch only; delaymgr = trigger-only, never blocks; vtask pool = bounded blocking only under admission reservation; long blocking I/O → dedicated slow lane. This spec brings the **code into compliance** with that contract. (The contract SUPERSEDED the old "delaymgr for blocking ops = Permitted" rule, which was the design flaw behind these findings — delaymgr is both the heartbeat driver and was the documented blocking sink.)
+- Existing primitives to reuse: `COMMIT_WAIT_RESERVED` admission cap (commit.c:195-203) already keeps vtask-pool headroom against blocking waits; the non-blocking-connect + bounded-`poll` pattern already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); `vtask_handle_msg` (vtaskmgr.c:356) is the dispatch path but is endpoint+inbound-packet-bound (`vtype=VTASK_IO`).
+- Confirmed already-correct: RequestVote fan-out, heartbeat fan-out, WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + fdatasync run on a vtask worker, not iomgr.
+
+## Desired Outcome
+
+The protocol drive is non-blocking on both liveness threads, by construction:
+
+- **iomgr** does only non-blocking socket I/O + dispatch — `peer_ctrl_msg_handler`'s real work is dispatched to the vtask pool (conforming to the existing `handle_msg` pattern), so no future handler addition can reintroduce inline blocking.
+- Inside the vtask, **fast protocol work runs inline** (decode ack → update match_index → advance commit → send next heartbeat — all bounded); **long I/O is triggered to a slow lane** (snapshot transfer, config-push), never run in the request-pool worker.
+- **delaymgr** only fires timers and triggers work; the peer connect is **bounded** (non-blocking + `poll` timeout) and **offloaded** off the timer thread.
+- Heartbeat liveness is provably protected: a partitioned (unreachable, not merely dead) peer cannot stall heartbeats to healthy peers or trigger a spurious election.
+
+## Approach
+
+Bring the code into compliance with the steering thread-role contract, structurally rather than per-op:
+
+1. **Dispatch `peer_ctrl_msg_handler` to the vtask pool** (the structural fix). iomgr becomes epoll + enqueue only. Preserve per-connection message ordering (VTASK_IO per-endpoint serialization is a feature here). Marshal any iomgr-owned state mutations (epoll/endpoint lifecycle) back to the iomgr thread (bug-646 discipline).
+2. **Two-tier execution inside the vtask:** fast bounded protocol work inline; **trigger** long I/O (snapshot send + `persist_checkpoint`, config-push) to a **dedicated slow-lane / admission-capped executor** so multi-second tasks cannot starve the request pool. Per-peer dedup (one snapshot in flight per standby); coordinate with ongoing WAL-catchup stream and the primary's appends/WAL-trim.
+3. **delaymgr connect → bounded + offloaded:** reuse the `xce_probe_config_tag` non-blocking-connect + bounded-`poll` pattern; trigger the reconnect as an idempotent, refcounted, deduped task off the delaymgr thread; marshal the connected-fd epoll registration back to iomgr. DNS (`gethostbyname`) is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — defer async DNS unless measured.
+4. **General (non-IO) task-submit path:** add the submit primitive needed for #2/#3 (current `vtask_handle_msg` requires an inbound packet); or generalize it.
+5. **Admission discipline:** any offloaded blocking task lives under a reservation (extend the `COMMIT_WAIT_RESERVED` pattern) so liveness-critical messages always have a free worker.
+
+## Scope
+
+- **In**:
+  - Dispatch `peer_ctrl_msg_handler` real work to the vtask pool (root-cause structural fix).
+  - Snapshot send + `persist_checkpoint` off iomgr → slow lane, with per-peer dedup + WAL coordination.
+  - Config-push wait off iomgr → slow lane / async ACK.
+  - delaymgr peer connect → bounded non-blocking + offloaded; reconnect dedup/refcount/marshal-back.
+  - General non-IO vtask submit path + admission reservation for offloaded blocking tasks.
+  - Partition (unreachable-peer) regression test; verify no spurious election + heartbeats to survivors continue.
+- **Out**:
+  - The commit-barrier async conversion (P1 / `repl_wait_majority_ack` → `resolve`) — owned by `raft-apply-decoupling` as its keystone (different concern: that's the *commit* drive, this is the *transport/timer* drive).
+  - Apply-after-commit / hook decoupling / facts-through-raft (raft-apply-decoupling).
+  - Async DNS (deferred unless measured).
+  - The MEDIUM vtask-pool blockers beyond what the admission discipline naturally covers (note them; fix opportunistically).
+  - Entity-bearing snapshots / config-upgrade snapshot (separate specs).
+
+## Boundary Candidates
+
+- The `peer_ctrl_msg_handler` → vtask dispatch seam.
+- The dedicated slow-lane executor + admission reservation.
+- The general non-IO vtask-submit primitive.
+- The bounded-connect + offloaded-reconnect path.
+- The iomgr-owned-state marshal-back mechanism (epoll/endpoint lifecycle).
+
+## Out of Boundary
+
+- Commit-engine internals (P1 owns the commit-drive async).
+- Snapshot *content* format (entity-bearing snapshots is a separate spec) — this spec only moves the *transfer* off the drive thread.
+
+## Upstream / Downstream
+
+- **Upstream**: the `ha-async-io.md` thread-role contract (authority); existing `COMMIT_WAIT_RESERVED` (commit.c) and `xce_probe_config_tag` bounded-connect pattern (reuse); `election-iomgr-refactor` (COMPLETED upstream — provides reusable primitives `delay_task_ms`, `on_connected`, `ha_start_election_async`/non-blocking vote fan-out).
+- **Downstream**: `raft-apply-decoupling` (its async commit drive + apply pump reuse the slow-lane + admission infrastructure this spec establishes; and the drive must be non-blocking before apply is restructured on top of it); safe failover for the whole bug-851 family depends on this landing.
+
+## Existing Spec Touchpoints
+
+- **Adjacent (RESOLVED standalone 2026-06-25)**: `election-iomgr-refactor` is ADJACENT-SAME-AREA — it fixed the *startup* blocking path and is fully implemented; this spec fixes the *runtime* drive-thread paths it never touched. Keep separate; consume it as upstream. (Scope-check verdict in research.md.)
+- **Coordinate**: `ha-term-safety` (touches the same `peer_ctrl_msg_handler` / repl dispatch region); `ha-wal-replication` (snapshot/catch-up transport); `controller-status` (heartbeat/committed propagation on the same drive).
+- **Verification home**: `raft-sim-conformance` (add the partition/liveness assertion) and `/xce-test-ha` (partition variant).
+
+## Constraints
+
+- The corrected thread-role contract (`ha-async-io.md`) is authoritative — iomgr and delaymgr never block inline; long I/O on the slow lane; bounded blocking in vtask only under admission reservation.
+- "Controllers never self-terminate" — an unreachable peer is a loud retry, never an abort.
+- Preserve per-connection message ordering when dispatching `peer_ctrl_msg_handler` to vtask.
+- Must not regress the heartbeat fast path measurably at cluster scale (3–5 peers); per-message vtask overhead accepted in exchange for uniformity + liveness safety.
+- Verified by a **partition** (unreachable, not killed) soak — the case `/xce-test-ha`'s current kill-only soak does not reproduce.
+
+## Provenance
+
+Surfaced 2026-06-25 during the `raft-apply-decoupling` discovery as prerequisite **P2** (drive hardening); promoted to its own spec because it threatens liveness independently of the apply refactor. Full analysis: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § "Two distinct prerequisites" + P2 offload mechanism. Thread-role contract: `.kiro/steering/ha-async-io.md` (corrected 2026-06-25).
--- a/.kiro/specs/raft-drive-async/design.md
+++ b/.kiro/specs/raft-drive-async/design.md
@ -0,0 +1,385 @@
+# Design Document — raft-drive-async
+
+## Overview
+
+**Purpose**: Make the XCE controller's Raft protocol **drive** fully event-driven by removing every blocking operation from the two threads that carry cluster liveness — the iomgr epoll loop and the delaymgr timer — so that a slow/large transfer or an unreachable peer can no longer stall heartbeats and trigger a spurious election.
+
+**Users**: Operators of multi-node controller clusters, who today see false failover under partition or large-snapshot conditions (the likely root of the bug-851 chaotic-failover family).
+
+**Impact**: Changes *how* drive-thread work is scheduled, not *what* the protocol does. `peer_ctrl_msg_handler`'s real work moves from inline-on-iomgr to the vtask pool; long blocking I/O (snapshot transfer, config-push, peer connect) moves to a new dedicated **slow-lane executor**; the delaymgr peer connect becomes bounded and offloaded. All existing consensus semantics, durability (persist-before-ACK), and the synchronous commit barrier are preserved unchanged.
+
+### Goals
+- No blocking operation executes inline on the iomgr loop or the delaymgr timer (R1, R2, R5).
+- Long blocking I/O runs on a bounded, admission-controlled lane isolated from the request vtask pool (R3).
+- Per-connection processing order and consensus correctness preserved when peer control moves to the pool (R4).
+- A network-partition (unreachable-peer) regression test proves heartbeat liveness and no spurious election (R7).
+- No regression of existing HA features; steady-state heartbeat path unchanged (R6, R8).
+
+### Non-Goals
+- Commit-barrier async (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` (P1).
+- Apply-after-commit / hook decoupling / facts-through-raft.
+- Asynchronous DNS resolution — deferred (HPC peers use IP/`/etc/hosts`); only `connect` is bounded here.
+- Snapshot *content*/format (entity-bearing snapshots) — this spec moves the *transfer* off the drive threads only.
+- The MEDIUM vtask-pool blockers (config-ack wait, 30 s checkpoint poll, term fsync on vote) beyond what the lane and existing `COMMIT_WAIT_RESERVED` already cover.
+- The admin `OP_upgradeConfig` push path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459) — it has a synchronous requester and stays a vtask-blocking operation under `COMMIT_WAIT_RESERVED`; only the requester-less iomgr transport push is offloaded.
+
+## Boundary Commitments
+
+### This Spec Owns
+- The scheduling discipline of drive-thread work: dispatching `peer_ctrl_msg_handler` to the vtask pool; the two-tier rule (fast protocol inline in the vtask, long I/O triggered to the slow lane).
+- A new **slow-lane executor** component: a bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight deduplication.
+- The bounded, offloaded peer-connect path (reorder of `socket_connect` + offloaded reconnect).
+- The network-partition regression scenario in the soak harness.
+
+### Out of Boundary
+- The commit engine and `repl_wait_majority_ack` (untouched; P1 owns its async conversion).
+- WAL/replication protocol content, snapshot format, config-upgrade materialization.
+- Election/vote protocol logic (consumed as-is; `election-iomgr-refactor` already made vote fan-out async).
+- The durability guarantee and synchronous commit barrier (preserved, not modified).
+
+### Allowed Dependencies
+- `ha-async-io.md` thread-role contract (authority).
+- Existing primitives reused, not modified in contract: `endpoint_block_tasks`/`endpoint_unblock_tasks` (per-endpoint serialization), lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (thread-safe epoll registration), `vtask_handle_msg`/VTASK_IO dispatch, `delay_task_ms` (non-blocking trigger only), `COMMIT_WAIT_RESERVED` (admission pattern), the `xce_probe_config_tag` bounded-connect pattern, the `fsync_worker`/`push_thread` background-thread precedent.
+- `election-iomgr-refactor` (completed upstream): `delay_task_ms`, `on_connected`, non-blocking vote fan-out.
+
+### Revalidation Triggers
+- Any change to the slow-lane submit contract or its admission cap.
+- Any change that makes a drive thread (iomgr/delaymgr) execute work synchronously again.
+- Changes to per-connection ordering semantics of peer-control dispatch (affects consensus correctness assumers: `ha-term-safety`, `raft-apply-decoupling`).
+- A change to `socket_connect`'s blocking/timeout behavior (affects all peer connection callers).
+
+## Architecture
+
+### Existing Architecture Analysis
+
+Three drive threads exist (`ha-async-io.md`): **iomgr** (single epoll loop), **vtaskmgr** (N-worker pool, default 8), **delaymgr** (single timer). The corrected thread-role contract (2026-06-25) requires iomgr and delaymgr to never block inline. Current violations:
+- `peer_ctrl_msg_handler` is registered directly as `ep->msg_handler` (ha.c:4265/4307/6349) → runs inline on iomgr; it triggers snapshot send (ha.c:2702 → config_repl.c:2446) and config-push (config_repl.c:1332) synchronously on iomgr.
+- `socket_connect` (netio.c:289/302) blocks (sets `O_NONBLOCK` only *after* connect, netio.c:716); reached from `primary_heartbeat_fn` on the delaymgr thread → an unreachable peer stalls all heartbeats.
+- The codebase's only deferral idiom, `delay_task_ms`, runs the deferred callback *on the delaymgr thread* — a liveness thread — so it cannot host blocking work under the new contract.
+
+### Architecture Pattern & Boundary Map
+
+Selected pattern: **dispatch-and-offload** — iomgr dispatches, the vtask pool does bounded protocol work, a new dedicated slow lane absorbs long blocking I/O; completions re-enter iomgr only through the existing lock-guarded registration call.
+
+```mermaid
+graph TB
+    subgraph drive[Drive threads - never block]
+        iomgr[iomgr epoll loop]
+        delaymgr[delaymgr timer]
+    end
+    subgraph pool[vtask pool - bounded work]
+        vtask[peer_ctrl vtask entry]
+    end
+    subgraph lane[slow-lane executor - dedicated bounded threads]
+        slow[blocking IO worker]
+    end
+    iomgr -->|dispatch VTASK_IO packet| vtask
+    vtask -->|fast protocol inline| vtask
+    vtask -->|trigger long IO| slow
+    delaymgr -->|trigger reconnect| slow
+    slow -->|blocking connect snapshot configpush| slow
+    slow -->|register connected fd via lock-guarded iomgr_monitor| iomgr
+```
+
+**Architecture Integration**:
+- Domain boundaries: iomgr = I/O + dispatch; vtask = bounded protocol handling (serialized per endpoint by `endpoint_block_tasks`); slow lane = long blocking I/O only; delaymgr = trigger-only.
+- Existing patterns preserved: VTASK_IO per-endpoint serialization, lock-guarded epoll registration, `delay_task_ms` as a non-blocking trigger, `COMMIT_WAIT_RESERVED` admission style.
+- New component rationale: the slow lane is the missing *non-liveness execution lane* — the one genuine gap; using dedicated threads (not the vtask pool) keeps long ops from starving request handling (R3 by construction).
+- Dependency direction: `netio` (lib) → `slowlane` → `ha`/`config_repl` (controller). Drive threads depend on the lane; the lane depends only on netio + libc.
+
+### Technology Stack
+
+| Layer | Choice / Version | Role in Feature | Notes |
+|-------|------------------|-----------------|-------|
+| Backend / Services | C (gnu17), pthreads | slow-lane executor; drive-thread dispatch changes | matches controller stack |
+| Messaging / Events | existing iomgr/vtaskmgr/delaymgr + new slow lane | thread-role-compliant scheduling | no new external dep |
+| Infrastructure / Runtime | epoll, `poll(2)`, pthread mutex/cond/sem | bounded connect, lane workers, admission | reuse existing patterns |
+| Test | soak harness + firewall DROP (iptables/nft) | partition (unreachable-peer) scenario | capability-gated; see Testing |
+
+## File Structure Plan
+
+### New Files
+```
+src/controller/
+├── slowlane.c             # Bounded slow-lane executor: submit, worker loop, admission cap, per-key dedup
+└── include/slowlane.h     # slow-lane public API (submit, init, fini)
+```
+
+### Modified Files
+- `src/lib/vtaskmgr.c` / `src/include/vtaskmgr.h` — **LivenessAdmission**: add `vtask_class_t` + a classified submit (`vtask_enq_classified`); dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus, fast-rejecting `VTASK_REQ_BLOCK` at the cap. Shared lib primitive → Revalidation Trigger.
+- `src/controller/commit.c` — fold `inflight_waits`/`COMMIT_WAIT_RESERVED` into the pool reservation: `commit_submit` submits a `VTASK_REQ_BLOCK` instead of self-counting.
+- `src/lib/netio.c` — `socket_connect`: set `O_NONBLOCK` before `connect`, bound with `poll(timeout)` (reuse `xce_probe_config_tag` pattern). Add a connect-timeout parameter/config.
+- `src/controller/ha.c` — register peer endpoints to dispatch peer-control through the vtask pool (revise `ha.c:4265/4307/6349`); convert `peer_ctrl_msg_handler` into a vtask entry that does fast protocol work inline and `slowlane_submit`s long ops; offload snapshot send (ha.c:2702) and reconnect (`primary_heartbeat_fn`/`ha_connect_to_standby`) to the slow lane with per-peer dedup.
+- `src/controller/config_repl.c` — **only the iomgr/promotion path**: run `push_config_transport_only` on the slow lane and replace the `wait_all_with_timeout` drive-thread join (reached from `peer_ctrl`/promotion at ha.c:1172/1177/2672/2677/3973) with a slow-lane-internal wait + non-blocking return. **In scope = the fire-and-forget transport push with NO waiting requester.** The admin `OP_upgradeConfig` path (`push_config_to_standbys` → `repl_wait_majority_ack`, config_repl.c:1459, "VTASK-CONTEXT ONLY") is **OUT of scope** — it has a synchronous admin requester awaiting the version/ACK reply and is a MEDIUM vtask-pool blocker bounded by `COMMIT_WAIT_RESERVED`; it must NOT be offloaded (doing so would break the synchronous admin reply).
+- `src/controller/cluster.c` (or `myapp.c`) — init/fini the slow-lane executor in the controller lifecycle; pass connect-timeout config.
+- `src/include/config.h` / `ha.h` — connect-timeout + slow-lane size/cap constants.
+- `.claude/skills/xce-test-ha/run-ha-soak.sh` (+ `xce_test`) — partition (DROP) primitive and the partition scenario.
+
+> Each file keeps one responsibility; `slowlane.c` is the only new logic module. peer-control logic stays in `ha.c` (moved to a vtask entry, not relocated to a new file) to preserve reviewability against existing tests.
+
+## System Flows
+
+### Reconnect to an unreachable peer (R2)
+```mermaid
+sequenceDiagram
+    participant DM as delaymgr timer
+    participant SL as slow lane
+    participant IO as iomgr
+    DM->>DM: heartbeat tick - peer disconnected
+    DM->>SL: slowlane_submit reconnect peer if not in-flight
+    DM-->>DM: return immediately - other peers still heartbeat
+    SL->>SL: nonblocking connect + poll bounded timeout
+    alt connect succeeds
+        SL->>SL: prepare endpoint set msg_handler
+        SL->>IO: iomgr_monitor lock-guarded register fd
+    else timeout or refused
+        SL->>SL: clear in-flight - retry next tick
+    end
+```
+Key decisions: delaymgr only triggers (idempotent per-peer flag); the connect is bounded so it never approaches the heartbeat timeout; registration reuses the lock-guarded `iomgr_monitor` (no new primitive).
+
+### Snapshot send triggered by a catch-up request (R1)
+```mermaid
+sequenceDiagram
+    participant IO as iomgr
+    participant VT as peer_ctrl vtask
+    participant SL as slow lane
+    IO->>VT: dispatch repl-ack packet VTASK_IO
+    VT->>VT: fast - decode, update match_index
+    VT->>SL: slowlane_submit snapshot if not in-flight per standby
+    VT-->>IO: handler returns - endpoint unblocked, peer messages flow
+    SL->>SL: persist_checkpoint + fread chunk loop + send
+```
+Key decisions: the vtask must **not** hold `ep->busy` across the transfer (R4.3) — it triggers and returns; per-standby dedup prevents concurrent transfers (R1.5).
+
+## Requirements Traceability
+
+| Requirement | Summary | Components | Flows |
+|-------------|---------|------------|-------|
+| 1.1–1.4 | liveness during snapshot/config I/O | SlowLane, PeerCtrlDispatch | Snapshot flow |
+| 1.5 | no concurrent snapshot per standby | SlowLane (dedup) | Snapshot flow |
+| 2.1–2.2 | unreachable peer doesn't stall heartbeats | BoundedConnect, ReconnectOffload | Reconnect flow |
+| 2.3 | bounded connect timeout | BoundedConnect | — |
+| 2.4–2.5 | retry without self-terminate; dedup | ReconnectOffload, SlowLane | Reconnect flow |
+| 2.6 | connect success registers without loss/reorder | ReconnectOffload (iomgr_monitor) | Reconnect flow |
+| 3.1–3.4 | admission protection; long IO off pool/drive | SlowLane (cap), LivenessAdmission (pool reservation), PeerCtrlDispatch | — |
+| 4.1–4.2 | per-connection processing order | PeerCtrlDispatch (endpoint_block_tasks) | — |
+| 4.3 | don't hold block across long IO | PeerCtrlDispatch | Snapshot flow |
+| 4.4 | iomgr-state mutation via lock-guarded iface | ReconnectOffload (iomgr_monitor) | Reconnect flow |
+| 4.5–4.6 | preserve vote/ack/term + persist-before-vote | PeerCtrlDispatch, LivenessAdmission (ack-progress reservation) | — |
+| 5.1–5.4 | thread-role contract compliance | all | — |
+| 6.1–6.7 | no regression of existing HA features | all (regression suite) | — |
+| 7.1–7.4 | partition verification | PartitionTest | — |
+| 8.1–8.2 | heartbeat fast-path unchanged | PeerCtrlDispatch | — |
+
+## Components and Interfaces
+
+| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
+|-----------|-------|--------|--------------|------------------|-----------|
+| SlowLane | controller/infra | bounded executor for long blocking I/O | 1, 3 | pthreads, sem (P0) | Service, Batch |
+| LivenessAdmission | lib/vtaskmgr | pool-level task-class reservation: liveness work always has a worker | 3, 4.5 | vtaskmgr dispatcher (P0) | Service |
+| PeerCtrlDispatch | controller/ha | move peer-control to vtask; two-tier | 1,4,5,8 | vtaskmgr, LivenessAdmission, endpoint_block_tasks (P0) | Service |
+| BoundedConnect | lib/netio | non-blocking + poll-bounded connect | 2 | poll (P0) | Service |
+| ReconnectOffload | controller/ha | delaymgr triggers offloaded reconnect | 2 | SlowLane, iomgr_monitor (P0) | Service |
+| PartitionTest | test | unreachable-peer regression | 7 | soak harness (P1) | Batch |
+
+### Controller / Infrastructure
+
+#### SlowLane
+
+| Field | Detail |
+|-------|--------|
+| Intent | Bounded, admission-capped pool that runs long blocking I/O off the drive threads, with per-key in-flight dedup |
+| Requirements | 1.1, 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4 |
+
+**Responsibilities & Constraints**
+- Owns a fixed set of dedicated worker threads (sized to peer count; small constant) — isolated from the vtask request pool, so long ops cannot starve request handling.
+- `slowlane_submit` is non-blocking (enqueue + return); callers on drive threads never wait.
+- Enforces an admission cap on concurrent in-flight tasks; excess is queued, never run on a drive thread (R3.3/3.4).
+- Per-key dedup: a submit with an in-flight key is a no-op (R1.5 snapshot-per-standby; R2.5 reconnect-per-peer).
+- Does not own the work content (connect, snapshot, config-push bodies stay in their modules); it owns scheduling/isolation.
+
+**Dependencies**
+- Inbound: PeerCtrlDispatch — submits long ops (P0); ReconnectOffload — submits reconnect (P0).
+- Outbound: `iomgr_monitor` (via the submitted task) — register connected fd (P0).
+- External: pthreads (threads, mutex, cond, sem) (P0).
+
+**Contracts**: Service [x] / Batch [x]
+
+##### Service Interface
+```c
+typedef void (*slowlane_fn_t)(void *arg);
+
+int  slowlane_init(slowlane_t *sl, int n_workers, int max_inflight);
+/* submit returns 0 (enqueued), -EEXIST (key in flight, deduped), -EAGAIN (cap full, queued/deferred);
+   never blocks the caller. key==0 disables dedup. */
+int  slowlane_submit(slowlane_t *sl, uint64_t key, slowlane_fn_t fn, void *arg);
+void slowlane_fini(slowlane_t *sl);   /* drain + join on shutdown only */
+```
+- Preconditions: `fn` performs the blocking work and is responsible for clearing any caller-side in-flight state on completion.
+- Postconditions: `fn` runs exactly once on a lane worker; never on iomgr/delaymgr.
+- Invariants: concurrent running tasks ≤ `max_inflight`; one in-flight task per non-zero key.
+
+**Snapshot/checkpoint concurrency invariant (Phase 3, R1.1/1.3/1.5 — MUST hold before snapshot offload lands)**
+Moving `persist_checkpoint` (which runs `trim_wal_segments` — *deletes* WAL segments) + the snapshot `fread` loop onto a lane worker runs them concurrently with live WAL appends, the `fsync_worker` flusher, and the WAL-catchup stream. The lane snapshot path must obey:
+- **Frozen boundary**: the snapshot is taken at a fixed `(term, lsn)` checkpoint boundary captured at submit time; the `fread`/send loop reads only data at or below that boundary, never live tail bytes.
+- **Trim safety (MECHANISM, not just goal — see bug-1147)**: `trim_wal_segments` (`persist.c:1374`) TODAY bounds deletion ONLY by `active_seg_lsn` (the open segment) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** snapshot boundary, and there is **no** working snapshot backfill (`write_snapshot` emits only a 16-byte P0 sentinel, not materialized state). So a standby lagging by more than ~one open segment already hits a permanent committed-prefix gap (the bug-795/828 family). Moving checkpoint/trim onto a concurrent lane worker makes this latent race *easier* to hit. Therefore this spec MUST add the cursor-bound guard as a stopgap: `trim_wal_segments` computes `floor = min(all standby acked_lsn, all in-flight snapshot boundaries)` under `p->lock` and refuses to delete any segment whose range reaches `floor` or above — never holding `p->lock` across the `fread`/send I/O. This is strategy-A retention discipline that *should* exist regardless of this spec; the full fix for empty/far-behind join (cursor=0) requires entity-bearing snapshots (separate spec) and is explicitly NOT solved here. The design must not assume "a snapshot will backfill the gap" — that path is not built.
+- **Single in-flight checkpoint**: at most one checkpoint/trim in flight (lane dedup key = checkpoint), and at most one snapshot per standby (dedup key = standby id, R1.5).
+- **No correctness change**: the lane only relocates execution; the checkpoint/trim/snapshot *content* is byte-identical to the current inline path.
+
+**Implementation Notes**
+- Integration: init in controller lifecycle (cluster.c/myapp.c); size from peer count.
+- Validation: assert submit never called from a context expecting completion; metric on queue depth/age (fail-loud on backlog); a test asserting concurrent append + checkpoint/trim produces no segment-below-boundary deletion and a consistent snapshot.
+- Risks: a lane worker blocked indefinitely (e.g., snapshot to a half-open peer) — **bound every in-lane blocking op**: the connect uses the bounded `socket_connect`; the snapshot send must use a bounded socket write (write timeout, mirroring the bounded-connect approach) so a standby going unreachable mid-transfer cannot pin a lane worker. Otherwise stuck transfers exhaust the bounded lane and delay reconnects (which share it). Phase-3 task constraint; size/segment the lane so reconnect and transfer do not starve each other.
+
+#### LivenessAdmission (lib/vtaskmgr)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Make "keep K workers free for liveness" a property of the vtask **pool**, not of the commit engine — so it covers every blocking op the pool runs once peer-control moves onto it, not just commit-waits |
+| Requirements | 3.1, 3.2, 3.3, 4.5 |
+
+**Why this exists (the gap it closes):** Today `vtaskmgr` (`include/vtaskmgr.h`) has a single FIFO `taskq`, N `vcpu` workers, and **no priority/reservation**; a blocking task holds its `vcpu` in `VCPU_BUSY` for the whole block (`vtaskmgr.c:248-275`). The only admission anywhere is `commit.c`'s `inflight_waits` counter (`COMMIT_WAIT_RESERVED=2`, `commit.c:200`) — external to vtaskmgr and counting **only** commit-waits. Once `peer_ctrl` moves onto the pool (PeerCtrlDispatch), the pool also runs the vote-grant term fsync (R4.6) and config-ack waits — blocking ops **invisible** to the commit counter. So "≥2 workers free of commit-waits" no longer means "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. The reservation must therefore become pool-level.
+
+**Responsibilities & Constraints**
+- Classify every vtask at enqueue into `VTASK_LIVENESS` (never parks — WAL-ack decode, heartbeat send, vote *receipt/decode*; `commit_on_ack` is documented never-block, `commit.c:16`), `VTASK_LIVE_BLOCK` (bounded mandatory park — vote-grant term fsync, R4.6), or `VTASK_REQ_BLOCK` (may park seconds — commit-wait, config-ack wait). Class is knowable from the message type at the dispatch shim.
+- The dispatcher reserves `VTASK_LIVENESS_RESERVED` vcpus: a `VTASK_REQ_BLOCK` task is **not admitted** when `count(vcpus running *_BLOCK) >= n_vcpus - VTASK_LIVENESS_RESERVED` — it **fast-rejects with `-XCE_ERR_NOT_READY`** (preserving today's client backpressure, Req 7.1/7.2), it does **not** silently queue or park a worker.
+- `VTASK_LIVENESS` tasks are **never** admission-gated → always find a reserved vcpu → acks always advance commit → parked `*_BLOCK` tasks always wake. **Deadlock closed by construction.**
+- `VTASK_LIVE_BLOCK` (vote fsync) runs in the reserved headroom and is bounded, so a vote response never queues behind a seconds-long commit-wait (removes a second spurious-election vector).
+- `commit.c`'s `inflight_waits`/`COMMIT_WAIT_RESERVED` is **folded into** this — `commit_submit` becomes "submit a `VTASK_REQ_BLOCK`"; the pool owns the single reservation (`commit_engine_init` no longer needs `n_vcpus`, `cluster.c:1332`).
+
+**Dependencies**: Inbound — PeerCtrlDispatch, commit (P0). Outbound — none (lib-internal). External — pthreads.
+
+**Contracts**: Service [x]
+```c
+typedef enum { VTASK_LIVENESS=0, VTASK_LIVE_BLOCK, VTASK_REQ_BLOCK } vtask_class_t;
+/* submit a classified task; VTASK_REQ_BLOCK returns -XCE_ERR_NOT_READY when the
+   reservation would be violated (retriable, never parks a worker). */
+int vtask_enq_classified(vtaskmgr_t*, vtask_t*, vtask_class_t);
+```
+- Invariant: `count(vcpus in *_BLOCK) <= n_vcpus - VTASK_LIVENESS_RESERVED`; ≥`VTASK_LIVENESS_RESERVED` vcpus always available to `VTASK_LIVENESS`.
+
+**Implementation Notes**
+- **Sizing (peer-scaled):** `R = VTASK_LIVENESS_RESERVED = clamp(n_peers − 1, 1, n_vcpus − 1)`; commit/`REQ_BLOCK` cap = `n_vcpus − R`. Rationale: per-endpoint serialization (`endpoint_block_tasks`) caps in-flight tasks at one per peer connection, so worst-case concurrent liveness work is **N−1** (one ack/vote per other peer) — reserving N−1 means no peer's liveness message ever queues behind a busy reserved worker. **Correctness needs only the floor of 1** (one guaranteed-free worker + liveness-never-gated + sub-ms cycling = deadlock-free regardless of peer count); the peer-scaled middle is a *latency* optimization. **Ceiling `n_vcpus − 1`** keeps ≥1 `REQ_BLOCK` slot so commits can still run (cap never 0). Both `n_peers` and `n_vcpus` are final at init (`cluster.c:1332` already passes both). Config-overridable. On a degenerate small-core box (`n_vcpus < n_peers`) R clamps to `n_vcpus − 1` (full per-peer reservation impossible) — still correct via fast cycling; emit a loud WARN at init so the under-provisioning is operator-visible. Validate the chosen `R` at supported sizes (R8).
+- Validation: misclassification is the one failure mode — add a debug assert/loud-WARN if a `VTASK_LIVENESS` task parks beyond a threshold; task 3.4 (acks advance commit under saturation) is the regression oracle.
+- Risk: a future blocking path tagged `VTASK_LIVENESS` silently breaks the guarantee — review discipline + the assert + the saturation test.
+- **Revalidation Trigger**: any change to the class set, the reservation size, or the dispatch admission rule (consensus correctness + the slow-lane admission contract depend on it).
+
+#### PeerCtrlDispatch
+
+| Field | Detail |
+|-------|--------|
+| Intent | Dispatch peer-control message handling to the vtask pool; fast work inline, long work to the lane |
+| Requirements | 1.4, 4.1, 4.2, 4.3, 4.5, 4.6, 5.1, 5.4, 8.1, 8.2 |
+
+**Responsibilities & Constraints**
+- Peer endpoints dispatch through the existing VTASK_IO path (the message is packet-triggered → reuses `vtask_handle_msg`), inheriting `endpoint_block_tasks` per-endpoint serialization (R4.1/4.2).
+- The vtask entry performs only bounded protocol work inline (decode, match_index update, commit eval, send next heartbeat) and `slowlane_submit`s any long op; it must not hold `ep->busy` across long I/O (R4.3).
+- Preserves existing vote/ack/term semantics and persist-before-vote ordering (R4.5/4.6) — the logic is unchanged, only its execution context moves.
+- **Commit-advance preservation (R4.5):** WAL-ack handling — which wakes the commit barrier (`repl_wait_majority_ack` / `commit_set_raft_commit_index`) — moves off iomgr to the vtask pool. The design preserves that a quorum of acks still advances `committed_lsn` promptly: ack handling is bounded fast work (never offloaded to the slow lane), per-endpoint serialization does not delay *cross-endpoint* acks (each peer's acks process independently), and the commit-advance path itself is untouched. This is the only interaction with P1 (`raft-apply-decoupling`, which owns the barrier's async conversion); this spec must not change ack→commit-advance semantics, only the thread the ack is decoded on.
+
+- **Deadlock-freedom argument (R4.5/R3.1 — REQUIRED reasoning, reverses a documented invariant):** Today `commit_on_ack` is explicitly "called from the iomgr read path and never blocks" (`commit.c:16`, `repl.c:220`) — the ack that advances `committed_lsn` and the jobsub vtasks blocked in `commit_submit(COMMIT_MODE_WAIT)` (the synchronous barrier this spec PRESERVES) run on **different** threads. Moving peer-control to the vtask pool **co-locates the wakers (acks) and the waiters (blocked commit-submits) on one pool** — exactly the configuration the iomgr placement was chosen to avoid. Safety rests on a worker reservation — but the existing `commit.c` cap is **insufficient as-is**: `COMMIT_WAIT_RESERVED` (`commit.c:200`) counts **only** commit-waits, while moving `peer_ctrl` onto the pool also puts the vote-grant term fsync (R4.6, `VTASK_LIVE_BLOCK`) and config-ack waits on it — blocking ops the commit counter can't see. "≥2 workers free of commit-waits" therefore does **not** mean "≥2 workers free for acks": a burst of vote-fsyncs can occupy the headroom and starve the ack that wakes the commit-waiters. **Closing the deadlock requires the LivenessAdmission component** (above) — the reservation must become a pool-level property covering every blocking class, with `VTASK_LIVENESS` (acks) never admission-gated. Required invariants this spec must hold (and test): (1) WAL-ack/commit-advance decode is tagged `VTASK_LIVENESS`, never routed to the slow lane, never gated — admissible independently of any blocking count; (2) the reserved headroom is sized for all concurrent liveness work (N-peer acks + vote responses + heartbeat sends), so the reservation — not luck — guarantees an ack always finds a worker. This co-location is **transient by design**: `raft-apply-decoupling` P1 converts `repl_wait_majority_ack` to event-driven (no parked thread per commit), removing the `VTASK_REQ_BLOCK` waiters from the pool entirely — but until P1 lands, correctness depends on LivenessAdmission holding. **Dependency: PeerCtrlDispatch (3.1) requires LivenessAdmission to land first (Phase 1).** Gate: task 3.4 (quorum-of-acks advances `committed_lsn` with the pool saturated by `*_BLOCK` tasks) is the **admission gate for 3.1** — 3.1 is not "done" until 3.4 is green.
+
+**Dependencies**
+- Inbound: iomgr — dispatches the packet (P0).
+- Outbound: vtaskmgr (P0); SlowLane — long ops (P0); existing commit/repl/persist logic (P0).
+
+**Contracts**: Service [x]
+
+**Implementation Notes**
+- Integration: change peer endpoint `msg_handler` registration (ha.c:4265/4307/6349) to the vtask-dispatching shim; convert `peer_ctrl_msg_handler` body into the vtask entry.
+- Validation: existing election/heartbeat/repl tests must pass unchanged (R6); assert no long op runs inline in the entry.
+- Risks: a heavy op accidentally left inline → per-peer head-of-line blocking (guard by review + a test asserting the handler returns promptly).
+
+#### BoundedConnect (lib/netio)
+
+| Field | Detail |
+|-------|--------|
+| Intent | Make `socket_connect` non-blocking + `poll`-bounded | 
+| Requirements | 2.3 |
+
+**Responsibilities & Constraints**: set `O_NONBLOCK` before `connect`; on `EINPROGRESS`, `poll` up to a configured timeout; return success/timeout/error. Reuses the `xce_probe_config_tag` pattern.
+**Contracts**: Service [x]
+```c
+/* connect_timeout_ms: bound the attempt; 0 = use default (config). */
+int socket_connect(const char *host, int port, int connect_timeout_ms);
+```
+**Implementation Notes**: callers updated to pass the timeout; default in config.h. This is a behavior change to a shared helper → Revalidation Trigger for all connect callers.
+
+#### ReconnectOffload (controller/ha)
+
+| Field | Detail |
+|-------|--------|
+| Intent | delaymgr triggers an offloaded, deduped, bounded reconnect | 
+| Requirements | 2.1, 2.2, 2.4, 2.5, 2.6 |
+
+**Responsibilities & Constraints**: `primary_heartbeat_fn` (delaymgr) detects a disconnected peer and `slowlane_submit`s the reconnect (per-peer key for dedup); the lane worker does the bounded connect, prepares the endpoint, registers it via lock-guarded `iomgr_monitor`; on failure clears the in-flight flag and lets the next tick retry (never self-terminates).
+**Contracts**: Service [x]
+**Implementation Notes**: refcount the endpoint across the in-flight window; resolve the inbound-vs-outbound connection race per the existing connection-ownership rules (`ha-async-io.md`).
+
+### Test
+
+#### PartitionTest
+
+| Field | Detail |
+|-------|--------|
+| Intent | Make one peer unreachable (blackhole), assert liveness + no spurious election + heal | 
+| Requirements | 7.1, 7.2, 7.3, 7.4 |
+
+**Contracts**: Batch [x]
+**Implementation Notes**: the partition primitive must render one peer **unreachable** (traffic dropped, connect hangs), distinct from `kill -9`. Two mechanisms, **primary = in-process** so R7 actually runs in this project's environment (the sibling specs `ha-term-safety` 5.2/5.4 are already perpetually ENV-BLOCKED on the unprivileged/overlayfs sandbox; a firewall-only R7 would be a permanent skip and the spec's headline guarantee would never be exercised):
+- **Primary (portable, no privilege): in-process socket-drop hook in `xce_test`** — a test-only seam (`XCE_TESTING`) that makes one node silently drop/blackhole inbound+outbound peer traffic at the endpoint layer (e.g. stop reading/writing the peer fd, or a drop flag in the I/O path), simulating "unreachable" without any kernel/firewall change. Runs anywhere the unit/soak suite runs.
+- **Secondary (infra-level, optional): firewall `iptables`/`nft` DROP** on the node's port for real-network fidelity; capability-gated with a **loud skip** when unprivileged. Complements, does not replace, the in-process path.
+
+Assert (both): heartbeats to survivors continue within the interval, no new-leader election while the leader holds majority, peer rejoins on heal, zero committed-job loss.
+
+## Error Handling
+
+### Error Strategy
+- **Connect timeout/refused** → bounded `poll` returns; lane clears in-flight; delaymgr retries next tick; loud WARN with peer + elapsed (fail-loud, never abort — controllers never self-terminate).
+- **Lane backlog** → submit returns `-EAGAIN` when the cap is reached; the work is queued, never run on a drive thread; emit a metric/WARN on queue depth + oldest-age so a saturated lane is operator-visible.
+- **Snapshot/config-push failure on the lane** → reported through the existing repl/config error paths unchanged; lane only relocates execution, not error semantics.
+- **Dedup race** → submit with an in-flight key is a no-op (`-EEXIST`); idempotent.
+
+### Monitoring
+- Metrics: lane queue depth, oldest-task age, in-flight count, per-peer reconnect attempts. WARN thresholds make a degraded/partitioned window visible (consistent with fail-loud).
+
+## Testing Strategy
+
+### Unit Tests
+- SlowLane: `slowlane_submit` is non-blocking; admission cap bounds concurrent tasks; per-key dedup returns `-EEXIST`; `fn` runs exactly once (3.1–3.4, 1.5).
+- BoundedConnect: `socket_connect` returns within the timeout against an unreachable host; succeeds against a reachable one (2.3).
+- PeerCtrlDispatch: the vtask entry returns promptly (no long op inline); per-endpoint serialization holds via `endpoint_block_tasks` (4.1–4.3).
+
+### Integration Tests
+- Reconnect offload: with one peer unreachable, delaymgr ticks continue and heartbeats to reachable peers are sent within the interval; reconnect is deduped per peer (2.1, 2.2, 2.5).
+- Snapshot offload: a catch-up triggering a snapshot does not block iomgr; other peers' heartbeats/votes processed during the transfer; no concurrent transfer per standby (1.1, 1.3, 1.4, 1.5).
+- Commit-advance under pool load (R4.5): with the vtask pool near saturation, a quorum of WAL-acks still advances `committed_lsn` promptly; cross-endpoint acks are not serialized behind one another.
+- Snapshot/checkpoint concurrency (R1.1/1.3/1.5): concurrent live appends + a lane checkpoint/trim never delete a segment below an in-flight snapshot boundary or a standby catch-up cursor; the produced snapshot is consistent.
+- Regression sweep (R6): run the existing suites unchanged — `test_ha_election`, `test_ha_prevote`, `test_ha_liveness`, `test_ha_failover`, `test_ha_rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability`; all pass without behavior changes (6.1–6.7).
+
+### E2E / Soak Tests
+- Partition soak (R7): blackhole one peer → assert heartbeats to survivors continue, no spurious election while leader holds majority, peer rejoins on heal, zero committed-job loss (7.1–7.4).
+- Failover soak (existing, kill-based) continues to pass (6.3).
+
+### Performance
+- Steady-state heartbeat cadence at 3–5 controllers shows no added latency attributable to the new dispatch path; the heartbeat/vote send paths remain non-blocking enqueues (8.1, 8.2).
+
+## Migration Strategy
+
+Phased, each phase independently testable (matches research recommendation):
+
+**Ordering rule:** the slow lane is foundational — every offload (reconnect, snapshot, config-push) submits to it, so it must exist first. And the peer_ctrl→vtask move must land **together with** its long-op offload in the same phase: moving peer_ctrl to a vtask while still running snapshot/config-push inline would hold `endpoint_block_tasks` (`ep->busy`) across multi-second I/O (transient R4.3 violation) and consume a pool worker. The two are therefore coupled.
+
+```mermaid
+flowchart TD
+    P1[Phase 1: SlowLane executor + BoundedConnect] --> P2[Phase 2: ReconnectOffload sender-side]
+    P2 --> P3[Phase 3: PeerCtrlDispatch to vtask PLUS snapshot and configpush offload coupled]
+    P3 --> P4[Phase 4: PartitionTest + regression sweep]
+```
+
+- **Phase 1** (foundational, lowest risk): build the SlowLane executor (bounded pool + admission + dedup — new component, no behavior change yet, unit-testable in isolation); reorder `socket_connect` to bounded non-blocking + `poll` (independent; already bounds the delaymgr stall even before reconnect is offloaded); **and add LivenessAdmission to vtaskmgr** (task-class reservation; fold in `COMMIT_WAIT_RESERVED`). LivenessAdmission is a **hard prerequisite of Phase 3** — peer_ctrl cannot move onto the pool deadlock-safely without it — so it must land in Phase 1 as dormant infra (no behavior change until peer_ctrl dispatches through it).
+- **Phase 2**: offload the delaymgr peer reconnect to the lane (deduped per peer, registered via lock-guarded `iomgr_monitor`) → closes sender-side heartbeat starvation. Shippable.
+- **Phase 3** (coupled — the R4.3-safe step): dispatch peer-control to the vtask pool **and** offload the long ops it triggers (snapshot send, config-push transport-only) to the lane in the same change, so no phase ever runs long I/O inline in a blocked vtask. Carries the snapshot/checkpoint concurrency invariant (frozen `(term,lsn)` boundary, trim-vs-catch-up-cursor safety, single in-flight checkpoint) and the bounded in-lane snapshot write.
+- **Phase 4**: partition test + full regression sweep.
+- Rollback: each phase is behavior-preserving; a phase can be reverted independently since the contract (non-blocking drive) is the only invariant added. P1 adds dormant infra; the first observable change is P2.
--- a/.kiro/specs/raft-drive-async/requirements.md
+++ b/.kiro/specs/raft-drive-async/requirements.md
@ -0,0 +1,111 @@
+# Requirements Document
+
+## Introduction
+
+The XCE controller's Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals — the iomgr epoll loop and the delaymgr timer thread. A blocking audit (2026-06-25) found three CRITICAL violations of the corrected `ha-async-io.md` thread-role contract: snapshot send + checkpoint inline on iomgr (`ha.c:2702`, `config_repl.c:2446`), config-push busy-poll + join inline on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr heartbeat thread (`netio.c:289/302`). Root cause: `peer_ctrl_msg_handler` is registered directly as the iomgr message handler (`ha.c:4265/4307/6349`) instead of dispatching to the vtask pool as normal cluster messages do (`cluster.c:711`). When a drive thread stalls, heartbeats/votes/ACKs stop flowing, peers declare the leader dead, and a spurious election follows — the likely root of the bug-851 chaotic-failover family.
+
+This feature brings the controller into compliance with the steering thread-role contract (`ha-async-io.md`, corrected 2026-06-25) so that no liveness-carrying thread blocks inline, while preserving every existing HA behavior and the persist-before-ACK / synchronous-commit policies. It is verified by a new network-partition (peer unreachable, not merely killed) regression scenario that the current process-kill-only soak cannot reproduce.
+
+The authoritative subject for the acceptance criteria below is **the XCE controller** and its named drive threads (iomgr loop, delaymgr timer, vtask pool).
+
+## Boundary Context
+
+- **In scope**: keeping the iomgr loop and delaymgr timer non-blocking; moving long replication/config I/O (snapshot transfer, config-push) off the drive threads to an admission-controlled execution path; bounding and offloading the peer connect; preserving per-connection message ordering and processing correctness; admission protection so liveness-critical message handling is never starved; a network-partition regression test; no regression of existing HA features.
+- **Out of scope**: the commit-barrier async conversion (`repl_wait_majority_ack` → event-driven `resolve`) — owned by `raft-apply-decoupling` as prerequisite P1; the apply-after-commit / hook-decoupling refactor; asynchronous DNS resolution (deferred unless measured — HPC peers use IP/`/etc/hosts`); the snapshot content/format (entity-bearing snapshots is a separate spec) — this feature only moves the snapshot *transfer* off the drive threads; the MEDIUM vtask-pool blockers beyond what admission control covers.
+- **Adjacent expectations**: relies on the corrected `ha-async-io.md` thread-role contract as the authority; reuses the existing `COMMIT_WAIT_RESERVED` admission cap and the `xce_probe_config_tag` bounded-connect pattern; consumes `election-iomgr-refactor` (completed) as upstream for its primitives (`delay_task_ms`, `on_connected`, non-blocking vote fan-out) and stays a standalone spec (scope-check 2026-06-25: ADJACENT-SAME-AREA — that spec fixed startup blocking, this fixes runtime drive-thread blocking); touches the same `peer_ctrl_msg_handler`/repl-dispatch region as `ha-term-safety`; preserves the durability and synchronous-commit guarantees owned by `ha-commit-durability` and `ha-commit-engine`.
+
+## Requirements
+
+### Requirement 1: Liveness During Long-Running Replication and Configuration I/O
+
+**Objective:** As a cluster operator, I want the controller to keep heartbeating and processing protocol messages while it transfers a snapshot or pushes a configuration to a standby, so that a large or slow transfer to one peer does not trigger a false failover of the cluster.
+
+#### Acceptance Criteria
+
+1. While the controller is transferring a snapshot to one standby, the controller shall continue to send heartbeats to every other peer within the configured heartbeat interval (default 3000 ms).
+2. While the controller is pushing a configuration to standbys, the controller shall continue to send heartbeats to every peer within the configured heartbeat interval and continue to process inbound votes and acknowledgments.
+3. When the controller initiates a snapshot transfer, checkpoint creation, or configuration push, the iomgr loop shall remain available to dispatch other inbound messages without waiting for that operation to complete.
+4. While a long-running replication or configuration operation is executing, the controller shall continue to process inbound RequestVote messages and respond within the configured election timeout (default 5000 ms).
+5. If a snapshot transfer to one standby is already in progress, then the controller shall not start a second concurrent transfer to that same standby.
+
+### Requirement 2: Liveness and Bounded Behavior When a Peer Is Unreachable
+
+**Objective:** As a cluster operator, I want an unreachable peer to never stall heartbeats to the healthy peers, so that one partitioned node cannot cause the surviving majority to elect a new leader unnecessarily.
+
+#### Acceptance Criteria
+
+1. While one peer is unreachable (its host accepts no connection or silently drops traffic), the controller shall continue to send heartbeats to all reachable peers within the configured heartbeat interval.
+2. While one peer is unreachable, the delaymgr timer thread shall continue to fire periodic and election-timeout callbacks without being delayed by the connection attempt to the unreachable peer.
+3. When the controller attempts to connect to a peer, the controller shall bound the connection attempt to a configured timeout and report failure on expiry rather than waiting for the operating-system connect timeout.
+4. If a connection attempt to a peer fails or times out, then the controller shall retry the connection later without terminating, consistent with the "controllers never self-terminate" invariant.
+5. When the controller is already attempting to (re)connect to a given peer, the controller shall not enqueue a duplicate concurrent connection attempt for that same peer.
+6. When a connection attempt to a peer succeeds, the controller shall register and begin using the new connection without losing or reordering messages already queued for that peer.
+
+### Requirement 3: Worker-Pool Admission Protection
+
+**Objective:** As a cluster operator, I want offloaded blocking work to never consume all worker capacity, so that liveness-critical message handling always has a worker available even under replication or reconnection load.
+
+#### Acceptance Criteria
+
+1. The controller shall reserve worker capacity such that liveness-critical message handling (heartbeats, votes, acknowledgments) always has at least one available worker even when blocking offloaded tasks are in flight.
+2. While the maximum number of admitted blocking offloaded tasks is in flight, the controller shall continue to admit and process non-blocking liveness-critical messages.
+3. When the number of concurrently requested blocking offloaded tasks exceeds the admission limit, the controller shall queue or defer the excess rather than block a liveness-carrying thread.
+4. The controller shall route long/unbounded blocking I/O (snapshot transfer, configuration push, peer reconnect) to an execution path that is subject to the admission limit, not to the iomgr loop or the delaymgr timer thread.
+
+### Requirement 4: Message Ordering and Processing Correctness Preserved
+
+**Objective:** As a cluster operator, I want peer control messages to be processed with the same ordering and correctness as today after they are moved off the iomgr loop, so that consensus correctness is unaffected by the threading change.
+
+#### Acceptance Criteria
+
+1. When peer control messages arrive on a single connection, the controller shall handle (process/apply) them in arrival order, even though handling is dispatched to the worker pool rather than run inline on the iomgr loop. (This concerns inbound *processing* order, distinct from the already-guaranteed outbound *send* order of the endpoint output queue.)
+2. While handling a peer control message for a given connection, the controller shall not begin handling another message from that same connection until the first completes, so per-connection processing remains strictly sequential.
+3. While handling a peer control message, the controller shall not hold its connection's sequential-processing exclusivity across a long-running offloaded operation (snapshot transfer, configuration push); such operations shall be triggered and the handler shall return, so the connection's subsequent messages (including its heartbeat acknowledgments) are not queued behind the long operation.
+4. When peer control message processing mutates iomgr-owned connection or epoll state, the controller shall do so only through the thread-safe (lock-guarded) registration interface, without data races against the iomgr event loop.
+5. The controller shall preserve the existing semantics of vote handling, acknowledgment handling, and term updates after relocating peer control message processing off the iomgr loop.
+6. The controller shall continue to persist term and vote state before responding to a vote, preserving the existing durability ordering on the vote path.
+
+### Requirement 5: Thread-Role Contract Compliance
+
+**Objective:** As a maintainer, I want the controller's drive threads to obey the steering thread-role contract by construction, so that future changes cannot reintroduce inline blocking on a liveness thread.
+
+#### Acceptance Criteria
+
+1. The iomgr loop shall perform only non-blocking I/O and message dispatch; it shall not execute blocking calls or inline message processing.
+2. The delaymgr timer thread shall only fire timers and trigger work; it shall not execute blocking calls (connect, file I/O, or busy-wait sleeps).
+3. The controller shall execute bounded blocking work in the worker pool only under the admission reservation, and long/unbounded blocking I/O only on the dedicated offload path.
+4. The controller shall conform to the thread-role contract defined in `ha-async-io.md` for all drive-thread code paths brought into scope by this feature.
+
+### Requirement 6: No Regression of Existing HA Features
+
+**Objective:** As a cluster operator, I want all existing HA behaviors to keep working unchanged after the drive is made non-blocking, so that the hardening introduces no functional regression.
+
+#### Acceptance Criteria
+
+1. When the existing controller test suites for election/voting, heartbeat/liveness, failover, snapshot recovery, configuration push/upgrade, WAL replication/catch-up, and the commit barrier are run, the controller shall pass them without modification to their asserted behavior.
+2. The controller shall continue to elect a single primary on quorum, increment terms, and reject stale-term votes exactly as the existing election tests assert.
+3. The controller shall continue to detect a dead primary and fail over within the configured heartbeat timeout (default 10000 ms) exactly as the existing liveness/failover tests assert.
+4. The controller shall continue to deliver snapshots and replicated WAL entries to standbys and recover them on restart exactly as the existing replication and recovery tests assert.
+5. The controller shall continue to push and apply configuration changes through majority acknowledgment exactly as the existing config-replication tests assert.
+6. The controller shall preserve the persist-before-ACK durability guarantee and the synchronous commit-barrier behavior, which this feature does not modify.
+7. If any existing HA regression test fails after the change, then the feature shall be considered incomplete until the behavior is restored.
+
+### Requirement 7: Network-Partition Verification
+
+**Objective:** As a cluster operator, I want a regression test that makes a peer unreachable (not merely killed), so that the heartbeat-starvation failure mode this feature fixes is provably closed and stays closed.
+
+#### Acceptance Criteria
+
+1. The verification suite shall include a scenario that renders one peer unreachable by dropping its traffic (network partition / blackhole), distinct from process termination.
+2. When one peer is made unreachable during the scenario, the verification shall assert that heartbeats to the surviving peers continue and that no spurious election of a new leader occurs while the existing leader remains alive and reachable to the majority.
+3. When the partition is healed, the verification shall assert that the previously unreachable peer rejoins and catches up without operator intervention.
+4. While the partition scenario runs, the verification shall assert that no committed job is lost, consistent with the existing failover soak guarantees.
+
+### Requirement 8: Heartbeat Fast-Path Performance
+
+**Objective:** As a cluster operator, I want the steady-state heartbeat and vote path to remain responsive after the threading change, so that making the drive non-blocking does not degrade normal operation at supported cluster sizes.
+
+#### Acceptance Criteria
+
+1. While the cluster is healthy and all peers are connected, the controller shall send heartbeats at the configured interval without added latency attributable to the new dispatch path at supported cluster sizes (3–5 controllers).
+2. The controller shall not introduce additional blocking or unbounded queuing on the steady-state heartbeat and vote send paths, which are already non-blocking enqueues.
--- a/.kiro/specs/raft-drive-async/research.md
+++ b/.kiro/specs/raft-drive-async/research.md
@ -0,0 +1,100 @@
+# Gap Analysis — raft-drive-async (2026-06-25)
+
+Analyzes the gap between the 8 requirements and the existing controller code, to inform the design phase. Evidence from this session's blocking audits + targeted greps.
+
+## Current State Investigation
+
+### The existing deferral idiom and why it's insufficient
+XCE already practices "don't block iomgr": the established pattern is `delay_task_ms(&c->delaymgr, ...)`, which `ordq_put`s a callback and returns immediately. Its own comments confirm the intent — "run off the iomgr walAck thread … delay_task_ms only enqueues" (`config_repl.c:2369`, `ha_convergence.c:207`, `config_repl.c:428`). **But the deferred callback then runs on the delaymgr thread — the heartbeat/election-timeout driver.** So the only existing "offload" target is itself a liveness thread. This is the conceptual heart of the gap: the codebase can *defer*, but its single deferral lane is the thread the corrected `ha-async-io.md` contract must keep free. **The missing capability is a non-liveness execution lane (a "slow lane").**
+
+### Inline-on-iomgr root cause
+`peer_ctrl_msg_handler` is assigned directly as `ep->msg_handler` at `ha.c:4265` (on_vote_connected), `ha.c:4307` (on_pre_vote_connected), `ha.c:6349` (standby accept) — so it runs on the iomgr loop. Normal inbound messages instead go `handle_msg` → `vtask_handle_msg` → vtask pool (`cluster.c:711`, registered `cluster.c:772`). The peer-control path is the anomaly that bypasses the pool.
+
+### Reusable building blocks that already exist
+- **vtask submit primitives**: `vtask_alloc` / `vtask_init` (vtaskmgr.c:21) / `vtask_enq` (vtaskmgr.c:342) exist and compose; the only public entry `vtask_handle_msg` (vtaskmgr.c:356) hardcodes `vtype=VTASK_IO` and *requires* `ep->fsm_in.pkt`. A general non-IO submit is a **small composition** of the existing primitives, not new infrastructure.
+- **Per-endpoint serialization (explicit primitive)**: `endpoint_block_tasks()` / `endpoint_unblock_tasks()` (netio.c:896-911) set a per-endpoint `ep->busy` flag; the VTASK_IO path auto-blocks on task start (vtaskmgr.c:249), unblocks on completion (vtaskmgr.c:66), and the handler path uses it (xcectx.c:440/449), while `vtask_eligible` (vtaskmgr.c:203) skips a busy endpoint's other tasks. This **directly and explicitly satisfies R4.1** (per-connection processing order) when peer control reuses the IO path — a reuse, not new logic. Cross-peer isolation is preserved (block is per-`ep`, so peer X busy ≠ peer Y blocked). **Design trap:** the block is held for the whole task duration — so a long op (snapshot send) run *inline* in a blocked vtask would queue that peer's own subsequent heartbeat-ACKs behind it. This makes the two-tier rule (fast work in vtask, **trigger** long I/O to the slow lane) mandatory for R4.1 + R1 to both hold.
+- **Admission cap pattern**: `COMMIT_WAIT_RESERVED=2` of `nvcpus` (default 8) caps blocking commit-waits at `max(1, n-2)` (commit.c:200, commit.h:17) — a reusable *pattern* for R3, but it is commit-engine-specific today.
+- **Dedicated-background-thread precedent**: `fsync_worker` (persist.c:624) is a long-lived background thread; `push_config_transport_only` **already spawns per-standby `push_thread`s** (config_repl.c:1412) — then blocks by `pthread_join`/`wait_all_with_timeout` on the drive thread (config_repl.c:1332). So the *threads* exist; the defect is the **blocking join on the caller**, not absent concurrency.
+- **Bounded-connect pattern**: `xce_probe_config_tag` already does `O_NONBLOCK`-before-connect + `poll(500ms)` (config_repl.c:1753-1761). The buggy path (`socket_connect`, netio.c:289/302) sets non-blocking *after* connect (netio.c:716) — so R2.3 is largely a **reorder + reuse**, not new code.
+
+### Confirmed absent
+- ~~No iomgr cross-thread wake primitive~~ — **CORRECTED 2026-06-25 (was overstated).** epoll registration is ALREADY thread-safe: `iomgr_monitor` (EPOLL_CTL_ADD), `iomgr_unmonitor` (DEL), `iomgr_update_poll` (MOD) all wrap `epoll_ctl` in `mylock_lock(&iomgr->lock)` (netio.c:1046-1082), and `epoll_ctl` is kernel-thread-safe regardless. The connect→register flow already runs from delaymgr/main threads today (e.g. `on_vote_connected`). So a worker can do the blocking connect off-thread, fully prepare the endpoint (set `ep->msg_handler` etc.) BEFORE it is monitored (an unmonitored fd yields no events → no race), then call the lock-guarded `iomgr_monitor`. **No new eventfd/iomgr-post primitive is needed; the marshal-back is "reuse `iomgr_monitor`."** Residual (design-verify, not a primitive): ensure worker-side handling does not concurrently touch an endpoint's read-FSM buffer that iomgr owns — already mitigated by `endpoint_block_tasks` per-endpoint serialization + lock-guarded poll changes.
+- **No network-partition test primitive**: the soak (`run-ha-soak.sh`) is process-kill only (`kill -9`, lines 74/440); unit "unreachable peer" cases use closed ports, not traffic drops. R7 has **no existing harness**.
+
+## Requirement → Asset Map
+
+| Req | Existing asset | Gap tag |
+|-----|----------------|---------|
+| R1 liveness during snapshot/config I/O | snapshot send `ha.c:2702`+`config_repl.c:2446`; config-push threads `config_repl.c:1412` | **Missing** off-load lane for snapshot; **Constraint** config-push blocks on join |
+| R2 unreachable-peer / bounded connect | `endpoint_connect`/`socket_connect` netio.c:289/302 (blocking); `xce_probe_config_tag` pattern (reuse); reconnect via delaymgr `primary_heartbeat_fn` | **Constraint** (reorder connect) + **Missing** (offload reconnect off delaymgr) |
+| R3 admission protection | `COMMIT_WAIT_RESERVED` pattern (commit-specific) | **Missing** (generalize to the slow lane) |
+| R4 ordering/correctness | `endpoint_block_tasks`/`endpoint_unblock_tasks` per-endpoint sequential primitive (netio.c:896-911, auto on VTASK_IO path); lock-guarded `iomgr_monitor`/`iomgr_unmonitor` (netio.c:1046-1082) for thread-safe epoll registration | **Reuse** (both primitives exist; no new marshal-back primitive needed) |
+| R5 contract compliance | `ha-async-io.md` contract (steering authority) | verified by R1–R4 (no standalone asset) |
+| R6 no regression | existing suites: `test_ha_election/prevote/liveness/failover/rdhup_integration`, `test_repl_majority/apply/catchup`, `test_config_repl`/`test_ha_repl_push`, `test_persist_recovery`, `test_commit_barrier_rollout`/`test_commit_durability` | **Reuse** as regression gate |
+| R7 partition verification | `run-ha-soak.sh` (kill-only) | **Missing** (no traffic-drop primitive) |
+| R8 heartbeat perf | heartbeat fan-out already non-blocking enqueue | **Constraint** (don't regress fast path) |
+
+## Implementation Approach Options
+
+### Option A — Extend in place (minimal)
+Dispatch `peer_ctrl_msg_handler` through the existing `handle_msg`/`vtask_handle_msg` path; add a general non-IO `vtask_enq` wrapper; reorder `socket_connect` to non-blocking + `poll`; replace config-push `wait_all_with_timeout` join with a non-blocking completion check; reuse `delay_task_ms` only for *non-blocking triggers*. Touches ha.c, config_repl.c, netio.c, vtaskmgr.c.
+- ✅ Least new surface; reuses every existing pattern; fastest path to closing the delaymgr+iomgr inline blocks.
+- ❌ Long I/O (snapshot, config-push) still lands in the **request** vtask pool → relocates the stall to pool exhaustion (R3 only partially met).
+
+### Option B — New dedicated slow-lane executor
+Build a dedicated bounded **slow-lane executor** (generalize `fsync_worker`) for all long blocking I/O. Peer control still dispatched to vtask, but heavy work re-offloaded to the slow lane. (No iomgr-post primitive needed — epoll registration reuses the lock-guarded `iomgr_monitor`.)
+- ✅ Clean separation; R3/R4 fully met by construction; isolates seconds-long I/O from request handling and from delaymgr.
+- ❌ One genuinely new component; more interface design than extending in place.
+
+### Option C — Hybrid (recommended)
+- **Extend** (reuse): peer_ctrl → vtask via the existing dispatch pattern; general non-IO vtask submit (compose `vtask_alloc/init/enq`); bounded connect (reorder + reuse `xce_probe_config_tag` pattern); preserve ordering via `VTASK_IO` serialization.
+- **New** (the one real gap): a small bounded **slow-lane executor** for snapshot-transfer + config-push (generalize the `fsync_worker`/`push_thread` precedent; per-peer dedup + admission cap parallel to `COMMIT_WAIT_RESERVED`). Epoll registration of a connected fd reuses the lock-guarded `iomgr_monitor` (no new primitive).
+- **Phased**:
+  - *Phase 1 (S–M, highest value/lowest risk):* bounded connect + offload reconnect off delaymgr → closes the heartbeat *sender*-side starvation (R2). Independent, shippable.
+  - *Phase 2 (M):* dispatch peer_ctrl to vtask (epoll registration reuses lock-guarded `iomgr_monitor`) → closes the iomgr *receiver*-side inline processing (R4, R5).
+  - *Phase 3 (L):* slow-lane executor + move snapshot-send/config-push to it + admission generalization (R1, R3).
+  - *Phase 4 (M):* partition test harness + regression sweep (R6, R7, R8).
+- ✅ Each phase is independently testable and de-risks the next; matches what already exists vs. what's truly missing.
+- ❌ Requires phase coordination; must keep `ha-async-io.md` compliance consistent across phases.
+
+## Effort & Risk
+
+| Work item | Effort | Risk | Justification |
+|-----------|--------|------|---------------|
+| Bounded connect + reconnect offload (Ph1) | S–M | Medium | reorder existing pattern; offload target needs to be non-delaymgr |
+| General non-IO vtask submit | S | Low | compose existing `vtask_alloc/init/enq` |
+| peer_ctrl → vtask dispatch | M | Medium | correctness-sensitive: ordering (mitigated by `endpoint_block_tasks`), term/vote semantics; epoll register reuses lock-guarded `iomgr_monitor` |
+| Slow-lane executor + snapshot-send offload | L | Medium-High | per-peer dedup + coordination with ongoing append/WAL-trim/catch-up stream |
+| Admission generalization (R3) | M | Medium | extend `COMMIT_WAIT_RESERVED` pattern to slow lane |
+| Partition test harness (R7) | M | Medium | no traffic-drop primitive today; soak infra is kill-only |
+| **Overall** | **L (1–2 wks)** | **Medium** | riskiest: snapshot-offload coordination + moving peer_ctrl off iomgr (no new epoll-core primitive needed) |
+
+## Recommendations for Design Phase
+
+- **Preferred approach:** Option C (Hybrid), phased. Ship Phase 1 (bounded connect) early — it independently closes the sender-side heartbeat starvation and is low-risk.
+- **Key decisions to settle in design:**
+  1. **Connected-fd registration:** reuse the lock-guarded `iomgr_monitor` (netio.c:1058) — a worker prepares the endpoint fully, then registers it; no new primitive. Design-verify only that worker handling does not touch an endpoint's read-FSM buffer concurrently with iomgr (serialized by `endpoint_block_tasks`).
+  2. **Slow-lane shape:** a shared bounded pool vs. reuse the existing per-standby `push_thread` model for config-push and a small pool for snapshot. Size to peer count.
+  3. **Fold-in question — RESOLVED 2026-06-25: keep `raft-drive-async` STANDALONE.** Scope-check verdict ADJACENT-SAME-AREA: `election-iomgr-refactor` is fully implemented and fixed the *startup* blocking path (pthread election before `cluster_run`); it touched none of raft-drive-async's runtime targets (`peer_ctrl_msg_handler` dispatch, delaymgr `connect`, snapshot/config-push on iomgr) and explicitly deferred async DNS as a non-goal. It is consumed as completed *upstream*, providing reusable primitives (`delay_task_ms`, `on_connected` fire-once callback, the `ha_start_election_async`/non-blocking vote-fan-out contract). Folding in would reopen a done spec for net-new infra it never held.
+
+## Research Needed (carry to design)
+
+- ~~election-iomgr-refactor scope overlap~~ — **RESOLVED 2026-06-25** (scope-check): ADJACENT-SAME-AREA, keep standalone; reuse its primitives (`delay_task_ms`, `on_connected`, `ha_start_election_async`). See Recommendations decision #3.
+- **Endpoint read-FSM ownership**: confirm worker-side peer-control handling never touches an endpoint's read-FSM buffer concurrently with the iomgr read path (expected safe via `endpoint_block_tasks` + handoff of `fsm_in.pkt`). (Epoll registration itself is already settled — lock-guarded `iomgr_monitor`; no new primitive.)
+- **Partition primitive choice** for R7: `iptables`/`tc` (root, infra-level) vs. an in-process socket-drop hook in `xce_test`. Determines test portability.
+- **Snapshot-send coordination** with the concurrent WAL-catch-up stream and the primary's append/`trim_wal_segments` while the transfer runs on the slow lane.
+- **Admission interaction** between the existing `COMMIT_WAIT_RESERVED` (commit waits) and the new slow-lane cap — shared budget vs. independent, so the two together can't starve the 8-worker pool.
+
+---
+
+## Design Synthesis (2026-06-25, /kiro-spec-design)
+
+Key simplification chosen at design time: **the offload target for long blocking I/O is a dedicated slow-lane executor (separate threads), NOT the vtask request pool.** Consequences:
+
+1. **R3 (pool protection) becomes structural** — long ops never run on the vtask pool, so they cannot starve liveness-critical message handling. The existing `COMMIT_WAIT_RESERVED` continues to cover commit-waits.
+2. **The "general non-IO vtask submit" research item is SUPERSEDED** — not needed. Peer-control handling is packet-triggered, so it reuses the existing VTASK_IO `vtask_handle_msg` path; the only packet-less trigger (delaymgr reconnect) goes to the slow lane, not the vtask pool. No new vtask submit primitive required.
+3. Combined with the already-refuted iomgr-post primitive (epoll registration reuses lock-guarded `iomgr_monitor`), **the only genuinely-new component is the slow-lane executor** (`slowlane.{c,h}`). Everything else is reuse + integration.
+
+Build-vs-adopt: slow lane generalizes the existing `fsync_worker`/`push_thread` background-thread precedent (adopt pattern, build the small bounded executor). Bounded connect reuses the `xce_probe_config_tag` pattern (reorder, not new). Per-endpoint ordering reuses `endpoint_block_tasks`.
+
+Phased migration (P1 bounded-connect → P2 peer_ctrl→vtask → P3 slow-lane+offload → P4 partition test) keeps each step behavior-preserving and independently revertible.
--- a/.kiro/specs/raft-drive-async/spec.json
+++ b/.kiro/specs/raft-drive-async/spec.json
@ -0,0 +1,22 @@
+{
+  "feature_name": "raft-drive-async",
+  "created_at": "2026-06-25T11:37:13Z",
+  "updated_at": "2026-06-26T04:43:19Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true
+}
--- a/.kiro/specs/raft-drive-async/tasks.md
+++ b/.kiro/specs/raft-drive-async/tasks.md
--- a/.kiro/specs/raft-log-replication/spec.json
+++ b/.kiro/specs/raft-log-replication/spec.json
@ -1,9 +1,9 @@
 {
  "feature_name": "raft-log-replication",
  "created_at": "2026-06-21T11:00:00Z",
-  "updated_at": "2026-06-21T11:00:00Z",
+  "updated_at": "2026-06-25T00:00:00Z",
  "language": "en",
-  "phase": "tasks-generated",
+  "phase": "implemented",
  "approvals": {
    "requirements": {
      "generated": true,
@ -19,5 +19,6 @@
    }
  },
  "ready_for_implementation": true,
+  "implementation_status": "implemented — library log half landed + validated (24/24 tasks, raft_sim GO per memory project_raft_log_replication_done). Phase reconciled 2026-06-25 (was mislabeled tasks-generated).",
  "note": "Auto-approved (-y mode). Extends election-only src/lib/raft into a strict raft.tla log-half implementation: ClientRequest/AppendEntries/HandleAppendEntriesRequest/HandleAppendEntriesResponse/UpdateTerm-on-any-RPC. Closes bug-851 G1/G2/G3/G5. Authoritative shared callback contract (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from) consumed verbatim by raft-commit-safety, raft-sim-conformance, and ha-term-safety. Boundary: owns src/lib/raft log additions + (term,index) reconcile seam only; NOT commit-index advance, vote freshness, config, or controller WAL/transport internals."
 }
--- a/.kiro/steering/ha-async-io.md
+++ b/.kiro/steering/ha-async-io.md
@ -10,15 +10,23 @@ The HA layer combines Raft-style consensus election with a single-writer replica

 ## Async I/O Model

-Three components in `src/lib/`:
+Three components in `src/lib/` plus a dedicated slow-lane executor:

-| Component | Thread model | Purpose | Blocking? |
-|-----------|-------------|---------|-----------|
-| **iomgr** | Single-threaded epoll loop | Inbound packet dispatch, endpoint lifecycle | **Never block** |
-| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | Executes vtask handlers for dispatched packets | **Return quickly** |
-| **delaymgr** | Single timer thread | Periodic/delayed callbacks, connect(), file I/O | **Permitted** |
+| Component | Thread model | May do | Must NOT do |
+|-----------|-------------|--------|-------------|
+| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
+| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
+| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
+| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |

-**Critical rule**: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.
+**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).**
+No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
+1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
+2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
+3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, commit.c:195-203) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
+4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.
+
+> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`) — tracked as the `raft-drive-async` / P2 work in roadmap.md.

 Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.

--- a/.kiro/steering/roadmap.md
+++ b/.kiro/steering/roadmap.md
@ -20,6 +20,33 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin

 > **ha-term-safety** (requirements + design generated 2026-06-21; `.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. Next: `/kiro-spec-tasks ha-term-safety`.

+## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`)
+
+**Goal:** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. Today `persist_job_hook_fun` (an `AFTER_EVENT` hook) performs the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutates in-memory state BEFORE that commit (`job.c:558` before `job.c:593`; `persist.c:2532`/`persist.c:2198`/`ha.c:3080`). A failed/timed-out commit does not cleanly revert the optimistic mutation — the structural root of the bug-795/828/851/1060 divergence family. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decompose via Path E (2 new specs + touchpoints).
+
+### Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`)
+
+The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_handler` runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (`ha-async-io.md`, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § Keystone + "Two distinct prerequisites" + P2 offload mechanism):
+
+- **P1 — commit-barrier async** (the *commit* drive). Convert `repl_wait_majority_ack()` from blocking `pthread_cond_timedwait` (commit.c:359) to event-driven `resolve(lsn,outcome)` on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. **Owned within `raft-apply-decoupling` as its keystone** (sequenced before its Phase A).
+- **P2 — drive hardening** (the *transport/timer* drive). Now its own spec **`raft-drive-async`** (brief: `.kiro/specs/raft-drive-async/brief.md`) — promoted because it threatens liveness independently. See the Specs list below.
+
+### Specs (dependency order — Raft Drive & Apply Async family)
+
+- [ ] **raft-drive-async** *(START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.**
+- [ ] raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single `apply_committed_entry()` seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). **Contains P1 (commit-barrier async) as its keystone.** Commands (WAL_MODE_WAIT) apply-after-commit; **facts (WAL_MODE_RETAIN) routed through commit-then-apply too** — D1 RESOLVED (block-but-durable under quorum loss); formalized as `ha-consistency-invariants` criterion 2.9. Dependencies: **raft-drive-async (HARD — reuses its slow-lane + admission infra; drive must be async first)**, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync (HARD for the facts-recovery half — see below); coordinate with ha-term-safety. Briefs: `.kiro/specs/raft-apply-decoupling/brief.md` + `facts-in-raft.md`.
+- [ ] worker-rejoin-resync — **elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Already on the backlog below; pulled forward into this family. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards).
+- [ ] job-array-single-wal-record — **PROMOTED from backlog 2026-06-24.** Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: `.kiro/specs/job-array-single-wal-record/brief.md`.
+
+### Existing Spec Updates (Raft Apply-Ordering phase)
+
+- [ ] raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with `/kiro-spec-requirements raft-sim-conformance` once raft-apply-decoupling requirements land.
+- [ ] ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.
+
+### Direct Implementation Candidates (Raft Apply-Ordering phase)
+
+- [ ] xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.
+
 ## Specs (dependency order)

 > `[x]` = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).
@ -37,16 +64,17 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
 - [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
 - [ ] admin-config-tool — drives `OP_upgradeConfig` (prerequisite for config-upgrade Phase 2). Encode ClusterConfig YAML/JSON → Cap'n Proto binary, SHA-256 over the bytes, connect to primary (SSL, same CA), send `OP_upgradeConfig`, display reply (version tag, acked count, errors), poll `OP_probe` after restart. Suggested CLI: `xce-config push --controller <host:port> --config cluster.yaml`. MUST validate that user configs don't use the reserved name `"lost_and_found"` for any queue/host (defence-in-depth: controller enforces at runtime via `load_primary_config()` FATAL, tool prevents at authoring time; migration: pre-existing clusters must rename before upgrade — see config-replication-design.md §6.5/§7.1). _(From FUTURE_WORK.md §1 + §8, 2026-05-12.)_
 - [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
- [ ] job-array-single-wal-record — **recorded 2026-06-16.** Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
+- [→] job-array-single-wal-record — **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
 - [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
 - [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
 - [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream (recorded 2026-06-21): `config-upgrade-snapshot` HARD-depends on this** — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857).
+- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream (recorded 2026-06-21): `config-upgrade-snapshot` HARD-depends on this** — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857). **This is also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) — recorded 2026-06-25:** WAL streaming alone cannot serve a standby whose needed range was trimmed (bug-795/828/1147), and a new node always starts empty, so safe aggressive WAL trimming + new-node join *requires* InstallSnapshot (snapshot-at-boundary + tail WAL). Until this lands, trim must stay cursor-bounded (strategy-A stopgap, see Hardening) and the system cannot safely admit a standby that has fallen below the trim floor. Scope to add when promoted: the standby-side snapshot-install / apply path (request snapshot when its cursor is below the leader's earliest retained LSN, install, resume from snapshot tail).

 ## Hardening / Direct-Implementation Items

 Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.

+- [ ] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — `trim_wal_segments` (`persist.c:1374`) bounds deletion ONLY by the open segment (`if (seg_start_lsn < active_seg_lsn) unlink()`) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** in-flight snapshot boundary. With no working snapshot backfill (`write_snapshot` emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is **strategy-A retention discipline that should exist regardless**. Stopgap fix: compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. **Folded into `raft-drive-async` task 3.2** (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs `entity-bearing snapshots` (strategy B, backlog). Related: bug-795, bug-828, bug-1060.
 - [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
 - [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_
 - [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_