Rongsong Shen
5f307e297b
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol step lifted out of the notification hook fan-out (root of the bug-795/828/ 851/1060 divergence family). Decision: apply-after-commit via a single shared apply_committed_entry() seam reused by online apply and restart replay. - raft-drive-async: full spec docs (brief/requirements/research/design) committed alongside its already-tracked tasks.md; async-drive prerequisite for the apply-decoupling work, sequenced to start first (threatens failover liveness today). - raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open decision D1). - job-array-single-wal-record: promoted from backlog to a tracked spec, sequenced after raft-apply-decoupling. - ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant), documenting the current on-receipt fact-reaction deviation this family will reconcile. - raft-commit-safety / raft-log-replication: reconcile spec.json phase to "implemented" (both were mislabeled tasks-generated despite full 18/18 and 24/24 task completion + raft_sim GO). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
92 lines
9.7 KiB
Markdown
92 lines
9.7 KiB
Markdown
# Brief: raft-drive-async
|
||
|
||
## Problem
|
||
|
||
The Raft **protocol drive** is not fully event-driven: blocking operations run inline on the two threads that carry cluster liveness signals (iomgr and delaymgr). When either stalls, heartbeats/votes/ACKs stop flowing → peers declare the leader dead → spurious election. This is a likely root of the **bug-851 chaotic-failover family**.
|
||
|
||
A blocking audit (2026-06-25) found three CRITICAL violations of the (newly corrected) thread-role contract in `ha-async-io.md`:
|
||
|
||
1. **Snapshot send + `persist_checkpoint` inline on iomgr** — `ha_dispatch_repl_action_for_standby` (ha.c:2702) runs `persist_checkpoint` (fdatasync + WAL-segment trim) + `ha_send_snapshot_to_standby` (`fopen`/`fread` chunk loop, config_repl.c:2446-2503) on the iomgr event loop. A large snapshot freezes ALL heartbeat/vote/client processing for its whole duration.
|
||
2. **Config-push wait inline on iomgr** — `push_config_transport_only` → `wait_all_with_timeout` busy-polls `usleep(5000)` + `pthread_join` (config_repl.c:1332,1340), called from `peer_ctrl_msg_handler` paths (ha.c:1172/1177/2672/2677/3973), up to `CONF_PUSH_TIMEOUT_MS`.
|
||
3. **Blocking `gethostbyname` + `connect` on the delaymgr heartbeat thread** — `endpoint_connect` (netio.c:289/302) via `ha_connect_to_standby` from `primary_heartbeat_fn`. One slow/unreachable peer stalls ALL heartbeats + election-timeouts for seconds. (`set_fd_nonblocking` is applied AFTER connect, netio.c:716 — the connect is inherently blocking.)
|
||
|
||
**Root cause:** `peer_ctrl_msg_handler` is registered **directly as `ep->msg_handler`** on peer/vote endpoints — it runs inline on iomgr instead of dispatching to the vtask pool like normal cluster messages (`handle_msg` → `vtask_handle_msg`). So any work it does, including the heavy I/O above, executes on the event loop.
|
||
|
||
Plus MEDIUM vtask-pool blockers (pool-exhaustion, not drive-thread): config-ack wait (config_repl.c:1459), a 30-second `usleep` checkpoint poll (config_repl.c:1661), `ha_persist_term` fsync on a vote grant (ha.c:3407 — adds election latency).
|
||
|
||
## Current State
|
||
|
||
- The corrected **thread-role contract** is now steering (`ha-async-io.md`, 2026-06-25): iomgr = non-blocking I/O + dispatch only; delaymgr = trigger-only, never blocks; vtask pool = bounded blocking only under admission reservation; long blocking I/O → dedicated slow lane. This spec brings the **code into compliance** with that contract. (The contract SUPERSEDED the old "delaymgr for blocking ops = Permitted" rule, which was the design flaw behind these findings — delaymgr is both the heartbeat driver and was the documented blocking sink.)
|
||
- Existing primitives to reuse: `COMMIT_WAIT_RESERVED` admission cap (commit.c:195-203) already keeps vtask-pool headroom against blocking waits; the non-blocking-connect + bounded-`poll` pattern already exists in `xce_probe_config_tag` (config_repl.c:1753-1761); `vtask_handle_msg` (vtaskmgr.c:356) is the dispatch path but is endpoint+inbound-packet-bound (`vtype=VTASK_IO`).
|
||
- Confirmed already-correct: RequestVote fan-out, heartbeat fan-out, WAL replication streaming all use non-blocking `xce_request()` enqueue; standby-side replicated WAL write + fdatasync run on a vtask worker, not iomgr.
|
||
|
||
## Desired Outcome
|
||
|
||
The protocol drive is non-blocking on both liveness threads, by construction:
|
||
|
||
- **iomgr** does only non-blocking socket I/O + dispatch — `peer_ctrl_msg_handler`'s real work is dispatched to the vtask pool (conforming to the existing `handle_msg` pattern), so no future handler addition can reintroduce inline blocking.
|
||
- Inside the vtask, **fast protocol work runs inline** (decode ack → update match_index → advance commit → send next heartbeat — all bounded); **long I/O is triggered to a slow lane** (snapshot transfer, config-push), never run in the request-pool worker.
|
||
- **delaymgr** only fires timers and triggers work; the peer connect is **bounded** (non-blocking + `poll` timeout) and **offloaded** off the timer thread.
|
||
- Heartbeat liveness is provably protected: a partitioned (unreachable, not merely dead) peer cannot stall heartbeats to healthy peers or trigger a spurious election.
|
||
|
||
## Approach
|
||
|
||
Bring the code into compliance with the steering thread-role contract, structurally rather than per-op:
|
||
|
||
1. **Dispatch `peer_ctrl_msg_handler` to the vtask pool** (the structural fix). iomgr becomes epoll + enqueue only. Preserve per-connection message ordering (VTASK_IO per-endpoint serialization is a feature here). Marshal any iomgr-owned state mutations (epoll/endpoint lifecycle) back to the iomgr thread (bug-646 discipline).
|
||
2. **Two-tier execution inside the vtask:** fast bounded protocol work inline; **trigger** long I/O (snapshot send + `persist_checkpoint`, config-push) to a **dedicated slow-lane / admission-capped executor** so multi-second tasks cannot starve the request pool. Per-peer dedup (one snapshot in flight per standby); coordinate with ongoing WAL-catchup stream and the primary's appends/WAL-trim.
|
||
3. **delaymgr connect → bounded + offloaded:** reuse the `xce_probe_config_tag` non-blocking-connect + bounded-`poll` pattern; trigger the reconnect as an idempotent, refcounted, deduped task off the delaymgr thread; marshal the connected-fd epoll registration back to iomgr. DNS (`gethostbyname`) is LOW real-world risk (HPC peers are IP/`/etc/hosts`) — defer async DNS unless measured.
|
||
4. **General (non-IO) task-submit path:** add the submit primitive needed for #2/#3 (current `vtask_handle_msg` requires an inbound packet); or generalize it.
|
||
5. **Admission discipline:** any offloaded blocking task lives under a reservation (extend the `COMMIT_WAIT_RESERVED` pattern) so liveness-critical messages always have a free worker.
|
||
|
||
## Scope
|
||
|
||
- **In**:
|
||
- Dispatch `peer_ctrl_msg_handler` real work to the vtask pool (root-cause structural fix).
|
||
- Snapshot send + `persist_checkpoint` off iomgr → slow lane, with per-peer dedup + WAL coordination.
|
||
- Config-push wait off iomgr → slow lane / async ACK.
|
||
- delaymgr peer connect → bounded non-blocking + offloaded; reconnect dedup/refcount/marshal-back.
|
||
- General non-IO vtask submit path + admission reservation for offloaded blocking tasks.
|
||
- Partition (unreachable-peer) regression test; verify no spurious election + heartbeats to survivors continue.
|
||
- **Out**:
|
||
- The commit-barrier async conversion (P1 / `repl_wait_majority_ack` → `resolve`) — owned by `raft-apply-decoupling` as its keystone (different concern: that's the *commit* drive, this is the *transport/timer* drive).
|
||
- Apply-after-commit / hook decoupling / facts-through-raft (raft-apply-decoupling).
|
||
- Async DNS (deferred unless measured).
|
||
- The MEDIUM vtask-pool blockers beyond what the admission discipline naturally covers (note them; fix opportunistically).
|
||
- Entity-bearing snapshots / config-upgrade snapshot (separate specs).
|
||
|
||
## Boundary Candidates
|
||
|
||
- The `peer_ctrl_msg_handler` → vtask dispatch seam.
|
||
- The dedicated slow-lane executor + admission reservation.
|
||
- The general non-IO vtask-submit primitive.
|
||
- The bounded-connect + offloaded-reconnect path.
|
||
- The iomgr-owned-state marshal-back mechanism (epoll/endpoint lifecycle).
|
||
|
||
## Out of Boundary
|
||
|
||
- Commit-engine internals (P1 owns the commit-drive async).
|
||
- Snapshot *content* format (entity-bearing snapshots is a separate spec) — this spec only moves the *transfer* off the drive thread.
|
||
|
||
## Upstream / Downstream
|
||
|
||
- **Upstream**: the `ha-async-io.md` thread-role contract (authority); existing `COMMIT_WAIT_RESERVED` (commit.c) and `xce_probe_config_tag` bounded-connect pattern (reuse); `election-iomgr-refactor` (COMPLETED upstream — provides reusable primitives `delay_task_ms`, `on_connected`, `ha_start_election_async`/non-blocking vote fan-out).
|
||
- **Downstream**: `raft-apply-decoupling` (its async commit drive + apply pump reuse the slow-lane + admission infrastructure this spec establishes; and the drive must be non-blocking before apply is restructured on top of it); safe failover for the whole bug-851 family depends on this landing.
|
||
|
||
## Existing Spec Touchpoints
|
||
|
||
- **Adjacent (RESOLVED standalone 2026-06-25)**: `election-iomgr-refactor` is ADJACENT-SAME-AREA — it fixed the *startup* blocking path and is fully implemented; this spec fixes the *runtime* drive-thread paths it never touched. Keep separate; consume it as upstream. (Scope-check verdict in research.md.)
|
||
- **Coordinate**: `ha-term-safety` (touches the same `peer_ctrl_msg_handler` / repl dispatch region); `ha-wal-replication` (snapshot/catch-up transport); `controller-status` (heartbeat/committed propagation on the same drive).
|
||
- **Verification home**: `raft-sim-conformance` (add the partition/liveness assertion) and `/xce-test-ha` (partition variant).
|
||
|
||
## Constraints
|
||
|
||
- The corrected thread-role contract (`ha-async-io.md`) is authoritative — iomgr and delaymgr never block inline; long I/O on the slow lane; bounded blocking in vtask only under admission reservation.
|
||
- "Controllers never self-terminate" — an unreachable peer is a loud retry, never an abort.
|
||
- Preserve per-connection message ordering when dispatching `peer_ctrl_msg_handler` to vtask.
|
||
- Must not regress the heartbeat fast path measurably at cluster scale (3–5 peers); per-message vtask overhead accepted in exchange for uniformity + liveness safety.
|
||
- Verified by a **partition** (unreachable, not killed) soak — the case `/xce-test-ha`'s current kill-only soak does not reproduce.
|
||
|
||
## Provenance
|
||
|
||
Surfaced 2026-06-25 during the `raft-apply-decoupling` discovery as prerequisite **P2** (drive hardening); promoted to its own spec because it threatens liveness independently of the apply refactor. Full analysis: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § "Two distinct prerequisites" + P2 offload mechanism. Thread-role contract: `.kiro/steering/ha-async-io.md` (corrected 2026-06-25).
|