lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: promote raft-apply-decoupling spec family from /kiro-discovery

Browse source 
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.

- raft-drive-async: full spec docs (brief/requirements/research/design)
  committed alongside its already-tracked tasks.md; async-drive prerequisite
  for the apply-decoupling work, sequenced to start first (threatens
  failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
  apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
  decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
  sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
  documenting the current on-receipt fact-reaction deviation this family
  will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
  "implemented" (both were mislabeled tasks-generated despite full 18/18
  and 24/24 task completion + raft_sim GO).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
Rongsong Shen 2026-07-03 02:41:11 +00:00
parent 9318f9977e
commit 5f307e297b
14 changed files with 1075 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

22
.kiro/steering/ha-async-io.md
View file

@ -10,15 +10,23 @@ The HA layer combines Raft-style consensus election with a single-writer replica
## Async I/O Model
Three components in `src/lib/`:
Three components in `src/lib/` plus a dedicated slow-lane executor:
| Component | Thread model | Purpose | Blocking? |
|-----------|-------------|---------|-----------|
| **iomgr** | Single-threaded epoll loop | Inbound packet dispatch, endpoint lifecycle | **Never block** |
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | Executes vtask handlers for dispatched packets | **Return quickly** |
| **delaymgr** | Single timer thread | Periodic/delayed callbacks, connect(), file I/O | **Permitted** |
| Component | Thread model | May do | Must NOT do |
|-----------|-------------|--------|-------------|
| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |
**Critical rule**: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.
**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).**
No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, commit.c:195-203) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.
> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`) — tracked as the `raft-drive-async` / P2 work in roadmap.md.
Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.