Rongsong Shen
59d9df246c
Per design invariant (a controller exits only on crash or admin kill), replace the self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual admin recovery). Records the never-self-terminate invariant as project-wide guidance in src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
84 lines
6.3 KiB
Markdown
84 lines
6.3 KiB
Markdown
# HA Protocol & Async I/O
|
|
|
|
## Philosophy
|
|
|
|
The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.
|
|
|
|
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
|
|
|
|
**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.
|
|
|
|
## Async I/O Model
|
|
|
|
Three components in `src/lib/`:
|
|
|
|
| Component | Thread model | Purpose | Blocking? |
|
|
|-----------|-------------|---------|-----------|
|
|
| **iomgr** | Single-threaded epoll loop | Inbound packet dispatch, endpoint lifecycle | **Never block** |
|
|
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | Executes vtask handlers for dispatched packets | **Return quickly** |
|
|
| **delaymgr** | Single timer thread | Periodic/delayed callbacks, connect(), file I/O | **Permitted** |
|
|
|
|
**Critical rule**: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.
|
|
|
|
Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.
|
|
|
|
## Connection Ownership
|
|
|
|
**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).
|
|
|
|
**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.
|
|
|
|
**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.
|
|
|
|
## Raft Election
|
|
|
|
`src/controller/raft.c` implements standard Raft-style election:
|
|
|
|
- **RequestVote + RequestVoteReply** over Cap'n Proto
|
|
- **Freshness gate**: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
|
|
- **Votes persisted atomically** with term increment (raft_cb_persist)
|
|
- **Majority wins**: quorum = ceil(n_peers / 2)
|
|
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning
|
|
- **WAL truncation** on promotion: `persist_truncate_wal_above(safe_committed_lsn)` removes diverging entries above the safe commit point
|
|
|
|
**Key safety invariants**:
|
|
- `currentTerm` is monotonic (never decreases)
|
|
- `votedFor` updated atomically with term increment before granting any vote
|
|
- `commitIndex` never decreases (monotonic)
|
|
|
|
## Heartbeat Model
|
|
|
|
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
|
|
|
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
|
|
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
|
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
|
|
|
## Replication Consistency (Two-Tier)
|
|
|
|
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).
|
|
|
|
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
|
|
|
|
## Dispatch Gate
|
|
|
|
`ha_dispatch_gate` controls which opcodes are accepted per role:
|
|
|
|
| Role | Allowed |
|
|
|------|---------|
|
|
| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
|
|
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
|
|
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
|
|
|
|
This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.
|
|
|
|
## Raft-to-Code Mapping
|
|
|
|
| TLA+ Variable | XCE Code |
|
|
|--------------|----------|
|
|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
|
|
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
|
|
| `votedFor[i]` | `ha->voted_for_name[]` |
|
|
| `log[i]` | WAL segments in `persist->wal_path` |
|
|
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
|
|
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |
|