Rongsong Shen
40f510fb14
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
103 lines
15 KiB
Markdown
103 lines
15 KiB
Markdown
# HA Protocol & Async I/O
|
||
|
||
## Philosophy
|
||
|
||
The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.
|
||
|
||
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
|
||
|
||
**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.
|
||
|
||
## Async I/O Model
|
||
|
||
Three components in `src/lib/` plus a dedicated slow-lane executor:
|
||
|
||
| Component | Thread model | May do | Must NOT do |
|
||
|-----------|-------------|--------|-------------|
|
||
| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
|
||
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
|
||
| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
|
||
| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |
|
||
|
||
**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).**
|
||
No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
|
||
1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
|
||
2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
|
||
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, commit.c:195-203) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
|
||
4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.
|
||
|
||
> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`) — tracked as the `raft-drive-async` / P2 work in roadmap.md.
|
||
|
||
Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.
|
||
|
||
## Connection Ownership
|
||
|
||
**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).
|
||
|
||
**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.
|
||
|
||
**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.
|
||
|
||
## Raft Election
|
||
|
||
The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.
|
||
|
||
**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix.
|
||
|
||
- **RequestVote + RequestVoteReply** over Cap'n Proto
|
||
- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
|
||
- **Votes persisted atomically** with term increment (raft_cb_persist)
|
||
- **Majority wins**: quorum = ceil(n_peers / 2)
|
||
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
|
||
- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
|
||
|
||
**Key safety invariants**:
|
||
- `currentTerm` is monotonic (never decreases)
|
||
- `votedFor` updated atomically with term increment before granting any vote
|
||
- `commitIndex` never decreases (monotonic)
|
||
|
||
## Heartbeat Model
|
||
|
||
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||
|
||
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
|
||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||
- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
|
||
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||
|
||
## Replication Consistency (Two-Tier)
|
||
|
||
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).
|
||
|
||
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
|
||
|
||
## Per-Entry Term & `(term,lsn)` Reconcile (ha-term-safety)
|
||
|
||
Every WAL entry carries the **leader's real Raft term** in `wal_entry_hdr_t.term` (a value fix — the 40-byte header layout is unchanged). The primary stamps `s->term` via `raft_log_append` → `on_persist_entry`; the **live replication carries that term** (the legacy term-0 `repl_broadcast` was retired — a standby persists the *leader's* term, not 0 or a local substitute). Standbys intake every heartbeat/replication through `raft_on_append_entries_request`, so `UpdateTerm` (adopt a higher term + step down) and `(term,lsn)` conflict-truncate happen on every RPC. **Reconcile by `(term, lsn)`, never by LSN alone** — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back `hdr.term` per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the **same unified log** as a `WAL_CONFIG_MARKER` (a `{version_tag, sha256}` reference, no config bytes) that commits + reconciles by `(term,lsn)` exactly like a job entry; the config *bytes* travel out-of-band and are **hash-verified against the committed marker before activation** (mismatch ⇒ loud serve-fence, stay alive + retry).
|
||
|
||
## Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling)
|
||
|
||
The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commit.c`, see `persistence.md`) is **authoritative** for advancing `committed_lsn`. The raft library's `commit_index` (fed via `on_commit_advanced` after `raft_advance_commit`) is kept **advisory only** — it is NOT folded in as a hard ceiling (`committed = min(durable-majority, raft_commit_index)` was tried and removed). Rationale: under the single-writer model the durable-majority watermark is *already* a current-term `(term,lsn)`-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a `min()` ceiling **starves commit below the safe durable majority** (jobs acked-durable on a majority but never committed). Keep driving the library (for its own `match_index`/`next_index`/send state) but never let its commit index *gate* `committed_lsn`. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling.
|
||
|
||
## Dispatch Gate
|
||
|
||
`ha_dispatch_gate` controls which opcodes are accepted per role:
|
||
|
||
| Role | Allowed |
|
||
|------|---------|
|
||
| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
|
||
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
|
||
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
|
||
|
||
This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.
|
||
|
||
## Raft-to-Code Mapping
|
||
|
||
| TLA+ Variable | XCE Code |
|
||
|--------------|----------|
|
||
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
|
||
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
|
||
| `votedFor[i]` | `ha->voted_for_name[]` |
|
||
| `log[i]` | WAL segments in `persist->wal_path` |
|
||
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
|
||
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |
|