diff --git a/.kiro/steering/ha-async-io.md b/.kiro/steering/ha-async-io.md index 1b82b2a..2e745cb 100644 --- a/.kiro/steering/ha-async-io.md +++ b/.kiro/steering/ha-async-io.md @@ -32,14 +32,14 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send ## Raft Election -`src/controller/raft.c` implements standard Raft-style election: +The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path. - **RequestVote + RequestVoteReply** over Cap'n Proto -- **Freshness gate**: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter` +- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below). - **Votes persisted atomically** with term increment (raft_cb_persist) - **Majority wins**: quorum = ceil(n_peers / 2) -- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning -- **WAL truncation** on promotion: `persist_truncate_wal_above(safe_committed_lsn)` removes diverging entries above the safe commit point +- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828). +- **WAL truncation rule (load-bearing — bug-828)**: the **ONLY** truncates are (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, and (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile — both run *before* the new leader commits anything. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`) — it lags the true commit point, so a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello". **Key safety invariants**: - `currentTerm` is monotonic (never decreases) @@ -60,6 +60,14 @@ Bidirectional heartbeat with Raft-style AppendEntries semantics: **Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push. +## Per-Entry Term & `(term,lsn)` Reconcile (ha-term-safety) + +Every WAL entry carries the **leader's real Raft term** in `wal_entry_hdr_t.term` (a value fix — the 40-byte header layout is unchanged). The primary stamps `s->term` via `raft_log_append` → `on_persist_entry`; the **live replication carries that term** (the legacy term-0 `repl_broadcast` was retired — a standby persists the *leader's* term, not 0 or a local substitute). Standbys intake every heartbeat/replication through `raft_on_append_entries_request`, so `UpdateTerm` (adopt a higher term + step down) and `(term,lsn)` conflict-truncate happen on every RPC. **Reconcile by `(term, lsn)`, never by LSN alone** — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back `hdr.term` per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the **same unified log** as a `WAL_CONFIG_MARKER` (a `{version_tag, sha256}` reference, no config bytes) that commits + reconciles by `(term,lsn)` exactly like a job entry; the config *bytes* travel out-of-band and are **hash-verified against the committed marker before activation** (mismatch ⇒ loud serve-fence, stay alive + retry). + +## Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling) + +The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commit.c`, see `persistence.md`) is **authoritative** for advancing `committed_lsn`. The raft library's `commit_index` (fed via `on_commit_advanced` after `raft_advance_commit`) is kept **advisory only** — it is NOT folded in as a hard ceiling (`committed = min(durable-majority, raft_commit_index)` was tried and removed). Rationale: under the single-writer model the durable-majority watermark is *already* a current-term `(term,lsn)`-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a `min()` ceiling **starves commit below the safe durable majority** (jobs acked-durable on a majority but never committed). Keep driving the library (for its own `match_index`/`next_index`/send state) but never let its commit index *gate* `committed_lsn`. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling. + ## Dispatch Gate `ha_dispatch_gate` controls which opcodes are accepted per role: diff --git a/.kiro/steering/persistence.md b/.kiro/steering/persistence.md index 0b17e26..8f19c97 100644 --- a/.kiro/steering/persistence.md +++ b/.kiro/steering/persistence.md @@ -13,7 +13,7 @@ The load-bearing rule: **a member contributes to the commit majority for an LSN `persist_t.durable_lsn` (`_Atomic uint64_t`) is the highest LSN whose bytes are `fdatasync`'d locally. It is a **prefix watermark**, never a per-entry set: `durable_lsn ≥ L` implies every entry `≤ L` is durable (because the WAL is a contiguous single-writer log). Invariants: - `durable_lsn <= current_lsn - 1` (you cannot be durable past what you have appended). -- Monotonic non-decreasing **except** clamp-on-truncate (`persist_truncate_wal_above` clamps `durable_lsn` down to the new tip). +- Monotonic non-decreasing **except** clamp-on-truncate (`persist_truncate_wal_above` clamps `durable_lsn` down to the new tip). **Because the clamp lowers `durable_lsn`, a truncate cutoff that lags the true commit point physically destroys committed entries** — only ever call `persist_truncate_wal_above` with the votes-derived post-election `safe_committed_lsn` or the library `(term,lsn)` conflict point, NEVER with the lagging `last_known_committed_lsn` (bug-828). - Advances **only after a successful `fdatasync`** — fail-loud: on `fdatasync` error the watermark holds and an ERROR is logged. Never advance a watermark on a failed sync. Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_candidate_locked` in `commit.c`) uses exactly this value.