myxce/.kiro/specs/raft-commit-safety/design.md
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

392 lines
34 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Design Document
## Overview
**Purpose**: This feature adds the **commit-safety and election-freshness** rules of `raft/raft.tla` (Ongaro canonical) to the pure Raft library (`src/lib/raft/raft.{c,h}`), on top of the log half delivered by `raft-log-replication`. It closes the two remaining safety gaps of bug-851: **G4** (term-blind commit advance) and **G6** (LSN-only vote freshness).
**Users**: The XCE controller (as a Raft node, which will route its commit barrier through the new entry point in the downstream `ha-term-safety` integration), the deterministic simulator (`raft_sim.c`), and the downstream `raft-sim-conformance` spec (Leader Completeness oracle).
**Impact**: Today the commit position advances **term-blind** in the controller (`ha_compute_safe_committed_lsn` + the commit barrier) — it can mark a previous-term entry committed by replica-count alone (G4) — and vote freshness compares LSNs only, never the last-log term (G6). After this feature, the library owns `commit_index` and advances it exactly as `raft.tla` `AdvanceCommitIndex` does (quorum agreement **and** current-term entry), notifying the controller via `on_commit_advanced` to **trigger** the existing synchronous commit barrier (the barrier itself is unchanged and remains the controller's). Vote and PreVote freshness become the `raft.tla` `(lastLogTerm, lastLogIndex)` lexicographic comparison. The change is a pure, synchronous, deterministic addition to the core; all durability/I/O stays in adapters.
### Goals
- Add a library-owned `commit_index` and `raft_advance_commit(...)` implementing `raft.tla` `AdvanceCommitIndex` (line 259) **with** the current-term rule `log[Max(agreeIndexes)].term == currentTerm` (line 270) — the Figure-8 safety rule (G4).
- Add the required callback `on_commit_advanced(ctx, commit_index)` that **triggers** the existing synchronous commit barrier; the library never performs the barrier itself.
- Upgrade vote freshness to the `raft.tla` `HandleRequestVoteRequest` `logOk` `(lastLogTerm, lastLogIndex)` comparison (line 285) on both the binding vote path and the PreVote eligibility path (G6).
- Extend `raft_sim.c` to assert the **Figure-8** refusal and **Leader Completeness** across forced leader changes.
### Non-Goals
- **No log model / AppendEntries / per-peer progress** — owned by `raft-log-replication` and reused verbatim (`log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`, `on_persist_entry`, `on_send_append_entries`, `on_truncate_from`, `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`).
- **No commit barrier / durability mechanism** — the synchronous barrier, fsync/group-commit, persist-before-ACK, reply gate, and snapshot transport stay in the controller (`ha-term-safety` adapter). This spec only triggers the barrier.
- **No config-change markers / upgrade snapshot** — owned by `ha-term-safety`.
- **No conformance / TLC trace-check harness** — owned by `raft-sim-conformance`.
- **No controller wiring** — replacing `ha_compute_safe_committed_lsn`'s term-blind advance with the library's `commit_index` is `ha-term-safety`; this spec defines only the library entry point + callback it will route through.
## Boundary Commitments
### This Spec Owns
- The library `commit_index` field in `raft_state_t` and its restore field in `raft_config_t` (`initial_commit_index`, default 0).
- The new entry point `raft_advance_commit(raft_state_t *s)` implementing `AdvanceCommitIndex` with the current-term rule.
- The new required callback `on_commit_advanced` in `raft_callbacks_t` (fired only on strict advance) + its `raft_init` validation.
- The `(lastLogTerm, lastLogIndex)` freshness upgrade on the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path (`raft_on_pre_vote_request`), including carrying `last_log_term` alongside `last_log_index`.
- Simulator extensions in `raft_sim.c` + tests in `src/lib/testsuite/test_raft.c` asserting Figure-8 refusal and Leader Completeness.
### Out of Boundary
- The log model, AppendEntries, and per-peer `match_index[]` themselves (`raft-log-replication`) — consumed verbatim, never redefined.
- The synchronous commit barrier, fsync/group-commit, persist-before-ACK durability, reply gate, snapshot transport (controller / `ha-term-safety`).
- Config markers in the unified log + the change-boundary snapshot (`ha-term-safety`).
- The conformance / TLC trace-check harness (`raft-sim-conformance`).
- The controller's replacement of `ha_compute_safe_committed_lsn` with `commit_index` and the controller's freshness-floor plumbing of `last_log_term` onto the wire (`ha-term-safety`). This spec specifies only **what the library entry point/callback route**, not the controller mechanics.
### Allowed Dependencies
- The implemented election core (`raft-library`) and the log half (`raft-log-replication`): `raft_state_t`, `raft_callbacks_t`, `raft_init`, the vote/PreVote path, `log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`. All reused; the vote path is extended additively.
- `errno.h`, `stdint.h`, `string.h` only — the core stays free-standing for the simulator.
- The simulator harness (`raft_sim.c`) for verification.
- Must NOT depend on any controller header, WAL type, capnp type, or transport — the core stays pure.
### Revalidation Triggers
Dependent specs (`raft-sim-conformance`, `ha-term-safety`) must re-check integration if any of the following change:
- The signature or semantics of `raft_advance_commit` or the `on_commit_advanced` callback (especially the strict-advance / fire-once rule and "trigger only, never performs the barrier").
- The placement of the current-term rule (`on_get_entry_term(Max(agreeIndexes)) == currentTerm`) or the `agreeIndexes`/quorum definition.
- The freshness signature change — the addition of `last_log_term` to `raft_on_vote_request`, the PreVote eligibility path, and any vote-request payload / `on_vote_eligible` extension.
- The `commit_index` restore convention (`initial_commit_index`, default 0) or its monotonicity guarantee.
## Architecture
### Existing Architecture Analysis
After `raft-log-replication`, the library owns the log as pure, synchronous, deterministic state: per-entry term via `on_get_entry_term`, per-peer `next_index[]` / `match_index[]`, and `leader_commit` carried **opaquely** (never acted on). The election core delegates freshness to `on_vote_eligible`, which today receives only `(from_idx, term)` — the caller checks `config_tag` / applied-LSN floor but has no last-log **term**. This feature follows the same discipline:
- **Pure / synchronous / deterministic core** — `raft_advance_commit` reads entry terms via `on_get_entry_term` and emits a single `on_commit_advanced` notification; it owns no entry bytes and performs no I/O. The freshness upgrade is a predicate computed from values already in the library log model plus the candidate's advertised `(last_log_term, last_log_index)`.
- **Persist-before-act ordering** — unchanged on the vote path: a vote that passes the new `(lastLogTerm, lastLogIndex)` predicate is still persisted via `on_persist` before being granted. `commit_index` advance is not itself persisted by the library — durability of the committed position remains the controller barrier's job, triggered by `on_commit_advanced`.
- **Status returns, never self-terminate** — `raft_advance_commit` returns a status; no new path aborts.
- **Separate actions** — mirroring `raft.tla`, `AdvanceCommitIndex` is a **distinct** action from `HandleAppendEntriesResponse`. `raft_on_append_entries_reply` (owned upstream) only updates `match_index[]`; commit advance is the separate `raft_advance_commit` call. This keeps atomic regions small and lets a single-node leader commit (the `{self}` term in `agreeIndexes`).
### Architecture Pattern & Boundary Map
```mermaid
flowchart TB
subgraph PureCore["src/lib/raft/raft.c — PURE CORE (this spec)"]
ADV["raft_advance_commit()<br/>AdvanceCommitIndex (tla:259)<br/>+ current-term rule (tla:270)"]
VOTE["raft_on_vote_request()<br/>logOk (lastLogTerm,lastLogIndex) (tla:285) G6"]
PVOTE["raft_on_pre_vote_request()<br/>same (lastLogTerm,lastLogIndex) freshness"]
end
subgraph Upstream["raft-log-replication — REUSED VERBATIM (not this spec)"]
MI["match_index[] / log_len"]
GET["on_get_entry_term(index) -> term|0"]
REPLY["raft_on_append_entries_reply()<br/>updates match_index[] only"]
end
subgraph CBs["raft_callbacks_t — NEW seam (this spec)"]
C5["on_commit_advanced(ctx, commit_index)<br/>fired ONLY on strict advance"]
end
subgraph Consumers["ADAPTERS — implement callback (NOT this spec's heavy wiring)"]
CTRL["controller adapter (ha-term-safety):<br/>on_commit_advanced -> EXISTING sync commit barrier<br/>(persist-before-ACK preserved, NOT replaced)"]
SIM["raft_sim.c: drive raft_advance_commit,<br/>assert Figure-8 + Leader Completeness (this spec)"]
end
REPLY -.updates.-> MI
ADV --> MI & GET
ADV --> C5
VOTE --> GET
PVOTE --> GET
C5 --> CTRL
C5 --> SIM
ADV --> SIM
```
**Architecture Integration**:
- **Selected pattern**: Pure state machine + callback ports (Hexagonal/Ports-and-Adapters), already established by the election and log cores. Commit advance is a new pure action; `on_commit_advanced` is a new port adapters implement.
- **Domain boundaries**: the commit-safety **decision** (which index is safe to commit, given quorum + current-term) stays in the library; the durable **act** (the synchronous commit barrier, fsync, ACK) stays in the controller adapter, triggered by the port.
- **Existing patterns preserved**: `raft_config_t` extensibility, persist-before-act on the vote path, NULL-checked required callbacks, status-code returns, `XCE_TESTING`-gated simulator.
- **Steering compliance**: matches `src/lib/CLAUDE.md` (free-standing C, gnu17), `ha-async-io.md` (no blocking in the core; the barrier/async lives in the adapter), `testing.md` (TDD: simulator/test first), `ha-async-io.md` durability policy ("persist-before-ACK + synchronous commit barrier" preserved — the callback triggers it), "controller never self-terminates" (library returns statuses only).
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Library core | C (gnu17), `errno.h`/`stdint.h`/`string.h` only | Pure commit-safety + freshness | Free-standing; no controller/capnp deps |
| Callbacks | `raft_callbacks_t` function pointers | Commit-advance trigger seam | 1 new required callback (`on_commit_advanced`) |
| Verification | `raft_sim.c` (XCE_TESTING) + `test_raft.c` | Figure-8 + Leader Completeness | Reuses the in-memory log backing from `raft-log-replication` |
| Adapter (downstream) | controller commit barrier (`ha-term-safety`) | Durable commit / ACK | Not built here; only triggered |
## File Structure Plan
### Modified Files
- `src/lib/raft/raft.h` — Add `uint64_t commit_index;` to `raft_state_t`; add `uint64_t initial_commit_index;` to `raft_config_t`; add the `on_commit_advanced` callback to `raft_callbacks_t`; declare `raft_advance_commit`; extend the vote-path signatures to carry `last_log_term` (alongside the existing `last_log_index`) per the chosen freshness-plumbing option below; add a Figure-8 / Leader-Completeness driver + assertion helper to the simulator section.
- `src/lib/raft/raft.c` — Implement `raft_advance_commit` (agreeIndexes + current-term rule + monotonic strict-advance firing `on_commit_advanced`); add the `on_commit_advanced` NULL check + `initial_commit_index` restore in `raft_init`; upgrade the freshness predicate in `raft_on_vote_request` and `raft_on_pre_vote_request` to the `(lastLogTerm, lastLogIndex)` comparison.
- `src/lib/raft/raft_sim.c` — Drive `raft_advance_commit` after delivery; the Figure-8 construction; the Leader-Completeness assertion across forced leader changes; the freshness-gate exercise.
- `src/lib/testsuite/test_raft.c` — New tests for `raft_advance_commit` (current-term hold/advance, monotonicity, non-leader), `on_commit_advanced` fire-once/never semantics, the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness, and the simulator invariants (TDD-first).
### Directory Structure (unchanged)
```
src/lib/raft/
├── raft.h # + commit_index, on_commit_advanced, raft_advance_commit, last_log_term on vote path
├── raft.c # + AdvanceCommitIndex(current-term rule), (lastLogTerm,lastLogIndex) freshness
└── raft_sim.c # + commit-advance driver, Figure-8, Leader Completeness assert
src/lib/testsuite/
└── test_raft.c # + commit-safety + freshness + Figure-8/Leader-Completeness tests
```
> Each file keeps one responsibility: `raft.c` = pure protocol, `raft_sim.c` = deterministic harness, `test_raft.c` = assertions. No new file is introduced; the existing module pattern is extended.
## System Flows
### Commit advance: separate action after replication (raft.tla AdvanceCommitIndex)
```mermaid
sequenceDiagram
participant L as Leader (raft.c)
participant LC as Leader adapter (cb)
Note over L: replication already updated match_index[]<br/>via raft_on_append_entries_reply (raft-log-replication)
Note over L: raft_advance_commit() — separate action (tla:259)
L->>L: agreeIndexes = { i in 1..log_len :<br/>{self} ∪ {p: match_index[p]>=i} ∈ Quorum }
alt agreeIndexes empty
Note over L: hold commit_index (else-branch)
else Max(agreeIndexes) from a PREVIOUS term
L->>LC: on_get_entry_term(Max) != currentTerm
Note over L: hold commit_index (Figure-8 refusal, G4)
else Max(agreeIndexes) from CURRENT term
L->>LC: on_get_entry_term(Max) == currentTerm
Note over L: commit_index = Max(agreeIndexes) (strict advance)
L->>LC: on_commit_advanced(ctx, commit_index)
Note over LC: trigger EXISTING sync commit barrier<br/>(persist-before-ACK, NOT replaced)
end
```
### Vote freshness: (lastLogTerm, lastLogIndex) comparison (raft.tla HandleRequestVoteRequest logOk)
```mermaid
sequenceDiagram
participant V as Voter (raft.c)
participant VC as Voter adapter (cb)
Note over V: raft_on_vote_request(..., cand_last_log_term, cand_last_log_index)
Note over V: UpdateTerm / stale-term / one-vote-per-term (unchanged)
V->>VC: on_get_entry_term(log_len) -> local_last_log_term (0 if empty)
Note over V: logOk = cand_last_log_term > local_last_log_term<br/>∨ (== ∧ cand_last_log_index >= log_len)
alt !logOk
Note over V: DENY (even if all other gates would grant) — G6
else logOk AND application gate AND invariants pass
V->>VC: on_persist(term, from_idx) %% persist-before-grant unchanged
Note over V: GRANT
end
```
Flow-level decisions:
- **Commit is a separate action** — `match_index[]` is updated by `raft_on_append_entries_reply` (upstream); the commit index advances only when `raft_advance_commit` is driven, exactly as `raft.tla` separates the two actions (R2.6). This lets a single-node leader commit (`{self}` ∈ `agreeIndexes`).
- **Current-term rule is the Figure-8 gate** — `commit_index` advances to `Max(agreeIndexes)` only when that entry is from `currentTerm`; a previous-term entry replicated to a quorum is **held** until a current-term entry above it commits, then committed indirectly (R1.3/R1.4, closes G4).
- **`on_commit_advanced` fires only on strict advance** — never on a held commit index, never from `raft_on_append_entries_reply` (R2.2/R2.3/R2.6).
- **Freshness denies before any grant** — a stale `(lastLogTerm, lastLogIndex)` denies even when the term/vote invariants and the application gate would grant (R3.4, closes G6); persist-before-grant is unchanged (R3.6).
## Freshness Plumbing — design decision
`raft.tla` `logOk` needs the candidate's `(lastLogTerm, lastLogIndex)` and the voter's own. Two options were considered for carrying the candidate values into the library:
- **Option A — extend the `on_vote_eligible` callback signature** to `on_vote_eligible(ctx, from_idx, term, last_log_term, last_log_index)` and have the library compute `logOk` against `on_get_entry_term(log_len)`/`log_len` before calling the gate. **Rejected as the sole mechanism**: it pushes the protocol-defining `logOk` either into the adapter (violating "library owns protocol decisions") or splits it.
- **Option B (chosen) — the library owns `logOk`.** The candidate's `last_log_term` and `last_log_index` are accepted as explicit arguments on `raft_on_vote_request` (and on the PreVote eligibility path), the library computes `logOk` against its own `on_get_entry_term(log_len)` / `log_len`, and only then consults the unchanged application gate `on_vote_eligible` for XCE-specific policy (primary-deny / heartbeat stickiness / catch-up). This keeps the `raft.tla` `logOk` predicate **inside** the library (R3.1, R5.5) while leaving `on_vote_eligible` for application policy that is genuinely outside Raft.
The candidate advertises its `(last_log_term, last_log_index)` in the `RequestVote` / PreVote wire RPC; the controller adapter decodes them and passes them to the library entry point. The on-wire encoding of `last_log_term` (and the controller's freshness-floor plumbing) is the `ha-term-safety` adapter's job — out of this spec's scope; this spec fixes only the library signatures and the predicate.
> **Note — freshness-decision locus moves from caller/dispatcher into the library (both paths).** Today the freshness decision lives **outside** the library: `raft.h` (lines 246–248) documents that PreVote freshness "is applied by the dispatcher before this call", and the binding vote path delegates the decision to `on_vote_eligible`. Option B **moves that decision locus into the library** for **both** the binding (`raft_on_vote_request`) and the PreVote (`raft_on_pre_vote_request`) paths: `raft.c` now computes the `(lastLogTerm, lastLogIndex)` `logOk` itself rather than relying on the caller/dispatcher. This **supersedes the existing `raft.h` lines 246–248 doc contract**, and updating those `raft.h` doc comments (so the header no longer states the dispatcher owns PreVote freshness) is part of the header-contract task.
## Requirements Traceability
| Requirement | Summary | Design Elements | raft.tla action |
|-------------|---------|-----------------|-----------------|
| 1.1–1.7 | Library `commit_index`, agreeIndexes, current-term rule, monotonic, leader-only | `raft_state_t.commit_index`; `raft_advance_commit()`; `on_get_entry_term`; `match_index[]` (upstream) | `AdvanceCommitIndex` (259), current-term rule (270) |
| 2.1–2.6 | `on_commit_advanced` trigger, fire-once, separate from reply | `raft_callbacks_t.on_commit_advanced`; strict-advance firing; `raft_init` validation | (seam; `AdvanceCommitIndex` distinct from `HandleAppendEntriesResponse`) |
| 3.1–3.6 | `(lastLogTerm,lastLogIndex)` binding-vote freshness | `raft_on_vote_request` logOk; `on_get_entry_term(log_len)`/`log_len`; persist-before-grant | `HandleRequestVoteRequest` logOk (285) |
| 4.1–4.5 | Same freshness on PreVote, side-effect-free | `raft_on_pre_vote_request` logOk; no persist/term/role mutation | `HandleRequestVoteRequest` logOk (285), PreVote extension |
| 5.1–5.4 | Barrier/durability preserved as policy; PreVote/priority unchanged; no abort | `on_commit_advanced` as trigger only; additive-only changes | (policy; not a TLA+ action) |
| 6.1–6.6 | Simulator Figure-8 + Leader Completeness + freshness gate | `raft_sim.c` commit driver, Figure-8 construction, Leader-Completeness assert, freshness exercise | Figure-8 / Leader Completeness invariant |
## Components and Interfaces
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| `commit_index` (`raft_state_t`) | Library core | Library-owned commit position | 1 | log core (P0) | State |
| `raft_advance_commit` | Library core | AdvanceCommitIndex + current-term rule | 1, 2 | `on_get_entry_term`, `match_index[]`, `on_commit_advanced` (P0) | Service |
| `on_commit_advanced` callback | Seam | Commit-advance trigger | 2, 5 | controller barrier / sim (P0) | Service |
| `raft_on_vote_request` freshness | Library core | `(lastLogTerm,lastLogIndex)` logOk | 3, 5 | `on_get_entry_term`, `log_len`, `on_persist`, `on_vote_eligible` (P0) | Service |
| `raft_on_pre_vote_request` freshness | Library core | Same logOk, side-effect-free | 4, 5 | `on_get_entry_term`, `log_len`, `on_vote_eligible` (P0) | Service |
| Simulator | Verification | Figure-8 + Leader Completeness | 6 | all of the above (P0) | State |
### Library Core
#### `commit_index` addition to `raft_state_t`
| Field | Detail |
|-------|--------|
| Intent | Hold the library's view of the safe commit position (TLA+ `commitIndex`) |
| Requirements | 1.1, 1.5 |
**Responsibilities & Constraints**
- `uint64_t commit_index;` — the highest 1-based index (== WAL LSN) the library considers safely committed. Monotonic non-decreasing.
- Restored at init from the adapter (`raft_config_t.initial_commit_index`, default 0; the library never reads storage — consistent with `initial_term` / `initial_log_len`).
- The library never advances `commit_index` except through `raft_advance_commit`; `raft_on_append_entries_reply` (upstream) leaves it untouched (R2.6).
#### `raft_advance_commit` — AdvanceCommitIndex (raft.tla:259) with the current-term rule (line 270)
| Field | Detail |
|-------|--------|
| Intent | Advance `commit_index` to `Max(agreeIndexes)` only when that entry is from `currentTerm` |
| Requirements | 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3 |
##### Service Interface
```c
/*
* raft_advance_commit: leader recomputes its commit index (raft.tla
* AdvanceCommitIndex, 259). A SEPARATE action from append-entries reply
* handling (mirrors raft.tla, so a single-node leader can commit and to
* keep atomic regions small).
*
* Computes agreeIndexes = { index in 1..log_len :
* ({self} ∪ {peer : match_index[peer] >= index}) is a quorum }.
* If agreeIndexes is non-empty AND on_get_entry_term(Max(agreeIndexes))
* == s->term, sets commit_index = Max(agreeIndexes) (Figure-8 current-term
* rule) and, on a STRICT advance, fires on_commit_advanced(ctx, commit_index)
* exactly once.
* Otherwise leaves commit_index unchanged and does NOT fire on_commit_advanced.
*
* The library performs NO durability work here — on_commit_advanced only
* TRIGGERS the controller's existing synchronous commit barrier.
*
* Returns RAFT_OK on success (whether or not the index advanced);
* -EPERM if role != RAFT_LEADER (commit_index left unchanged).
*/
int raft_advance_commit(raft_state_t *s);
```
- Preconditions: `s->role == RAFT_LEADER` (else `-EPERM`, no change — R1.6).
- Postconditions: `commit_index` either unchanged or advanced to `Max(agreeIndexes)` when that entry is current-term; `on_commit_advanced` fired iff `commit_index` strictly increased.
- Invariants: `commit_index` never decreases (R1.5); the library reads entry terms only via `on_get_entry_term` (R1.7); `commit_index ≤ log_len`.
> **Quorum**: `(n_peers / 2) + 1`, identical to the election-core quorum; `match_index[self]` is implicitly `log_len` (self always agrees up to its own log length), realized by seeding the agree-count with self for each candidate index. `agreeIndexes` is computed by scanning candidate indexes from `log_len` downward and taking the first that reaches quorum (equivalent to `Max(agreeIndexes)` without materializing the set).
#### Freshness upgrade — `raft_on_vote_request` (raft.tla:285, closes G6)
| Field | Detail |
|-------|--------|
| Intent | Add the `(lastLogTerm, lastLogIndex)` lexicographic logOk before the application gate |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 |
##### Service Interface
```c
/*
* raft_on_vote_request: process an incoming RequestVote RPC (extended).
* Adds the raft.tla HandleRequestVoteRequest logOk (285): the library
* computes
* logOk = (cand_last_log_term > LastTerm(local))
* || (cand_last_log_term == LastTerm(local)
* && cand_last_log_index >= log_len)
* where LastTerm(local) = (log_len > 0) ? on_get_entry_term(log_len) : 0.
* Step order (unchanged except the new logOk gate):
* 1. stale-term deny / UpdateTerm (unchanged)
* 2. role gate, one-vote-per-term (unchanged)
* 3. logOk gate (NEW, G6): deny if !logOk, before on_vote_eligible
* 4. application gate on_vote_eligible (unchanged)
* 5. on_persist before grant (unchanged)
* Returns RAFT_GRANT / RAFT_DENY; sets *reply_term.
*/
int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term,
uint64_t cand_last_log_term,
uint64_t cand_last_log_index,
uint64_t *reply_term);
```
- Preconditions: none beyond a valid `s`.
- Postconditions: a vote is granted only when the existing invariants, the new `(lastLogTerm, lastLogIndex)` logOk, and the application gate all pass; persisted before grant.
- Invariants: `logOk` is evaluated **before** `on_vote_eligible`; a `!logOk` denies regardless of application policy (R3.4); no weakening of stale-term / one-vote-per-term (R3.5).
> **Signature change is a Revalidation Trigger.** `raft_on_vote_request` gains `cand_last_log_term` and `cand_last_log_index`. The controller and the simulator (the two callers) must pass the candidate's advertised last-log `(term, index)`. The PreVote path mirrors this.
#### Freshness upgrade — `raft_on_pre_vote_request` (side-effect-free)
| Field | Detail |
|-------|--------|
| Intent | Apply the identical `(lastLogTerm, lastLogIndex)` logOk on the straw poll, mutating nothing |
| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5 |
##### Service Interface
```c
/*
* raft_on_pre_vote_request: answer a PreVote probe (extended).
* Applies the SAME (lastLogTerm, lastLogIndex) logOk as the binding vote
* (Requirement 3.1), in addition to the existing stale-term deny and the
* on_vote_eligible gate. Mutates NOTHING: no term/voted_for/role/commit_index
* change, no on_persist (PreVote invariant preserved).
* Returns RAFT_GRANT / RAFT_DENY.
*/
int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term,
uint64_t cand_last_log_term,
uint64_t cand_last_log_index);
```
- Preconditions: none beyond a valid `s`.
- Postconditions: grant only when stale-term, logOk, and the application gate all pass; no persistent state touched.
- Invariants: identical freshness rule to the binding path (R4.1, R4.2); side-effect-free (R4.4).
### Seam — `raft_callbacks_t` addition (verbatim Authoritative shared contract)
Appended to `raft_callbacks_t` and validated **required** in `raft_init`. Reproduced verbatim from the brief so downstream specs depend on the exact shape:
```c
/* REQUIRED: fired AFTER commit_index strictly advances. The controller adapter
* feeds this into the EXISTING synchronous commit barrier / persist-before-ACK
* durability — this callback TRIGGERS commit-eval; it does NOT replace the
* barrier. */
void on_commit_advanced(void *ctx, uint64_t commit_index);
```
**Contract rules (R2.4, R5.1)**: the library decides *whether* the commit index advanced; the callback only triggers the adapter's commit work. It is fired exactly once per strict advance (R2.2), never on a held commit index (R2.3), and never from `raft_on_append_entries_reply` (R2.6). It is NULL-checked and required in `raft_init` (`-EINVAL` if NULL — R2.5), matching how the other required callbacks are validated.
### Controller adapter (seam only — heavy wiring is `ha-term-safety`)
This spec describes the adapter **only at the seam**; the full controller integration is the downstream `ha-term-safety` consumer.
- `on_commit_advanced` → the controller routes the new safe `commit_index` into its **existing** synchronous commit barrier (commit-eval sweep, persist-before-ACK, reply gate) — replacing the term-blind `ha_compute_safe_committed_lsn` advance with the library's current-term-gated value. The barrier mechanism is unchanged; only its **input** (the safe commit position) now comes from the library.
- The candidate's advertised `(last_log_term, last_log_index)` is decoded from the `RequestVote` / PreVote wire RPC and passed to `raft_on_vote_request` / `raft_on_pre_vote_request`; `last_log_term` is read back from the WAL term (`on_get_entry_term`-backed index) on recovery. The on-wire encoding and the freshness-floor plumbing are `ha-term-safety`.
- On recovery, the controller restores `commit_index` (= recovered committed LSN) into `raft_config_t.initial_commit_index`.
## Data Models
### Domain Model
- **Commit position (logical)**: `commit_index : uint64` — the highest 1-based index (== WAL LSN) safely committed, owned by the library, monotonic.
- **Convention**: callers pass `s->commit_index` as the `leader_commit` argument to `raft_send_append_entries` — the leader's committed value rides AppendEntries from `commit_index`, which this spec owns (upstream `raft-log-replication` carries `leader_commit` opaquely; the value it carries is the `commit_index` defined here).
- **Freshness tuple**: `(last_log_term : uint64, last_log_index : uint64)` per candidate, compared lexicographically against the voter's own `(on_get_entry_term(log_len), log_len)`.
### Logical Data Model
- `raft_state_t` gains `commit_index`.
- `raft_config_t` gains `initial_commit_index` (restored committed LSN at boot; 0 on first boot — the library never reads storage, consistent with `initial_term` / `initial_log_len`).
- No new persisted on-disk format: `commit_index` durability is the controller barrier's existing responsibility; the library imposes no new persisted layout. `last_log_term` reuses the WAL `hdr.term` already written by `raft-log-replication`'s adapter.
### Data Contracts & Integration
- **Freshness tuple**: `raft_on_vote_request` and `raft_on_pre_vote_request` accept `(cand_last_log_term, cand_last_log_index)` as explicit arguments. The voter's own values come from `on_get_entry_term(log_len)` (term `0` when `log_len == 0`, consistent with `raft.tla` `LastTerm`) and `log_len`.
- **Commit trigger**: `on_commit_advanced(ctx, commit_index)` carries only the new committed index; no entry bytes, no term — the adapter looks up whatever it needs through its own state.
## Error Handling
### Error Strategy
The library returns status codes only and **never aborts** (preserves "controller never self-terminates"). Callbacks for commit-advance are trigger-only and return void; the freshness path keeps the existing persist-failure handling.
### Error Categories and Responses
- **Caller misuse**: `raft_advance_commit` while not leader → `-EPERM`, `commit_index` unchanged, no `on_commit_advanced`. NULL `on_commit_advanced` at `raft_init` → `-EINVAL`.
- **Held commit (Figure-8)**: when `Max(agreeIndexes)` is a previous-term entry, `commit_index` is held — this is the correct safety behavior, not an error; no callback fires.
- **Empty agreeIndexes**: no quorum agrees on any index → hold, no callback.
- **Stale freshness**: `!logOk` → vote/PreVote DENY (not an error; the standard freshness rejection).
- **Persist failure on grant**: unchanged from the existing vote path — a failed `on_persist` denies the vote without recording it.
### Monitoring
Reuse the optional `on_trace` callback for new events: `"commit_advance"`, `"commit_hold_prevterm"`, `"commit_hold_noquorum"`, `"vote_denied_log_term"` (stale `lastLogTerm`), `"vote_denied_log_index"` (tie term, stale index), `"pre_vote_denied_freshness"`. NULL-checked exactly like existing trace calls.
## Testing Strategy
TDD-mandatory (`testing.md`): tests and simulator drivers are written **before** the `raft.c` implementation (RED → GREEN). All tests are `XCE_TESTING`-gated and live in `src/lib/testsuite/test_raft.c` driving `raft_sim.c`. They reuse the in-memory per-node log backing added by `raft-log-replication`.
### Unit Tests (entry points, against the in-memory sim log)
1. `raft_advance_commit` current-term advance: a leader whose `match_index[]` shows a quorum agreeing through index `k`, with the entry at `k` from `currentTerm`, advances `commit_index` to `k` and fires `on_commit_advanced(k)` once (R1.2, R1.3, R2.2).
2. `raft_advance_commit` Figure-8 hold: a leader with a quorum agreeing through a **previous-term** entry at `k` (no current-term entry above it) holds `commit_index` and fires nothing (R1.4, R2.3 — G4).
3. `raft_advance_commit` monotonicity + non-leader: never decreases `commit_index`; returns `-EPERM` with no change and no callback when not leader (R1.5, R1.6).
4. `on_commit_advanced` separation: `raft_on_append_entries_reply` updating `match_index[]` does **not** by itself advance `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does (R2.6).
5. `raft_init` requires `on_commit_advanced`: `-EINVAL` when NULL; restores `initial_commit_index` (R2.5, R1.1).
6. `(lastLogTerm, lastLogIndex)` vote freshness: a candidate with a stale `last_log_term` is denied even with a greater `last_log_index`; a candidate with a fresher `last_log_term` is granted with a smaller `last_log_index`; a tie on term grants only when `last_log_index >= log_len` (R3.1, R3.4). PreVote applies the identical rule with no persist/term/role mutation (R4.1, R4.4).
### Integration / Simulator Tests (Figure-8, Leader Completeness)
1. **Figure-8 scenario**: construct a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does **not** move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over **both** (indirect commit) (R6.2, R6.3 — G4).
2. **Leader Completeness**: across forced leader changes (timeout / partition-driven), assert that any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader (R6.4) — the `(lastLogTerm, lastLogIndex)` freshness gate (R6.5) is what makes this hold, so the two tests are complementary.
3. **Freshness gate exercise**: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome (stale-term-but-higher-index denied; fresher-term-but-lower-index granted) (R6.5).
4. **Trigger semantics**: assert `on_commit_advanced` fires exactly on strict advance and not while the current-term rule holds the commit index (R6.6).