lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

31
.kiro/specs/raft-commit-safety/brief.md Normal file
View file

@ -0,0 +1,31 @@
# Brief: raft-commit-safety
## Problem
After `raft-log-replication` gives the library a real `(term, index)` log, two `raft.tla` commit/election-safety rules are still missing in XCE (gap analysis: `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`):
- **G4** — commit-index advance is **term-blind**. `raft.tla` `AdvanceCommitIndex` (line 259) only advances `commitIndex` to `Max(agreeIndexes)` **if `log[Max].term == currentTerm`** (the Figure-8 safety rule: a leader never commits a previous-term entry by replica-count alone). XCE's commit barrier + `ha_compute_safe_committed_lsn` advanced committed "from 1 granted voter" with no current-term check (because the WAL term was dead); the own-term `WAL_NOOP` gate is only a partial mitigation.
- **G6** — vote freshness is **LSN-only**. `raft.tla` `HandleRequestVoteRequest` `logOk` (line 285) compares `(lastLogTerm, lastLogIndex)` lexicographically; the library's `on_vote_eligible` gate uses `config_tag ≥` + `applied LSN ≥` only — it never compares `lastLogTerm`.
## Approach
Add the commit-safety and freshness rules to the library, strictly per `raft.tla`, on top of the `raft-log-replication` log model.
### TLA+ actions implemented
- `AdvanceCommitIndex(i)` (raft.tla:259) → `raft_advance_commit(...)`: compute `agreeIndexes` from per-peer `matchIndex` (a quorum agrees), and advance `commitIndex` to `Max(agreeIndexes)` **only when `on_get_entry_term(Max) == currentTerm`**. Otherwise hold.
- `HandleRequestVoteRequest` `logOk` (raft.tla:285) → upgrade the freshness gate to `(lastLogTerm, lastLogIndex)`: grant only if `cand.lastLogTerm > local.lastLogTerm ∨ (== ∧ cand.lastLogIndex ≥ local.lastLogIndex)`, applied to both `raft_on_vote_request` and the PreVote eligibility path.
## Authoritative shared contract
Uses the `raft-log-replication` contract verbatim (1-based index = WAL LSN; per-entry `uint64_t term`; `on_get_entry_term`, per-peer `matchIndex`). New callback:
- `void on_commit_advanced(void *ctx, uint64_t commit_index)` — fired AFTER `commitIndex` advances; the controller adapter feeds this into the **existing synchronous commit barrier / durability** (persist-before-ACK preserved — this callback does not replace the barrier, it triggers commit-eval).
Vote RPC freshness gains `last_log_term` alongside the existing `last_log_index`/`config_tag` carried to `on_vote_eligible` (extend the eligibility signature or the vote-request payload accordingly).
## Scope
- **In**: `AdvanceCommitIndex` with the current-term rule; `(lastLogTerm, lastLogIndex)` vote/PreVote freshness; `on_commit_advanced` callback wired to the controller commit barrier; simulator assertion of **Leader Completeness** (a committed entry is present in every future leader's log).
- **Out**: the log model + AppendEntries (owned by `raft-log-replication`); the conformance harness (`raft-sim-conformance`); config markers/snapshot (`ha-term-safety`); the barrier/durability mechanism itself (controller — this spec only triggers it via `on_commit_advanced`).
## Preserve as policy
The synchronous commit barrier + persist-before-ACK durability is STRONGER than canonical Raft async apply and MUST remain: `on_commit_advanced` triggers the existing barrier; it does not weaken it. PreVote and priority leadership unchanged.
## Verification
Extend `raft_sim.c`: construct Figure-8 scenarios (a previous-term entry replicated to a quorum but not yet committed by the current leader) and assert the library refuses to commit it until a current-term entry is committed; assert **Leader Completeness** across forced leader changes. Replace `ha_compute_safe_committed_lsn`'s term-blind advance with the library's current-term-gated `commitIndex`.
## Dependencies
`raft-log-replication` (HARD — needs the `(term,index)` log + `matchIndex`). Feeds `raft-sim-conformance` and `ha-term-safety`. Closes bug-851 G4/G6.

392
.kiro/specs/raft-commit-safety/design.md Normal file
View file

@ -0,0 +1,392 @@
# Design Document
## Overview
**Purpose**: This feature adds the **commit-safety and election-freshness** rules of `raft/raft.tla` (Ongaro canonical) to the pure Raft library (`src/lib/raft/raft.{c,h}`), on top of the log half delivered by `raft-log-replication`. It closes the two remaining safety gaps of bug-851: **G4** (term-blind commit advance) and **G6** (LSN-only vote freshness).
**Users**: The XCE controller (as a Raft node, which will route its commit barrier through the new entry point in the downstream `ha-term-safety` integration), the deterministic simulator (`raft_sim.c`), and the downstream `raft-sim-conformance` spec (Leader Completeness oracle).
**Impact**: Today the commit position advances **term-blind** in the controller (`ha_compute_safe_committed_lsn` + the commit barrier) — it can mark a previous-term entry committed by replica-count alone (G4) — and vote freshness compares LSNs only, never the last-log term (G6). After this feature, the library owns `commit_index` and advances it exactly as `raft.tla` `AdvanceCommitIndex` does (quorum agreement **and** current-term entry), notifying the controller via `on_commit_advanced` to **trigger** the existing synchronous commit barrier (the barrier itself is unchanged and remains the controller's). Vote and PreVote freshness become the `raft.tla` `(lastLogTerm, lastLogIndex)` lexicographic comparison. The change is a pure, synchronous, deterministic addition to the core; all durability/I/O stays in adapters.
### Goals
- Add a library-owned `commit_index` and `raft_advance_commit(...)` implementing `raft.tla` `AdvanceCommitIndex` (line 259) **with** the current-term rule `log[Max(agreeIndexes)].term == currentTerm` (line 270) — the Figure-8 safety rule (G4).
- Add the required callback `on_commit_advanced(ctx, commit_index)` that **triggers** the existing synchronous commit barrier; the library never performs the barrier itself.
- Upgrade vote freshness to the `raft.tla` `HandleRequestVoteRequest` `logOk` `(lastLogTerm, lastLogIndex)` comparison (line 285) on both the binding vote path and the PreVote eligibility path (G6).
- Extend `raft_sim.c` to assert the **Figure-8** refusal and **Leader Completeness** across forced leader changes.
### Non-Goals
- **No log model / AppendEntries / per-peer progress** — owned by `raft-log-replication` and reused verbatim (`log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`, `on_persist_entry`, `on_send_append_entries`, `on_truncate_from`, `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`).
- **No commit barrier / durability mechanism** — the synchronous barrier, fsync/group-commit, persist-before-ACK, reply gate, and snapshot transport stay in the controller (`ha-term-safety` adapter). This spec only triggers the barrier.
- **No config-change markers / upgrade snapshot** — owned by `ha-term-safety`.
- **No conformance / TLC trace-check harness** — owned by `raft-sim-conformance`.
- **No controller wiring** — replacing `ha_compute_safe_committed_lsn`'s term-blind advance with the library's `commit_index` is `ha-term-safety`; this spec defines only the library entry point + callback it will route through.
## Boundary Commitments
### This Spec Owns
- The library `commit_index` field in `raft_state_t` and its restore field in `raft_config_t` (`initial_commit_index`, default 0).
- The new entry point `raft_advance_commit(raft_state_t *s)` implementing `AdvanceCommitIndex` with the current-term rule.
- The new required callback `on_commit_advanced` in `raft_callbacks_t` (fired only on strict advance) + its `raft_init` validation.
- The `(lastLogTerm, lastLogIndex)` freshness upgrade on the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path (`raft_on_pre_vote_request`), including carrying `last_log_term` alongside `last_log_index`.
- Simulator extensions in `raft_sim.c` + tests in `src/lib/testsuite/test_raft.c` asserting Figure-8 refusal and Leader Completeness.
### Out of Boundary
- The log model, AppendEntries, and per-peer `match_index[]` themselves (`raft-log-replication`) — consumed verbatim, never redefined.
- The synchronous commit barrier, fsync/group-commit, persist-before-ACK durability, reply gate, snapshot transport (controller / `ha-term-safety`).
- Config markers in the unified log + the change-boundary snapshot (`ha-term-safety`).
- The conformance / TLC trace-check harness (`raft-sim-conformance`).
- The controller's replacement of `ha_compute_safe_committed_lsn` with `commit_index` and the controller's freshness-floor plumbing of `last_log_term` onto the wire (`ha-term-safety`). This spec specifies only **what the library entry point/callback route**, not the controller mechanics.
### Allowed Dependencies
- The implemented election core (`raft-library`) and the log half (`raft-log-replication`): `raft_state_t`, `raft_callbacks_t`, `raft_init`, the vote/PreVote path, `log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`. All reused; the vote path is extended additively.
- `errno.h`, `stdint.h`, `string.h` only — the core stays free-standing for the simulator.
- The simulator harness (`raft_sim.c`) for verification.
- Must NOT depend on any controller header, WAL type, capnp type, or transport — the core stays pure.
### Revalidation Triggers
Dependent specs (`raft-sim-conformance`, `ha-term-safety`) must re-check integration if any of the following change:
- The signature or semantics of `raft_advance_commit` or the `on_commit_advanced` callback (especially the strict-advance / fire-once rule and "trigger only, never performs the barrier").
- The placement of the current-term rule (`on_get_entry_term(Max(agreeIndexes)) == currentTerm`) or the `agreeIndexes`/quorum definition.
- The freshness signature change — the addition of `last_log_term` to `raft_on_vote_request`, the PreVote eligibility path, and any vote-request payload / `on_vote_eligible` extension.
- The `commit_index` restore convention (`initial_commit_index`, default 0) or its monotonicity guarantee.
## Architecture
### Existing Architecture Analysis
After `raft-log-replication`, the library owns the log as pure, synchronous, deterministic state: per-entry term via `on_get_entry_term`, per-peer `next_index[]` / `match_index[]`, and `leader_commit` carried **opaquely** (never acted on). The election core delegates freshness to `on_vote_eligible`, which today receives only `(from_idx, term)` — the caller checks `config_tag` / applied-LSN floor but has no last-log **term**. This feature follows the same discipline:
- **Pure / synchronous / deterministic core** — `raft_advance_commit` reads entry terms via `on_get_entry_term` and emits a single `on_commit_advanced` notification; it owns no entry bytes and performs no I/O. The freshness upgrade is a predicate computed from values already in the library log model plus the candidate's advertised `(last_log_term, last_log_index)`.
- **Persist-before-act ordering** — unchanged on the vote path: a vote that passes the new `(lastLogTerm, lastLogIndex)` predicate is still persisted via `on_persist` before being granted. `commit_index` advance is not itself persisted by the library — durability of the committed position remains the controller barrier's job, triggered by `on_commit_advanced`.
- **Status returns, never self-terminate** — `raft_advance_commit` returns a status; no new path aborts.
- **Separate actions** — mirroring `raft.tla`, `AdvanceCommitIndex` is a **distinct** action from `HandleAppendEntriesResponse`. `raft_on_append_entries_reply` (owned upstream) only updates `match_index[]`; commit advance is the separate `raft_advance_commit` call. This keeps atomic regions small and lets a single-node leader commit (the `{self}` term in `agreeIndexes`).
### Architecture Pattern & Boundary Map
```mermaid
flowchart TB
subgraph PureCore["src/lib/raft/raft.c — PURE CORE (this spec)"]
ADV["raft_advance_commit()<br/>AdvanceCommitIndex (tla:259)<br/>+ current-term rule (tla:270)"]
VOTE["raft_on_vote_request()<br/>logOk (lastLogTerm,lastLogIndex) (tla:285) G6"]
PVOTE["raft_on_pre_vote_request()<br/>same (lastLogTerm,lastLogIndex) freshness"]
end
subgraph Upstream["raft-log-replication — REUSED VERBATIM (not this spec)"]
MI["match_index[] / log_len"]
GET["on_get_entry_term(index) -> term|0"]
REPLY["raft_on_append_entries_reply()<br/>updates match_index[] only"]
end
subgraph CBs["raft_callbacks_t — NEW seam (this spec)"]
C5["on_commit_advanced(ctx, commit_index)<br/>fired ONLY on strict advance"]
end
subgraph Consumers["ADAPTERS — implement callback (NOT this spec's heavy wiring)"]
CTRL["controller adapter (ha-term-safety):<br/>on_commit_advanced -> EXISTING sync commit barrier<br/>(persist-before-ACK preserved, NOT replaced)"]
SIM["raft_sim.c: drive raft_advance_commit,<br/>assert Figure-8 + Leader Completeness (this spec)"]
end
REPLY -.updates.-> MI
ADV --> MI & GET
ADV --> C5
VOTE --> GET
PVOTE --> GET
C5 --> CTRL
C5 --> SIM
ADV --> SIM
```
**Architecture Integration**:
- **Selected pattern**: Pure state machine + callback ports (Hexagonal/Ports-and-Adapters), already established by the election and log cores. Commit advance is a new pure action; `on_commit_advanced` is a new port adapters implement.
- **Domain boundaries**: the commit-safety **decision** (which index is safe to commit, given quorum + current-term) stays in the library; the durable **act** (the synchronous commit barrier, fsync, ACK) stays in the controller adapter, triggered by the port.
- **Existing patterns preserved**: `raft_config_t` extensibility, persist-before-act on the vote path, NULL-checked required callbacks, status-code returns, `XCE_TESTING`-gated simulator.
- **Steering compliance**: matches `src/lib/CLAUDE.md` (free-standing C, gnu17), `ha-async-io.md` (no blocking in the core; the barrier/async lives in the adapter), `testing.md` (TDD: simulator/test first), `ha-async-io.md` durability policy ("persist-before-ACK + synchronous commit barrier" preserved — the callback triggers it), "controller never self-terminates" (library returns statuses only).
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Library core | C (gnu17), `errno.h`/`stdint.h`/`string.h` only | Pure commit-safety + freshness | Free-standing; no controller/capnp deps |
| Callbacks | `raft_callbacks_t` function pointers | Commit-advance trigger seam | 1 new required callback (`on_commit_advanced`) |
| Verification | `raft_sim.c` (XCE_TESTING) + `test_raft.c` | Figure-8 + Leader Completeness | Reuses the in-memory log backing from `raft-log-replication` |
| Adapter (downstream) | controller commit barrier (`ha-term-safety`) | Durable commit / ACK | Not built here; only triggered |
## File Structure Plan
### Modified Files
- `src/lib/raft/raft.h` — Add `uint64_t commit_index;` to `raft_state_t`; add `uint64_t initial_commit_index;` to `raft_config_t`; add the `on_commit_advanced` callback to `raft_callbacks_t`; declare `raft_advance_commit`; extend the vote-path signatures to carry `last_log_term` (alongside the existing `last_log_index`) per the chosen freshness-plumbing option below; add a Figure-8 / Leader-Completeness driver + assertion helper to the simulator section.
- `src/lib/raft/raft.c` — Implement `raft_advance_commit` (agreeIndexes + current-term rule + monotonic strict-advance firing `on_commit_advanced`); add the `on_commit_advanced` NULL check + `initial_commit_index` restore in `raft_init`; upgrade the freshness predicate in `raft_on_vote_request` and `raft_on_pre_vote_request` to the `(lastLogTerm, lastLogIndex)` comparison.
- `src/lib/raft/raft_sim.c` — Drive `raft_advance_commit` after delivery; the Figure-8 construction; the Leader-Completeness assertion across forced leader changes; the freshness-gate exercise.
- `src/lib/testsuite/test_raft.c` — New tests for `raft_advance_commit` (current-term hold/advance, monotonicity, non-leader), `on_commit_advanced` fire-once/never semantics, the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness, and the simulator invariants (TDD-first).
### Directory Structure (unchanged)
```
src/lib/raft/
├── raft.h # + commit_index, on_commit_advanced, raft_advance_commit, last_log_term on vote path
├── raft.c # + AdvanceCommitIndex(current-term rule), (lastLogTerm,lastLogIndex) freshness
└── raft_sim.c # + commit-advance driver, Figure-8, Leader Completeness assert
src/lib/testsuite/
└── test_raft.c # + commit-safety + freshness + Figure-8/Leader-Completeness tests
```
> Each file keeps one responsibility: `raft.c` = pure protocol, `raft_sim.c` = deterministic harness, `test_raft.c` = assertions. No new file is introduced; the existing module pattern is extended.
## System Flows
### Commit advance: separate action after replication (raft.tla AdvanceCommitIndex)
```mermaid
sequenceDiagram
participant L as Leader (raft.c)
participant LC as Leader adapter (cb)
Note over L: replication already updated match_index[]<br/>via raft_on_append_entries_reply (raft-log-replication)
Note over L: raft_advance_commit() — separate action (tla:259)
L->>L: agreeIndexes = { i in 1..log_len :<br/>{self} ∪ {p: match_index[p]>=i} ∈ Quorum }
alt agreeIndexes empty
Note over L: hold commit_index (else-branch)
else Max(agreeIndexes) from a PREVIOUS term
L->>LC: on_get_entry_term(Max) != currentTerm
Note over L: hold commit_index (Figure-8 refusal, G4)
else Max(agreeIndexes) from CURRENT term
L->>LC: on_get_entry_term(Max) == currentTerm
Note over L: commit_index = Max(agreeIndexes) (strict advance)
L->>LC: on_commit_advanced(ctx, commit_index)
Note over LC: trigger EXISTING sync commit barrier<br/>(persist-before-ACK, NOT replaced)
end
```
### Vote freshness: (lastLogTerm, lastLogIndex) comparison (raft.tla HandleRequestVoteRequest logOk)
```mermaid
sequenceDiagram
participant V as Voter (raft.c)
participant VC as Voter adapter (cb)
Note over V: raft_on_vote_request(..., cand_last_log_term, cand_last_log_index)
Note over V: UpdateTerm / stale-term / one-vote-per-term (unchanged)
V->>VC: on_get_entry_term(log_len) -> local_last_log_term (0 if empty)
Note over V: logOk = cand_last_log_term > local_last_log_term<br/>∨ (== ∧ cand_last_log_index >= log_len)
alt !logOk
Note over V: DENY (even if all other gates would grant) — G6
else logOk AND application gate AND invariants pass
V->>VC: on_persist(term, from_idx) %% persist-before-grant unchanged
Note over V: GRANT
end
```
Flow-level decisions:
- **Commit is a separate action** — `match_index[]` is updated by `raft_on_append_entries_reply` (upstream); the commit index advances only when `raft_advance_commit` is driven, exactly as `raft.tla` separates the two actions (R2.6). This lets a single-node leader commit (`{self}` ∈ `agreeIndexes`).
- **Current-term rule is the Figure-8 gate** — `commit_index` advances to `Max(agreeIndexes)` only when that entry is from `currentTerm`; a previous-term entry replicated to a quorum is **held** until a current-term entry above it commits, then committed indirectly (R1.3/R1.4, closes G4).
- **`on_commit_advanced` fires only on strict advance** — never on a held commit index, never from `raft_on_append_entries_reply` (R2.2/R2.3/R2.6).
- **Freshness denies before any grant** — a stale `(lastLogTerm, lastLogIndex)` denies even when the term/vote invariants and the application gate would grant (R3.4, closes G6); persist-before-grant is unchanged (R3.6).
## Freshness Plumbing — design decision
`raft.tla` `logOk` needs the candidate's `(lastLogTerm, lastLogIndex)` and the voter's own. Two options were considered for carrying the candidate values into the library:
- **Option A — extend the `on_vote_eligible` callback signature** to `on_vote_eligible(ctx, from_idx, term, last_log_term, last_log_index)` and have the library compute `logOk` against `on_get_entry_term(log_len)`/`log_len` before calling the gate. **Rejected as the sole mechanism**: it pushes the protocol-defining `logOk` either into the adapter (violating "library owns protocol decisions") or splits it.
- **Option B (chosen) — the library owns `logOk`.** The candidate's `last_log_term` and `last_log_index` are accepted as explicit arguments on `raft_on_vote_request` (and on the PreVote eligibility path), the library computes `logOk` against its own `on_get_entry_term(log_len)` / `log_len`, and only then consults the unchanged application gate `on_vote_eligible` for XCE-specific policy (primary-deny / heartbeat stickiness / catch-up). This keeps the `raft.tla` `logOk` predicate **inside** the library (R3.1, R5.5) while leaving `on_vote_eligible` for application policy that is genuinely outside Raft.
The candidate advertises its `(last_log_term, last_log_index)` in the `RequestVote` / PreVote wire RPC; the controller adapter decodes them and passes them to the library entry point. The on-wire encoding of `last_log_term` (and the controller's freshness-floor plumbing) is the `ha-term-safety` adapter's job — out of this spec's scope; this spec fixes only the library signatures and the predicate.
> **Note — freshness-decision locus moves from caller/dispatcher into the library (both paths).** Today the freshness decision lives **outside** the library: `raft.h` (lines 246–248) documents that PreVote freshness "is applied by the dispatcher before this call", and the binding vote path delegates the decision to `on_vote_eligible`. Option B **moves that decision locus into the library** for **both** the binding (`raft_on_vote_request`) and the PreVote (`raft_on_pre_vote_request`) paths: `raft.c` now computes the `(lastLogTerm, lastLogIndex)` `logOk` itself rather than relying on the caller/dispatcher. This **supersedes the existing `raft.h` lines 246–248 doc contract**, and updating those `raft.h` doc comments (so the header no longer states the dispatcher owns PreVote freshness) is part of the header-contract task.
## Requirements Traceability
| Requirement | Summary | Design Elements | raft.tla action |
|-------------|---------|-----------------|-----------------|
| 1.1–1.7 | Library `commit_index`, agreeIndexes, current-term rule, monotonic, leader-only | `raft_state_t.commit_index`; `raft_advance_commit()`; `on_get_entry_term`; `match_index[]` (upstream) | `AdvanceCommitIndex` (259), current-term rule (270) |
| 2.1–2.6 | `on_commit_advanced` trigger, fire-once, separate from reply | `raft_callbacks_t.on_commit_advanced`; strict-advance firing; `raft_init` validation | (seam; `AdvanceCommitIndex` distinct from `HandleAppendEntriesResponse`) |
| 3.1–3.6 | `(lastLogTerm,lastLogIndex)` binding-vote freshness | `raft_on_vote_request` logOk; `on_get_entry_term(log_len)`/`log_len`; persist-before-grant | `HandleRequestVoteRequest` logOk (285) |
| 4.1–4.5 | Same freshness on PreVote, side-effect-free | `raft_on_pre_vote_request` logOk; no persist/term/role mutation | `HandleRequestVoteRequest` logOk (285), PreVote extension |
| 5.1–5.4 | Barrier/durability preserved as policy; PreVote/priority unchanged; no abort | `on_commit_advanced` as trigger only; additive-only changes | (policy; not a TLA+ action) |
| 6.1–6.6 | Simulator Figure-8 + Leader Completeness + freshness gate | `raft_sim.c` commit driver, Figure-8 construction, Leader-Completeness assert, freshness exercise | Figure-8 / Leader Completeness invariant |
## Components and Interfaces
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| `commit_index` (`raft_state_t`) | Library core | Library-owned commit position | 1 | log core (P0) | State |
| `raft_advance_commit` | Library core | AdvanceCommitIndex + current-term rule | 1, 2 | `on_get_entry_term`, `match_index[]`, `on_commit_advanced` (P0) | Service |
| `on_commit_advanced` callback | Seam | Commit-advance trigger | 2, 5 | controller barrier / sim (P0) | Service |
| `raft_on_vote_request` freshness | Library core | `(lastLogTerm,lastLogIndex)` logOk | 3, 5 | `on_get_entry_term`, `log_len`, `on_persist`, `on_vote_eligible` (P0) | Service |
| `raft_on_pre_vote_request` freshness | Library core | Same logOk, side-effect-free | 4, 5 | `on_get_entry_term`, `log_len`, `on_vote_eligible` (P0) | Service |
| Simulator | Verification | Figure-8 + Leader Completeness | 6 | all of the above (P0) | State |
### Library Core
#### `commit_index` addition to `raft_state_t`
| Field | Detail |
|-------|--------|
| Intent | Hold the library's view of the safe commit position (TLA+ `commitIndex`) |
| Requirements | 1.1, 1.5 |
**Responsibilities & Constraints**
- `uint64_t commit_index;` — the highest 1-based index (== WAL LSN) the library considers safely committed. Monotonic non-decreasing.
- Restored at init from the adapter (`raft_config_t.initial_commit_index`, default 0; the library never reads storage — consistent with `initial_term` / `initial_log_len`).
- The library never advances `commit_index` except through `raft_advance_commit`; `raft_on_append_entries_reply` (upstream) leaves it untouched (R2.6).
#### `raft_advance_commit` — AdvanceCommitIndex (raft.tla:259) with the current-term rule (line 270)
| Field | Detail |
|-------|--------|
| Intent | Advance `commit_index` to `Max(agreeIndexes)` only when that entry is from `currentTerm` |
| Requirements | 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3 |
##### Service Interface
```c
/*
* raft_advance_commit: leader recomputes its commit index (raft.tla
* AdvanceCommitIndex, 259). A SEPARATE action from append-entries reply
* handling (mirrors raft.tla, so a single-node leader can commit and to
* keep atomic regions small).
*
* Computes agreeIndexes = { index in 1..log_len :
* ({self} ∪ {peer : match_index[peer] >= index}) is a quorum }.
* If agreeIndexes is non-empty AND on_get_entry_term(Max(agreeIndexes))
* == s->term, sets commit_index = Max(agreeIndexes) (Figure-8 current-term
* rule) and, on a STRICT advance, fires on_commit_advanced(ctx, commit_index)
* exactly once.
* Otherwise leaves commit_index unchanged and does NOT fire on_commit_advanced.
*
* The library performs NO durability work here — on_commit_advanced only
* TRIGGERS the controller's existing synchronous commit barrier.
*
* Returns RAFT_OK on success (whether or not the index advanced);
* -EPERM if role != RAFT_LEADER (commit_index left unchanged).
*/
int raft_advance_commit(raft_state_t *s);
```
- Preconditions: `s->role == RAFT_LEADER` (else `-EPERM`, no change — R1.6).
- Postconditions: `commit_index` either unchanged or advanced to `Max(agreeIndexes)` when that entry is current-term; `on_commit_advanced` fired iff `commit_index` strictly increased.
- Invariants: `commit_index` never decreases (R1.5); the library reads entry terms only via `on_get_entry_term` (R1.7); `commit_index ≤ log_len`.
> **Quorum**: `(n_peers / 2) + 1`, identical to the election-core quorum; `match_index[self]` is implicitly `log_len` (self always agrees up to its own log length), realized by seeding the agree-count with self for each candidate index. `agreeIndexes` is computed by scanning candidate indexes from `log_len` downward and taking the first that reaches quorum (equivalent to `Max(agreeIndexes)` without materializing the set).
#### Freshness upgrade — `raft_on_vote_request` (raft.tla:285, closes G6)
| Field | Detail |
|-------|--------|
| Intent | Add the `(lastLogTerm, lastLogIndex)` lexicographic logOk before the application gate |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 |
##### Service Interface
```c
/*
* raft_on_vote_request: process an incoming RequestVote RPC (extended).
* Adds the raft.tla HandleRequestVoteRequest logOk (285): the library
* computes
* logOk = (cand_last_log_term > LastTerm(local))
* || (cand_last_log_term == LastTerm(local)
* && cand_last_log_index >= log_len)
* where LastTerm(local) = (log_len > 0) ? on_get_entry_term(log_len) : 0.
* Step order (unchanged except the new logOk gate):
* 1. stale-term deny / UpdateTerm (unchanged)
* 2. role gate, one-vote-per-term (unchanged)
* 3. logOk gate (NEW, G6): deny if !logOk, before on_vote_eligible
* 4. application gate on_vote_eligible (unchanged)
* 5. on_persist before grant (unchanged)
* Returns RAFT_GRANT / RAFT_DENY; sets *reply_term.
*/
int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term,
uint64_t cand_last_log_term,
uint64_t cand_last_log_index,
uint64_t *reply_term);
```
- Preconditions: none beyond a valid `s`.
- Postconditions: a vote is granted only when the existing invariants, the new `(lastLogTerm, lastLogIndex)` logOk, and the application gate all pass; persisted before grant.
- Invariants: `logOk` is evaluated **before** `on_vote_eligible`; a `!logOk` denies regardless of application policy (R3.4); no weakening of stale-term / one-vote-per-term (R3.5).
> **Signature change is a Revalidation Trigger.** `raft_on_vote_request` gains `cand_last_log_term` and `cand_last_log_index`. The controller and the simulator (the two callers) must pass the candidate's advertised last-log `(term, index)`. The PreVote path mirrors this.
#### Freshness upgrade — `raft_on_pre_vote_request` (side-effect-free)
| Field | Detail |
|-------|--------|
| Intent | Apply the identical `(lastLogTerm, lastLogIndex)` logOk on the straw poll, mutating nothing |
| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5 |
##### Service Interface
```c
/*
* raft_on_pre_vote_request: answer a PreVote probe (extended).
* Applies the SAME (lastLogTerm, lastLogIndex) logOk as the binding vote
* (Requirement 3.1), in addition to the existing stale-term deny and the
* on_vote_eligible gate. Mutates NOTHING: no term/voted_for/role/commit_index
* change, no on_persist (PreVote invariant preserved).
* Returns RAFT_GRANT / RAFT_DENY.
*/
int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term,
uint64_t cand_last_log_term,
uint64_t cand_last_log_index);
```
- Preconditions: none beyond a valid `s`.
- Postconditions: grant only when stale-term, logOk, and the application gate all pass; no persistent state touched.
- Invariants: identical freshness rule to the binding path (R4.1, R4.2); side-effect-free (R4.4).
### Seam — `raft_callbacks_t` addition (verbatim Authoritative shared contract)
Appended to `raft_callbacks_t` and validated **required** in `raft_init`. Reproduced verbatim from the brief so downstream specs depend on the exact shape:
```c
/* REQUIRED: fired AFTER commit_index strictly advances. The controller adapter
* feeds this into the EXISTING synchronous commit barrier / persist-before-ACK
* durability — this callback TRIGGERS commit-eval; it does NOT replace the
* barrier. */
void on_commit_advanced(void *ctx, uint64_t commit_index);
```
**Contract rules (R2.4, R5.1)**: the library decides *whether* the commit index advanced; the callback only triggers the adapter's commit work. It is fired exactly once per strict advance (R2.2), never on a held commit index (R2.3), and never from `raft_on_append_entries_reply` (R2.6). It is NULL-checked and required in `raft_init` (`-EINVAL` if NULL — R2.5), matching how the other required callbacks are validated.
### Controller adapter (seam only — heavy wiring is `ha-term-safety`)
This spec describes the adapter **only at the seam**; the full controller integration is the downstream `ha-term-safety` consumer.
- `on_commit_advanced` → the controller routes the new safe `commit_index` into its **existing** synchronous commit barrier (commit-eval sweep, persist-before-ACK, reply gate) — replacing the term-blind `ha_compute_safe_committed_lsn` advance with the library's current-term-gated value. The barrier mechanism is unchanged; only its **input** (the safe commit position) now comes from the library.
- The candidate's advertised `(last_log_term, last_log_index)` is decoded from the `RequestVote` / PreVote wire RPC and passed to `raft_on_vote_request` / `raft_on_pre_vote_request`; `last_log_term` is read back from the WAL term (`on_get_entry_term`-backed index) on recovery. The on-wire encoding and the freshness-floor plumbing are `ha-term-safety`.
- On recovery, the controller restores `commit_index` (= recovered committed LSN) into `raft_config_t.initial_commit_index`.
## Data Models
### Domain Model
- **Commit position (logical)**: `commit_index : uint64` — the highest 1-based index (== WAL LSN) safely committed, owned by the library, monotonic.
- **Convention**: callers pass `s->commit_index` as the `leader_commit` argument to `raft_send_append_entries` — the leader's committed value rides AppendEntries from `commit_index`, which this spec owns (upstream `raft-log-replication` carries `leader_commit` opaquely; the value it carries is the `commit_index` defined here).
- **Freshness tuple**: `(last_log_term : uint64, last_log_index : uint64)` per candidate, compared lexicographically against the voter's own `(on_get_entry_term(log_len), log_len)`.
### Logical Data Model
- `raft_state_t` gains `commit_index`.
- `raft_config_t` gains `initial_commit_index` (restored committed LSN at boot; 0 on first boot — the library never reads storage, consistent with `initial_term` / `initial_log_len`).
- No new persisted on-disk format: `commit_index` durability is the controller barrier's existing responsibility; the library imposes no new persisted layout. `last_log_term` reuses the WAL `hdr.term` already written by `raft-log-replication`'s adapter.
### Data Contracts & Integration
- **Freshness tuple**: `raft_on_vote_request` and `raft_on_pre_vote_request` accept `(cand_last_log_term, cand_last_log_index)` as explicit arguments. The voter's own values come from `on_get_entry_term(log_len)` (term `0` when `log_len == 0`, consistent with `raft.tla` `LastTerm`) and `log_len`.
- **Commit trigger**: `on_commit_advanced(ctx, commit_index)` carries only the new committed index; no entry bytes, no term — the adapter looks up whatever it needs through its own state.
## Error Handling
### Error Strategy
The library returns status codes only and **never aborts** (preserves "controller never self-terminates"). Callbacks for commit-advance are trigger-only and return void; the freshness path keeps the existing persist-failure handling.
### Error Categories and Responses
- **Caller misuse**: `raft_advance_commit` while not leader → `-EPERM`, `commit_index` unchanged, no `on_commit_advanced`. NULL `on_commit_advanced` at `raft_init` → `-EINVAL`.
- **Held commit (Figure-8)**: when `Max(agreeIndexes)` is a previous-term entry, `commit_index` is held — this is the correct safety behavior, not an error; no callback fires.
- **Empty agreeIndexes**: no quorum agrees on any index → hold, no callback.
- **Stale freshness**: `!logOk` → vote/PreVote DENY (not an error; the standard freshness rejection).
- **Persist failure on grant**: unchanged from the existing vote path — a failed `on_persist` denies the vote without recording it.
### Monitoring
Reuse the optional `on_trace` callback for new events: `"commit_advance"`, `"commit_hold_prevterm"`, `"commit_hold_noquorum"`, `"vote_denied_log_term"` (stale `lastLogTerm`), `"vote_denied_log_index"` (tie term, stale index), `"pre_vote_denied_freshness"`. NULL-checked exactly like existing trace calls.
## Testing Strategy
TDD-mandatory (`testing.md`): tests and simulator drivers are written **before** the `raft.c` implementation (RED → GREEN). All tests are `XCE_TESTING`-gated and live in `src/lib/testsuite/test_raft.c` driving `raft_sim.c`. They reuse the in-memory per-node log backing added by `raft-log-replication`.
### Unit Tests (entry points, against the in-memory sim log)
1. `raft_advance_commit` current-term advance: a leader whose `match_index[]` shows a quorum agreeing through index `k`, with the entry at `k` from `currentTerm`, advances `commit_index` to `k` and fires `on_commit_advanced(k)` once (R1.2, R1.3, R2.2).
2. `raft_advance_commit` Figure-8 hold: a leader with a quorum agreeing through a **previous-term** entry at `k` (no current-term entry above it) holds `commit_index` and fires nothing (R1.4, R2.3 — G4).
3. `raft_advance_commit` monotonicity + non-leader: never decreases `commit_index`; returns `-EPERM` with no change and no callback when not leader (R1.5, R1.6).
4. `on_commit_advanced` separation: `raft_on_append_entries_reply` updating `match_index[]` does **not** by itself advance `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does (R2.6).
5. `raft_init` requires `on_commit_advanced`: `-EINVAL` when NULL; restores `initial_commit_index` (R2.5, R1.1).
6. `(lastLogTerm, lastLogIndex)` vote freshness: a candidate with a stale `last_log_term` is denied even with a greater `last_log_index`; a candidate with a fresher `last_log_term` is granted with a smaller `last_log_index`; a tie on term grants only when `last_log_index >= log_len` (R3.1, R3.4). PreVote applies the identical rule with no persist/term/role mutation (R4.1, R4.4).
### Integration / Simulator Tests (Figure-8, Leader Completeness)
1. **Figure-8 scenario**: construct a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does **not** move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over **both** (indirect commit) (R6.2, R6.3 — G4).
2. **Leader Completeness**: across forced leader changes (timeout / partition-driven), assert that any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader (R6.4) — the `(lastLogTerm, lastLogIndex)` freshness gate (R6.5) is what makes this hold, so the two tests are complementary.
3. **Freshness gate exercise**: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome (stale-term-but-higher-index denied; fresher-term-but-lower-index granted) (R6.5).
4. **Trigger semantics**: assert `on_commit_advanced` fires exactly on strict advance and not while the current-term rule holds the commit index (R6.6).

112
.kiro/specs/raft-commit-safety/requirements.md Normal file
View file

@ -0,0 +1,112 @@
# Requirements Document
## Introduction
`src/lib/raft/raft.{c,h}` is the pure, callback-driven Raft library for XCE. The `raft-library` spec gave it the **election half** of `raft/raft.tla` (Ongaro's canonical Raft TLA⁺), and the upstream `raft-log-replication` spec gave it the **log half**: a library-owned log where every entry carries a `uint64_t` term (1-based index = WAL LSN), `ClientRequest` append (`raft_log_append`), `AppendEntries` emission/handling, per-peer `next_index[]` / `match_index[]`, the `on_get_entry_term` / `on_persist_entry` / `on_send_append_entries` / `on_truncate_from` callbacks, and `UpdateTerm` on every RPC. That upstream spec deliberately carries `leader_commit` **opaquely** through `on_send_append_entries` and `raft_on_append_entries_request` **without acting on it** — it never advances a commit index.
This feature closes the remaining two `raft.tla` **safety** gaps from the gap analysis (`.kiro/specs/ha-term-safety/TLA-gap-analysis.md`):
- **G4 — term-blind commit advance.** `raft.tla` `AdvanceCommitIndex` (line 259) advances `commitIndex` to `Max(agreeIndexes)` **only if `log[Max(agreeIndexes)].term == currentTerm`** — the Figure-8 safety rule: a leader must never mark a previous-term entry committed by replica-count alone; it commits the older entry only indirectly, once an entry from its **own** current term reaches a quorum. XCE's commit barrier + `ha_compute_safe_committed_lsn` advanced `committed` term-blind ("from 1 granted voter") because the WAL term was dead (G1); the own-term `WAL_NOOP` gate is only a partial mitigation. This spec adds `raft_advance_commit(...)` to the library, computing `agreeIndexes` from per-peer `match_index[]` and advancing `commit_index` **only when the entry at the max agreed index is from the current term**, with a new `on_commit_advanced(ctx, commit_index)` callback that **triggers** the existing synchronous commit barrier.
- **G6 — LSN-only vote freshness.** `raft.tla` `HandleRequestVoteRequest` `logOk` (line 285) compares the candidate's `(lastLogTerm, lastLogIndex)` against the voter's **lexicographically**: grant only if the candidate's last log term is strictly greater, or the terms tie and the candidate's last log index is at least the voter's. XCE's `on_vote_eligible` gate uses `config_tag ≥` plus an applied/durable-LSN floor **only** — it never compares `lastLogTerm`. With G6 a node whose log has a higher term but a lower index can be denied (correct) or, worse, a stale-term-but-higher-index candidate can be wrongly granted. This spec upgrades the freshness gate on **both** the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path to carry and compare `last_log_term` alongside the existing `last_log_index`.
The library core stays **pure / synchronous / deterministic** (no I/O, no threads, no allocation in the hot path): `raft_advance_commit` reads entry terms only through the existing `on_get_entry_term` callback and reports the new commit index only through `on_commit_advanced`; the controller adapter performs the durable commit work. The feature is verified end-to-end in the deterministic simulator (`raft_sim.c`) against **Leader Completeness** and the Figure-8 scenario, without a live cluster.
## Boundary Context
- **In scope**:
- `raft_advance_commit(...)`: compute `agreeIndexes` from the library's per-peer `match_index[]` (a quorum agrees, including self), and advance `commit_index` to `Max(agreeIndexes)` **only when `on_get_entry_term(Max(agreeIndexes)) == currentTerm`** (the Figure-8 current-term rule, `raft.tla` `AdvanceCommitIndex` line 259, closes G4). Otherwise hold `commit_index` unchanged.
- A new `commit_index` field in the library state and the new required callback `void on_commit_advanced(void *ctx, uint64_t commit_index)` fired **after** `commit_index` strictly advances, so the controller adapter feeds it into the **existing synchronous commit barrier / persist-before-ACK durability** (the callback triggers commit-evaluation; it does not perform or replace the barrier).
- `(lastLogTerm, lastLogIndex)` vote freshness (`raft.tla` `HandleRequestVoteRequest` `logOk` line 285, closes G6): carry `last_log_term` alongside `last_log_index` into the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path, and grant only on the lexicographic `(lastLogTerm, lastLogIndex)` comparison.
- Simulator coverage in `raft_sim.c`: a **Figure-8** scenario (a previous-term entry replicated to a quorum but not yet committed by the current leader) asserting the library refuses to commit it until a current-term entry is committed; and a **Leader Completeness** assertion (a committed entry is present in every future leader's log) across forced leader changes.
- **Out of scope** (owned elsewhere):
- The library log model + `ClientRequest` / `AppendEntries` / `HandleAppendEntries*` / `UpdateTerm`-on-any-RPC, the per-entry term, and the per-peer `next_index[]` / `match_index[]` themselves → `raft-log-replication` (consumed here verbatim; not redefined).
- The synchronous commit-barrier mechanism, fsync/group-commit, persist-before-ACK durability, the reply gate, and snapshot transport → controller (`ha-term-safety` adapter). This spec only **triggers** the barrier via `on_commit_advanced`; it neither implements nor weakens it.
- Configuration-change markers in the unified log and the upgrade snapshot → `ha-term-safety`.
- The full conformance / TLC trace-check harness → `raft-sim-conformance`.
- The replacement of `ha_compute_safe_committed_lsn`'s term-blind advance in the controller with the library's current-term-gated `commit_index` → that controller wiring is `ha-term-safety`; this spec defines only the library entry point and callback the controller will route through.
- **Adjacent expectations**:
- Builds on `raft-log-replication` (HARD dependency): needs the `(term, index)` library log, `on_get_entry_term`, and per-peer `match_index[]`. All of that contract is reused verbatim and not redefined here.
- Preserves as policy (must NOT be canonicalized away): the synchronous commit barrier + persist-before-ACK durability are **stronger** than canonical Raft's async apply and remain — `on_commit_advanced` triggers the existing barrier, it does not replace it. PreVote and priority leadership are unchanged in mechanism; only the freshness predicate they consult gains `last_log_term`.
- "Controller never self-terminates": the library returns status codes only; it never aborts a process. The controller owns liveness/fencing.
- Feeds `raft-sim-conformance` (Leader Completeness oracle) and `ha-term-safety` (the controller integration that routes the WAL/commit barrier through `raft_advance_commit` + `on_commit_advanced`). Closes bug-851 G4/G6.
## Requirements
### Requirement 1: Library-owned commit index with the current-term advance rule (closes G4)
**Objective:** As the leader of a Raft cluster, I want my commit index to advance only over an index that a quorum holds **and** whose entry is from my current term, so that a previous-term entry is never marked committed by replica-count alone (Figure-8 safety) and Leader Completeness holds.
#### Acceptance Criteria
1. The Raft Library shall maintain, per node, a `commit_index` (the `raft.tla` `commitIndex`) initialized to the value restored from the adapter at boot (default 0), so the library — not term-blind controller code — owns the commit position.
2. When `raft_advance_commit(...)` is called while the node's role is `RAFT_LEADER`, the Raft Library shall compute `agreeIndexes` as the set of indexes `index` in `1..log_len` for which the set `{self} ∪ {peer : match_index[peer] ≥ index}` is a quorum, mirroring `raft.tla` `AdvanceCommitIndex` `agreeIndexes`.
3. When `agreeIndexes` is non-empty and `on_get_entry_term(Max(agreeIndexes)) == currentTerm`, the Raft Library shall set `commit_index` to `Max(agreeIndexes)`, mirroring the `raft.tla` `AdvanceCommitIndex` `newCommitIndex` then-branch (the Figure-8 current-term rule).
4. If `agreeIndexes` is empty, or `on_get_entry_term(Max(agreeIndexes)) != currentTerm`, then the Raft Library shall leave `commit_index` unchanged, mirroring the `raft.tla` `AdvanceCommitIndex` else-branch (refusing to commit a previous-term entry by replica-count alone).
5. The Raft Library shall never decrease `commit_index`; `raft_advance_commit` shall be monotonic, only ever moving `commit_index` forward.
6. If `raft_advance_commit(...)` is called while the node's role is not `RAFT_LEADER`, then the Raft Library shall leave `commit_index` unchanged and report a non-success status, mirroring the `raft.tla` `AdvanceCommitIndex` `state[i] = Leader` precondition.
7. The Raft Library shall obtain the term of the entry at `Max(agreeIndexes)` exclusively through the existing `on_get_entry_term` callback and shall perform no storage I/O itself.
### Requirement 2: on_commit_advanced callback triggering the existing commit barrier
**Objective:** As the XCE controller adapter, I want the library to notify me exactly when the committed position moves forward, so that I can drive the existing synchronous commit barrier / persist-before-ACK durability off the library's safe commit index without the library performing or weakening that durability work.
#### Acceptance Criteria
1. The Raft Library shall define the required callback `void on_commit_advanced(void *ctx, uint64_t commit_index)`, fired with the new `commit_index` after it strictly advances.
2. When `raft_advance_commit(...)` advances `commit_index` to a strictly greater value, the Raft Library shall invoke `on_commit_advanced(ctx, commit_index)` exactly once with the new value.
3. If `raft_advance_commit(...)` does not change `commit_index` (held by the current-term rule, empty `agreeIndexes`, or non-leader), then the Raft Library shall not invoke `on_commit_advanced`.
4. The Raft Library shall treat `on_commit_advanced` as a trigger only: it shall not itself fsync, persist entries, send the client ACK, or otherwise implement the commit barrier — those remain the controller adapter's responsibility (persist-before-ACK durability preserved as policy).
5. The Raft Library shall validate `on_commit_advanced` as a required, non-NULL callback at `raft_init`, returning `-EINVAL` when it is NULL, consistent with how existing required callbacks are validated.
6. The Raft Library shall not advance `commit_index` or fire `on_commit_advanced` as a side effect of `raft_on_append_entries_reply`; updating per-peer `match_index[]` and advancing the commit index remain separate steps (mirroring `raft.tla`, where `AdvanceCommitIndex` is a distinct action from `HandleAppendEntriesResponse`).
### Requirement 3: (lastLogTerm, lastLogIndex) vote freshness on the binding vote path (closes G6)
**Objective:** As any cluster member receiving a `RequestVote`, I want to grant my vote only to a candidate whose log is at least as up-to-date as mine by the `(lastLogTerm, lastLogIndex)` comparison, so that an election can never elect a leader missing a committed entry (Leader Completeness).
#### Acceptance Criteria
1. When `raft_on_vote_request(...)` evaluates freshness, the Raft Library shall compute `logOk = (cand_last_log_term > local_last_log_term) ∨ (cand_last_log_term == local_last_log_term ∧ cand_last_log_index ≥ local_last_log_index)`, mirroring the `raft.tla` `HandleRequestVoteRequest` `logOk` predicate (closes G6).
2. The Raft Library shall accept the candidate's `last_log_term` alongside the existing `last_log_index` on the vote-request path so the `(lastLogTerm, lastLogIndex)` comparison can be made (extending the eligibility signature or the vote-request payload accordingly).
3. The Raft Library shall obtain the voter's own `local_last_log_term` and `local_last_log_index` from its library log model (`on_get_entry_term(log_len)` and `log_len`, with term `0` when the log is empty), consistent with `raft.tla` `LastTerm(log)`.
4. If `logOk` is false, then the Raft Library shall deny the vote, even when all other Raft invariants (correct term, no prior vote this term) and application gates would otherwise grant it.
5. The Raft Library shall apply the `(lastLogTerm, lastLogIndex)` freshness check without weakening the existing election-safety invariants (stale-term deny, one-vote-per-term, persist-before-grant) or the existing application policy gate (primary-deny / heartbeat stickiness / catch-up), which remain in force.
6. The Raft Library shall preserve the existing persist-before-grant ordering: a vote that passes the `(lastLogTerm, lastLogIndex)` freshness check is persisted via `on_persist` before being granted.
### Requirement 4: (lastLogTerm, lastLogIndex) freshness on the PreVote eligibility path (closes G6 for PreVote)
**Objective:** As a node answering a PreVote straw poll, I want to apply the same `(lastLogTerm, lastLogIndex)` freshness rule as the binding vote, so that PreVote never green-lights a candidacy that a real election would reject for log staleness — without mutating any persistent state.
#### Acceptance Criteria
1. When `raft_on_pre_vote_request(...)` evaluates eligibility, the Raft Library shall apply the same `(lastLogTerm, lastLogIndex)` freshness predicate as the binding vote path (Requirement 3.1), using the candidate's advertised `last_log_term` and `last_log_index`.
2. The Raft Library shall carry `last_log_term` alongside `last_log_index` on the PreVote eligibility path consistently with the binding vote path, so the two paths apply an identical freshness rule.
3. If the `(lastLogTerm, lastLogIndex)` freshness predicate is false, then the Raft Library shall deny the PreVote.
4. While answering a PreVote, the Raft Library shall not mutate `currentTerm`, `voted_for`, `voted_for_term`, `role`, or `commit_index`, and shall not invoke `on_persist` — preserving the existing PreVote side-effect-free invariant.
5. The Raft Library shall keep the existing PreVote stale-term deny and application eligibility gate (`on_vote_eligible`) in force; the `(lastLogTerm, lastLogIndex)` freshness is applied in addition to them, not in place of them.
### Requirement 5: Preserve the synchronous commit barrier and durability as policy
**Objective:** As the operator of an XCE HA cluster, I want the commit-safety upgrade to keep XCE's stronger-than-canonical durability guarantees intact, so that acknowledged work remains durable on a majority and the term-safety fix does not regress availability or durability behavior.
#### Acceptance Criteria
1. The Raft Library shall expose the safe commit index only as a trigger (`on_commit_advanced`); the synchronous commit barrier, persist-before-ACK durability, fsync/group-commit, and the reply gate shall remain the controller adapter's responsibility and shall not be moved into or weakened by the library.
2. The Raft Library shall not change the existing PreVote mechanism or priority-leadership behavior; the freshness predicate these paths consult gains `last_log_term`, and the locus of that freshness decision moves from the caller/dispatcher into the library (see Requirement 5.4).
3. The Raft Library shall return status codes only and shall never abort a process on any commit-safety or freshness condition, preserving the "controller never self-terminates" invariant.
4. The Raft Library shall leave all existing election and log-replication entry points and callbacks (from `raft-library` and `raft-log-replication`) behaviorally unchanged, **except** for: (a) the additive freshness arguments on the vote/PreVote paths, (b) the additive `commit_index` / `on_commit_advanced` state and callback, and (c) the **freshness-decision locus move** — the vote/PreVote entry points gain the `last_log_term` / `last_log_index` arguments **and** the library now owns the `logOk` decision that was previously the caller's/dispatcher's (the binding path no longer delegates freshness to `on_vote_eligible`, and the PreVote path no longer relies on the dispatcher applying freshness before the call).
### Requirement 6: Deterministic Figure-8 and Leader Completeness verification in the simulator
**Objective:** As a maintainer, I want the deterministic simulator to drive commit advance and forced leader changes and assert the Figure-8 refusal and Leader Completeness, so that the commit-safety and freshness rules are verifiable without a live cluster.
#### Acceptance Criteria
1. The Raft Simulator shall drive `raft_advance_commit` on the leader after replication so the commit index advances deterministically over the in-memory per-node log backing the existing callbacks.
2. The Raft Simulator shall construct a **Figure-8** scenario — a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry — and shall assert that the library does **not** advance `commit_index` over that previous-term entry until an entry from the current term is committed.
3. When the current-term entry that covers the previous-term entry is replicated to a quorum and `raft_advance_commit` is driven, the Raft Simulator shall assert that `commit_index` then advances over both entries (indirect commit of the previous-term entry).
4. The Raft Simulator shall assert the **Leader Completeness** invariant across forced leader changes: any entry that was committed (present at or below `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader.
5. The Raft Simulator shall exercise the `(lastLogTerm, lastLogIndex)` freshness gate: a candidate with a stale last-log-term shall be denied the vote even when its last-log-index is greater, and a candidate with a fresher last-log-term shall be granted regardless of a smaller last-log-index, asserting the lexicographic comparison.
6. The Raft Simulator shall assert that `on_commit_advanced` fires exactly when `commit_index` strictly advances and does not fire when the current-term rule holds the commit index, so the trigger semantics consumed by the controller barrier are verified.

23
.kiro/specs/raft-commit-safety/spec.json Normal file
View file

@ -0,0 +1,23 @@
{
"feature_name": "raft-commit-safety",
"created_at": "2026-06-21T12:00:00Z",
"updated_at": "2026-06-21T12:00:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true,
"note": "Auto-approved (-y mode). Adds the commit-safety + election-freshness half of raft.tla on top of raft-log-replication's library log. raft_advance_commit (AdvanceCommitIndex, raft.tla:259) advances commit_index to Max(agreeIndexes from match_index[]) ONLY when on_get_entry_term(Max)==currentTerm (Figure-8 current-term rule, G4); new callback on_commit_advanced(ctx, commit_index) TRIGGERS the existing synchronous commit barrier / persist-before-ACK durability (does NOT replace it). Vote/PreVote freshness upgraded from LSN-only to (lastLogTerm,lastLogIndex) lexicographic compare (HandleRequestVoteRequest logOk, raft.tla:285, G6) carried via last_log_term in raft_on_vote_request + on_vote_eligible/vote-request payload. Simulator asserts Figure-8 refusal + Leader Completeness. Boundary: owns commit-index advance + vote freshness in src/lib/raft ONLY; NOT the log model/AppendEntries (raft-log-replication), the barrier/durability mechanism (controller), config markers/snapshot (ha-term-safety), or the conformance harness (raft-sim-conformance). Depends: raft-log-replication (HARD). Closes bug-851 G4/G6."
}

165
.kiro/specs/raft-commit-safety/tasks.md Normal file
View file

@ -0,0 +1,165 @@
# Implementation Plan
> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/lib/testsuite/test_raft.c` (and the simulator backing in `raft_sim.c`) that fails before the `raft.c` change and passes after. Build/run library tests via `/xce-build lib`.
>
> **Spec boundary** (applies to ALL tasks): this spec owns the commit-index advance (`raft.tla` `AdvanceCommitIndex` + the current-term Figure-8 rule, closes G4) and the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness (`raft.tla` `HandleRequestVoteRequest` `logOk`, closes G6) **in `src/lib/raft` only**. It does **NOT** own the log model / AppendEntries / per-peer `match_index[]` (those come from `raft-log-replication` and are reused verbatim), the synchronous commit-barrier / durability mechanism (controller — only TRIGGERED via `on_commit_advanced`), config markers / upgrade snapshot (`ha-term-safety`), or the conformance harness (`raft-sim-conformance`). No task here edits any controller file; the controller adapter is described, not built.
>
> **Depends (HARD): `raft-log-replication`** — needs `log_len`, `match_index[]`, and `on_get_entry_term` already present in `raft.{c,h}` and the in-memory per-node log + AppendEntries routing already in `raft_sim.c`.
## 1. Header contract: commit_index, on_commit_advanced, raft_advance_commit, freshness signatures
- [ ] 1.1 Extend `raft.h` with the commit-index state and restore field
- Add `uint64_t commit_index;` to `raft_state_t`
- Add `uint64_t initial_commit_index;` to `raft_config_t` (restored committed LSN at boot, default 0; library never reads storage — consistent with `initial_term`/`initial_log_len`)
- Observable completion: `raft.h` compiles; `sizeof(raft_state_t)` grows by the new field; no entry-byte storage added
- _Requirements: 1.1, 1.5_
- _Boundary: raft.h_
- [ ] 1.2 Add the `on_commit_advanced` REQUIRED callback to `raft_callbacks_t` (verbatim contract)
- Declare `void on_commit_advanced(void *ctx, uint64_t commit_index)` with the exact signature from design.md / brief
- Document it as REQUIRED, fired only on strict advance, and as a TRIGGER for the existing commit barrier (never performs durability itself)
- Observable completion: the function-pointer member exists in `raft_callbacks_t` with the verbatim signature; header doc comment matches the Authoritative shared contract
- _Requirements: 2.1, 2.4, 5.1_
- _Boundary: raft.h_
- _Depends: 1.1_
- [ ] 1.3 Declare `raft_advance_commit` and extend the vote/PreVote signatures with last_log_term/last_log_index
- Declare `int raft_advance_commit(raft_state_t *s)`
- Extend `raft_on_vote_request` and `raft_on_pre_vote_request` to accept `cand_last_log_term` and `cand_last_log_index` (the chosen "library owns logOk" plumbing — Option B in design.md)
- Update the header doc comments to describe the new logOk gate and the commit-advance action
- Observable completion: header declares `raft_advance_commit` and the extended vote/PreVote signatures; a stub TU including `raft.h` compiles
- _Requirements: 1.2, 2.2, 3.2, 4.2_
- _Boundary: raft.h_
- _Depends: 1.2_
## 2. Init validation and commit-index restore
- [ ] 2.1 RED+GREEN: `raft_init` requires `on_commit_advanced` and restores `initial_commit_index`
- Test: `raft_init` returns `-EINVAL` when `on_commit_advanced` is NULL; returns 0 when present; `s->commit_index == cfg->initial_commit_index` after init
- Implement the NULL check in `raft_init` (mirroring the existing required-callback checks); restore `s->commit_index = cfg->initial_commit_index`
- Observable completion: new `-EINVAL` test case passes; restore assertion passes; existing election + log-replication init tests still green
- _Requirements: 2.5, 1.1_
- _Boundary: raft.c, test_raft.c_
- _Depends: 1.3_
## 3. AdvanceCommitIndex with the current-term Figure-8 rule (raft.tla:259/270, closes G4)
- [ ] 3.1 RED: commit-advance tests (current-term advance, Figure-8 hold, monotonic, non-leader)
- Test (advance): leader with `match_index[]` quorum-agreeing through index `k`, entry at `k` from `currentTerm` → `commit_index` advances to `k`, `on_commit_advanced(k)` fired once
- Test (Figure-8 hold): quorum agrees through a previous-term entry at `k` with no current-term entry above → `commit_index` unchanged, `on_commit_advanced` NOT fired
- Test (empty agreeIndexes): no index reaches quorum → unchanged, no callback
- Test (monotonic): a later call computing a smaller `Max(agreeIndexes)` never lowers `commit_index`
- Test (non-leader): `raft_advance_commit` returns `-EPERM`, no change, no callback
- Observable completion: tests compile and FAIL (no implementation yet)
- _Requirements: 1.2, 1.3, 1.4, 1.5, 1.6, 2.2, 2.3_
- _Boundary: test_raft.c_
- _Depends: 2.1_
- [ ] 3.2 GREEN: implement `raft_advance_commit`
- Leader-only guard (`-EPERM` otherwise); compute `Max(agreeIndexes)` by scanning candidate indexes from `log_len` downward, seeding the agree-count with self, until a quorum `((n_peers/2)+1)` is reached
- Apply the current-term rule: advance to `Max(agreeIndexes)` only when `on_get_entry_term(Max) == s->term`; else hold
- Monotonic: only ever move `commit_index` forward; fire `on_commit_advanced(commit_index)` exactly once on a strict advance; read entry terms only via `on_get_entry_term`
- Observable completion: 3.1 tests pass; the current-term check is unmistakably present before the advance in `raft.c`; `commit_index <= log_len` holds
- _Requirements: 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3_
- _Boundary: raft.c_
- _Depends: 3.1_
- [ ] 3.3 RED+GREEN: commit-advance is a separate action from append-entries reply
- Test: a `raft_on_append_entries_reply` that updates `match_index[]` to a committable quorum does NOT by itself change `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does
- Verify (no implementation change expected in `raft_on_append_entries_reply` — it is owned upstream and must stay untouched); if a regression is found, the fix belongs upstream, not here
- Observable completion: test passes; `raft_on_append_entries_reply` remains free of any commit-advance side effect
- _Requirements: 2.6_
- _Boundary: test_raft.c_
- _Depends: 3.2_
## 4. (lastLogTerm, lastLogIndex) vote freshness (raft.tla:285, closes G6)
- [ ] 4.1 RED: binding-vote freshness tests
- Test: candidate with stale `last_log_term` denied even when `last_log_index` is greater; candidate with fresher `last_log_term` granted with a smaller `last_log_index`; tie on term grants only when `cand_last_log_index >= log_len`; the voter's own last-log term/index come from `on_get_entry_term(log_len)`/`log_len` (term 0 when empty)
- Test: `!logOk` denies even when `on_vote_eligible` would grant (logOk applied BEFORE the application gate); persist-before-grant unchanged on the grant path
- Observable completion: tests compile and FAIL
- _Requirements: 3.1, 3.3, 3.4, 3.5, 3.6_
- _Boundary: test_raft.c_
- _Depends: 1.3_
- [ ] 4.2 GREEN: implement the logOk gate in `raft_on_vote_request`
- Compute `local_last_log_term = (log_len > 0) ? on_get_entry_term(log_len) : 0`; `logOk = (cand_last_log_term > local_last_log_term) || (cand_last_log_term == local_last_log_term && cand_last_log_index >= log_len)`
- Insert the logOk deny BEFORE `on_vote_eligible`, after the existing stale-term/UpdateTerm/one-vote-per-term steps; leave persist-before-grant and all existing invariants in place
- Observable completion: 4.1 tests pass; no regression in existing vote tests; the logOk block is clearly before `on_vote_eligible` in `raft.c`
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
- _Boundary: raft.c_
- _Depends: 4.1_
## 5. (lastLogTerm, lastLogIndex) PreVote freshness (raft.tla:285, side-effect-free)
- [ ] 5.1 RED: PreVote freshness tests (identical rule, no side effects)
- Test: `raft_on_pre_vote_request` applies the SAME `(lastLogTerm, lastLogIndex)` logOk as the binding vote; a stale tuple is denied; the existing stale-term deny and `on_vote_eligible` gate remain in force
- Test: across grant and deny, no mutation of `term`/`voted_for`/`voted_for_term`/`role`/`commit_index` and no `on_persist` call (PreVote invariant)
- Observable completion: tests compile and FAIL
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: test_raft.c_
- _Depends: 1.3_
- [ ] 5.2 GREEN: implement the logOk gate in `raft_on_pre_vote_request`
- Apply the identical logOk predicate (factored to match 4.2) before/with the existing eligibility gate; mutate nothing; no `on_persist`
- Observable completion: 5.1 tests pass; no regression in existing PreVote tests; the predicate matches the binding-vote path exactly
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: raft.c_
- _Depends: 5.1_
## 6. Simulator backing: commit-advance driver + freshness plumbing
- [ ] 6.1 Wire `on_commit_advanced` and a commit-advance driver in `raft_sim.c`
- Add a per-`raft_sim_node_t` `commit_index` mirror updated from `on_commit_advanced`; implement the callback (record the value + count fires for assertions)
- Add a sim helper to drive `raft_advance_commit` on a given node (or all leaders) after delivery
- Pass each node's restored `initial_commit_index` (default 0) in `raft_sim_init`; wire `on_commit_advanced` for every node
- Observable completion: after replication + delivery, driving the helper advances the leader's mirrored `commit_index`; the callback fire-count is observable
- _Requirements: 6.1, 6.6, 2.5_
- _Boundary: raft_sim.c_
- _Depends: 3.2_
- [ ] 6.2 Plumb `(last_log_term, last_log_index)` through the sim vote/PreVote routing
- Update the sim's vote and PreVote delivery to read the candidate's last-log `(term, index)` from its in-memory log (`term at log_len` / `log_len`) and pass them to the extended `raft_on_vote_request` / `raft_on_pre_vote_request`
- Observable completion: existing election sim tests still pass with the extended signatures; a node with a stale last-log term is denied a vote it would previously have won on LSN alone
- _Requirements: 6.5, 3.2, 4.2_
- _Boundary: raft_sim.c_
- _Depends: 4.2, 5.2_
## 7. Simulator invariant tests (Figure-8, Leader Completeness, freshness, trigger semantics)
- [ ] 7.1 Figure-8 commit-refusal test (G4)
- Test: construct a previous-term entry replicated to a quorum but not covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does NOT move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over BOTH (indirect commit)
- Observable completion: test passes end-to-end via the sim; the held phase and the released phase are both asserted
- _Requirements: 6.2, 6.3_
- _Boundary: test_raft.c, raft_sim.c_
- _Depends: 6.1_
- [ ] 7.2 Leader Completeness across forced leader changes
- Test: drive replication to a commit, then force leader changes (timeout / partition); assert any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader
- Observable completion: test passes; no future leader is missing a committed entry across the churn
- _Requirements: 6.4_
- _Boundary: test_raft.c, raft_sim.c_
- _Depends: 6.2, 7.1_
- [ ] 7.3 Freshness-gate lexicographic exercise
- Test: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome — stale-term-but-higher-index denied; fresher-term-but-lower-index granted; tie-term decided by index
- Observable completion: test passes for both vote and PreVote paths
- _Requirements: 6.5_
- _Boundary: test_raft.c, raft_sim.c_
- _Depends: 6.2_
- [ ] 7.4 Trigger-semantics test for `on_commit_advanced`
- Test: `on_commit_advanced` fires exactly once on a strict advance and does NOT fire while the current-term rule holds the commit index (Figure-8 held phase) or when not leader
- Observable completion: test passes; the callback fire-count matches the number of strict advances
- _Requirements: 6.6, 2.2, 2.3_
- _Boundary: test_raft.c, raft_sim.c_
- _Depends: 6.1, 7.1_
## 8. Verification
- [ ] 8.1 Full library suite green under ASAN
- Run `/xce-build lib`; all existing election + log-replication tests plus the new commit-safety and freshness tests pass under ASAN with zero leaks
- Observable completion: test binary exits 0; ASAN clean; no regression in the election-core or log-replication tests
- _Requirements: 1.2, 1.3, 2.2, 3.1, 4.1, 6.2, 6.4_
- _Boundary: src/lib/raft, src/lib/testsuite_
- _Depends: 7.1, 7.2, 7.3, 7.4_