checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00 · 2026-06-21 14:02:15 +00:00 · 20e59d0b42
commit 20e59d0b42
parent 7d70820aac
85 changed files with 10525 additions and 1993 deletions
--- a/.kiro/specs/raft-sim-conformance/brief.md
+++ b/.kiro/specs/raft-sim-conformance/brief.md
@ -0,0 +1,26 @@
+# Brief: raft-sim-conformance
+
+## Problem
+Once the library owns the full Raft protocol (election via `raft-library`, log via `raft-log-replication`, commit safety via `raft-commit-safety`), XCE needs an **independent verification harness** that proves the consensus core conforms to `raft/raft.tla` — without a live cluster, threads, or sockets. Today `raft_sim.c` exists but only exercises the election half; there is no oracle for the four Raft safety invariants. This is the "use the verified spec to help" payoff: a deterministic regression oracle that would have caught **bug-851** before it shipped.
+
+## Approach
+Extend `src/lib/raft/raft_sim.c` into a **conformance harness** that drives N library instances purely in-process through transitions equivalent to the `raft.tla` `Next` actions, and continuously checks the TLA+ safety invariants.
+
+### TLA+ actions to drive (raft.tla `Next`, line 454)
+`Restart`, `Timeout`, `RequestVote`, `BecomeLeader`, `ClientRequest`, `AppendEntries`, `AdvanceCommitIndex`, `Receive` (vote/AE request+response, `UpdateTerm`, drop-stale), `DuplicateMessage`, `DropMessage`. The harness owns a message bag and per-node state mirroring `raft.tla`'s variables.
+
+### Invariants asserted every step (the four Raft safety properties)
+1. **Election Safety** — at most one leader per term.
+2. **Log Matching** — if two logs share an entry at `(index, term)`, all prior entries are identical.
+3. **Leader Completeness** — an entry committed in term T is present in every leader's log for terms > T.
+4. **State Machine Safety** — no two nodes apply different commands at the same index.
+
+## Scope
+- **In**: a deterministic, seedable driver (randomized + targeted scenarios: partitions, restarts, message drop/dup/reorder, Figure-8); per-step invariant checks; failure reproduction (print the action trace that broke an invariant); CI-runnable test target alongside the existing library tests. Optionally: emit a trace consumable by TLC, or compare against TLC-generated traces of `raft.tla`.
+- **Out**: the protocol implementation itself (owned by the three upstream specs); the live controller / `/xce-deploy-ha` integration (that is `ha-term-safety`'s chaotic-failover regression); config markers/snapshot.
+
+## Verification
+The harness IS the verification. Acceptance: it reproduces a bug-851-class divergence on a deliberately-broken build (e.g. term stamping disabled, or `(term,lsn)` truncation removed) and passes on the correct build; it runs M randomized seeds in CI with zero invariant violations.
+
+## Dependencies
+`raft-log-replication`, `raft-commit-safety` (HARD — the harness checks invariants over the log + commit behaviour they add). Complements `ha-term-safety`'s live regression (this is the in-process oracle; that is the end-to-end one).
--- a/.kiro/specs/raft-sim-conformance/design.md
+++ b/.kiro/specs/raft-sim-conformance/design.md
@ -0,0 +1,584 @@
+# Design Document
+
+## Overview
+
+**Purpose**: This feature turns `src/lib/raft/raft_sim.c` (today an election-only RPC
+router) into an **in-process TLA+ conformance harness** for the XCE Raft core. It drives
+N library instances synchronously through transitions equivalent to the `raft.tla`
+`Next` actions (`raft.tla:454`) and asserts the four Raft safety invariants after every
+step. It delivers an independent regression oracle for the consensus core — the "use the
+verified spec to help" payoff — that would have caught **bug-851** (the silent
+committed-prefix divergence) before it shipped.
+
+**Users**: Raft library maintainers (verifying `raft.c`), the `raft-log-replication` and
+`raft-commit-safety` authors (whose entry points this harness exercises), and CI (which
+runs the harness as a first-class library test target).
+
+**Impact**: Replaces ad-hoc election-only simulation with a complete in-process model of
+the `raft.tla` state space: a shared message bag mirroring `messages`, per-node state
+mirroring `serverVars`/`logVars`/`leaderVars`, an action scheduler, four invariant
+checkers, and a failure-trace recorder. The protocol implementation is unchanged — the
+harness drives it only through the published entry points and callbacks from the two
+upstream specs; the harness itself **is** the in-memory adapter (it implements the WAL/
+transport callbacks against in-memory structures).
+
+### Goals
+
+- A deterministic, seedable driver that mirrors the `raft.tla` `messages` bag and the
+  per-node variables this harness tracks, driving the library entry points only.
+- Per-step assertion of all four safety invariants (Election Safety, Log Matching, Leader
+  Completeness, State Machine Safety) computed purely from observable node state.
+- Targeted adversarial scenarios (partition/heal, restart, drop/dup/reorder, Figure-8,
+  bug-851 G5 term-reuse) plus randomized seed sweeps.
+- A replayable failure trace (seed + ordered action list) and a CI-runnable target.
+- A deliberately-broken-build negative test proving the oracle catches a bug-851-class
+  divergence and passes on the correct build.
+- Optionally, a TLC-consumable trace export.
+
+### Non-Goals
+
+- **No protocol semantics.** logOk, conflict truncation, the current-term commit rule,
+  `(lastLogTerm, lastLogIndex)` freshness, and `UpdateTerm` adoption live in `raft.c`
+  (owned by `raft-log-replication` / `raft-commit-safety`). The harness drives and checks
+  them; it never re-implements them.
+- **No live cluster / controller integration.** The `/xce-deploy-ha` chaotic-failover
+  regression, controller WAL/repl/recovery/commit-barrier internals, config markers, and
+  the upgrade snapshot are `ha-term-safety`.
+- **No changes to `raft/raft.tla`** and **no TLC model-checker harness** for the spec
+  (only an *optional export* the maintainer can feed to TLC manually).
+- **No async I/O, threads, or sockets.** The harness is synchronous and deterministic by
+  construction.
+
+## Boundary Commitments
+
+### This Spec Owns
+
+- `src/lib/raft/raft_sim.c` — the harness body: the shared message bag, per-node
+  in-memory log + commit + applied state, the in-memory adapter callbacks, the seeded
+  action scheduler, the four invariant checkers, the trace recorder, the scenario
+  generators, and the deliberately-broken-build fault switch wiring (test-only).
+- The harness public surface in `raft.h`'s `XCE_TESTING` section: new sim types
+  (message-bag entry, per-node log array, action record) and the new `raft_sim_*` driver
+  functions. These extend — never alter — the protocol declarations.
+- The conformance test entry point + scenarios in `src/lib/testsuite/test_raft.c`
+  (`XCE_TESTING`-gated), including the negative (broken-build) test.
+
+### Out of Boundary
+
+- The protocol entry points and callback **semantics** (`raft_log_append`,
+  `raft_send_append_entries`, `raft_on_append_entries_request/reply`,
+  `raft_advance_commit`, the vote path, `on_persist_entry`, `on_get_entry_term`,
+  `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) — owned by
+  `raft-log-replication` / `raft-commit-safety`. This spec consumes them verbatim.
+- Any `raft.c` change to protocol logic. The harness must not need one; if it does, that
+  is an upstream-spec gap, not a harness change.
+- The controller-side adapters, WAL byte layout, repl transport, recovery, commit
+  barrier, config markers, snapshot — `ha-term-safety`.
+- `raft/raft.tla` itself and any TLC tooling.
+
+### Allowed Dependencies
+
+- The `raft-log-replication` shared callback contract **verbatim**: `on_persist_entry`,
+  `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`; the 1-based index =
+  WAL LSN identity; the `term 0` = absent convention; per-peer `next_index[]`/
+  `match_index[]`; entry points `raft_log_append`, `raft_send_append_entries`,
+  `raft_on_append_entries_request`, `raft_on_append_entries_reply`,
+  `raft_append_result_t`.
+- The `raft-commit-safety` additions: `raft_advance_commit` (current-term-gated
+  `AdvanceCommitIndex`), `on_commit_advanced`, and the `(lastLogTerm, lastLogIndex)` vote
+  freshness already enforced inside the vote handlers.
+- The implemented election core (`raft-library`): `raft_init`, `raft_start_election`,
+  `raft_on_vote_request/reply`, `raft_on_election_timeout`, PreVote, `raft_snapshot`.
+- Only `errno.h`, `stdint.h`, `string.h`, `stdio.h`, `unity/unity.h` (test) — the
+  simulator stays free-standing, like the rest of the library, and is `XCE_TESTING`-gated.
+
+### Revalidation Triggers
+
+Re-check this design if any upstream changes:
+
+- The signature/semantics of any of the four log-replication callbacks or
+  `on_commit_advanced`.
+- The signature/return semantics of `raft_log_append`, `raft_send_append_entries`,
+  `raft_on_append_entries_request`, `raft_on_append_entries_reply`, `raft_advance_commit`,
+  or `raft_append_result_t`.
+- The 1-based-index ↔ WAL-LSN identity or the `term 0` = absent convention.
+- The placement of `UpdateTerm` / `DropStaleResponse` inside the handlers (the harness
+  relies on the handlers performing these, not the harness).
+
+## Architecture
+
+### Existing Architecture Analysis
+
+`raft_sim.c` already establishes the harness idiom this design extends:
+
+- A `raft_sim_cluster_t` owning `nodes[RAFT_SIM_MAX_NODES]`, a `pending[]` RPC queue, and
+  cluster-level counters.
+- Each `raft_sim_node_t` carries `raft_state_t state`, a `raft_callbacks_t cbs` wired to
+  sim routing functions, a back-pointer `ctx` to the cluster, an `idx`, a `partition`
+  flag, and a `trace_buf`.
+- Callbacks cast `ctx` back to `raft_sim_node_t*`, reaching the cluster via `node->ctx`.
+- Routing is synchronous: `raft_sim_deliver_pending` drains the queue, calling
+  `raft_on_vote_request` on the destination and routing the reply back.
+
+This design keeps that idiom and adds: a richer message bag (vote **and** AppendEntries,
+requests **and** responses), an in-memory log + commit + applied state per node, the four
+new adapter callbacks, a seeded action scheduler, invariant checkers, and a trace
+recorder. No new file is introduced; the existing module is extended (consistent with how
+`raft-log-replication` extends `raft_sim.c`).
+
+### TLA+ variable → harness state mapping
+
+The harness mirrors exactly the `raft.tla` variables it needs to drive actions and check
+invariants. The protocol-internal portions of `serverVars`/`leaderVars` (`currentTerm`,
+`votedFor`, `state`, `nextIndex`, `matchIndex`) live inside each `raft_state_t` and are
+**read** through `raft_snapshot` and the library's own fields — the harness does not
+duplicate them.
+
+| `raft.tla` variable | Harness home | Read/owned |
+|---|---|---|
+| `messages` (bag of in-flight RPCs) | `cluster.bag[]` (requests + responses) | harness-owned |
+| `currentTerm[i]`, `state[i]`, `votedFor[i]` | `node.state` (`raft_state_t`) | library-owned, read via snapshot/fields |
+| `log[i]` (sequence of `(index, term, value)`) | `node.log[]` in-memory array | harness-owned (the adapter) |
+| `commitIndex[i]` | `node.commit_index` | harness-owned, advanced via `raft_advance_commit` + `on_commit_advanced` |
+| `nextIndex[i][j]`, `matchIndex[i][j]` | `node.state.next_index[]/match_index[]` | library-owned |
+| applied state machine (for SM Safety) | `node.applied[]` (value per committed index) | harness-owned |
+| `allLogs` (history of every log) | `cluster.all_logs` snapshot used by checkers | harness-owned |
+
+### Component / data-flow map
+
+```mermaid
+flowchart TB
+    subgraph SCHED["Action Scheduler (raft_sim.c) — seeded PRNG"]
+        PICK["pick action ~ raft.tla Next (454)"]
+        REC["record action -> trace buffer (R1.4)"]
+    end
+    subgraph NODES["N library instances (raft.c — UNCHANGED)"]
+        ST["raft_state_t (term, role, votedFor, next/match)"]
+        EP["entry points: start_election, on_vote_*,<br/>log_append, send_append_entries,<br/>on_append_entries_*, advance_commit"]
+    end
+    subgraph ADAPT["In-memory adapter callbacks (raft_sim.c = the adapter)"]
+        A1["on_persist_entry -> node.log[]"]
+        A2["on_get_entry_term -> node.log[idx].term | 0"]
+        A3["on_send_append_entries -> bag.push(AE req)"]
+        A4["on_truncate_from -> node.log truncate"]
+        A5["on_commit_advanced -> node.commit_index + apply()"]
+        A6["on_send_vote_request -> bag.push(vote req)"]
+    end
+    subgraph BAG["Message bag (cluster.bag[]) — mirrors raft.tla messages"]
+        B["vote/AE requests + responses;<br/>Drop/Duplicate/Reorder live here"]
+    end
+    subgraph CHK["Invariant checkers (after every step, R3)"]
+        I1["Election Safety"]
+        I2["Log Matching"]
+        I3["Leader Completeness"]
+        I4["State Machine Safety"]
+    end
+    PICK --> REC --> EP
+    EP --> A1 & A2 & A3 & A4 & A5 & A6
+    A3 & A6 --> BAG
+    BAG -->|Receive/Deliver| EP
+    EP --> CHK
+    CHK -->|violation| FAIL["halt + dump seed & trace (R5)"]
+```
+
+**Architecture integration**
+
+- **Pattern**: model-based test harness over a pure state machine. The library is the
+  system-under-test (Ports-and-Adapters: the harness implements all callback ports
+  against in-memory structures). The scheduler is a seeded interleaving engine over the
+  `raft.tla` action set; the checkers are pure observers.
+- **Domain boundary**: protocol decisions stay in `raft.c`; *driving* (which action,
+  which message) and *checking* (the four invariants) stay in the harness.
+- **Preserved patterns**: `ctx`-back-pointer callbacks, synchronous routing,
+  `XCE_TESTING` gating, per-node trace buffers, the `raft_sim_*` naming.
+- **Steering compliance**: `src/lib/CLAUDE.md` (free-standing gnu17, tests in
+  `testsuite/`), `ha-async-io.md` (no blocking — and here, no async at all),
+  `testing.md` (TDD: checkers/scenarios written before they pass), "controller never
+  self-terminates" (the harness `abort()`s a test on violation, never the library).
+
+### Technology Stack
+
+| Layer | Choice | Role | Notes |
+|---|---|---|---|
+| Harness core | C (gnu17), `string.h`/`stdint.h`/`stdio.h` | message bag, scheduler, checkers, trace | `XCE_TESTING`-gated; free-standing |
+| PRNG | small inlined deterministic generator (e.g. SplitMix64/xorshift) seeded by a `uint64_t` | all nondeterministic choices | no libc `rand()` (not reproducible across platforms) |
+| Adapter | in-memory `node.log[]`, `commit_index`, `applied[]` | implements the 6 callbacks | the harness IS the WAL/transport adapter |
+| Test runner | Unity (`unity/unity.h`) in `test_raft.c` | scenario + seed-sweep + negative test | built by `src/lib/meson.build` test target |
+| Fault switch | compile-time macro (e.g. `RAFT_SIM_BREAK_*`) in the test build only | negative test | never set in default CI run |
+
+## File Structure Plan
+
+### Modified Files
+
+- `src/lib/raft/raft.h` (`XCE_TESTING` section only) — extend the sim types: a unified
+  message-bag entry (`raft_sim_msg_t`: kind = vote-req/vote-rsp/ae-req/ae-rsp, src, dst,
+  and a union of payload fields incl. term, prev_log_index/term, leader_commit, entry
+  bytes + entry_term, match_index, granted/success); a per-node in-memory log entry
+  (`raft_sim_log_entry_t { uint64_t term; uint64_t value; }`); an action record
+  (`raft_sim_action_t`: kind + actor + target/message identity); extend
+  `raft_sim_node_t` with `log[]`, `log_len`, `commit_index`, `applied[]`; extend
+  `raft_sim_cluster_t` with the bag, the PRNG state, the trace buffer, and the seed;
+  declare the new `raft_sim_*` driver/scenario/check functions. **No protocol
+  declaration is changed.**
+- `src/lib/raft/raft_sim.c` — the harness implementation (sections below): adapter
+  callbacks, bag operations (push/deliver/drop/duplicate/reorder), the seeded action
+  scheduler (`raft_sim_step` / `raft_sim_run`), the four invariant checkers
+  (`raft_sim_check_*` + `raft_sim_check_all`), the trace recorder + dumper, the targeted
+  scenario drivers, and the fault-switch hooks (compile-time-gated).
+- `src/lib/testsuite/test_raft.c` — the conformance test cases: targeted scenarios,
+  the randomized M-seed sweep, the replay-reproduction test, and the negative
+  (broken-build) test.
+
+### Directory Structure (unchanged)
+
+```
+src/lib/raft/
+├── raft.h          # + sim message bag, in-memory log, action record, driver decls (XCE_TESTING)
+├── raft.c          # UNCHANGED by this spec (protocol owned upstream)
+└── raft_sim.c      # + bag, scheduler, 4 invariant checkers, trace, scenarios, fault hooks
+src/lib/testsuite/
+└── test_raft.c     # + conformance scenarios, M-seed sweep, replay test, negative test
+```
+
+> Single-responsibility kept: `raft.c` = pure protocol (untouched), `raft_sim.c` =
+> deterministic harness + in-memory adapter + checkers, `test_raft.c` = assertions and
+> the CI batch.
+
+## System Flows
+
+### One scheduler step (the `Next` interleaving)
+
+```mermaid
+sequenceDiagram
+    participant S as Scheduler (seeded PRNG)
+    participant L as Library node (raft.c)
+    participant A as Adapter cb (raft_sim.c)
+    participant B as Message bag
+    participant C as Checkers
+    S->>S: action = pick(Next set) from PRNG
+    S->>S: record(action) into trace (R1.4)
+    alt node-local (Timeout / RequestVote / BecomeLeader / ClientRequest)
+        S->>L: call entry point (start_election / log_append / ...)
+        L->>A: on_persist_entry / on_send_* (drives bag/log)
+        A->>B: push request(s)
+    else AppendEntries (leader replicate)
+        S->>L: raft_send_append_entries(peer, leader_commit)
+        L->>A: on_send_append_entries -> bag.push(AE req)
+    else AdvanceCommitIndex
+        S->>L: raft_advance_commit()
+        L->>A: on_commit_advanced -> commit_index += ; apply()
+    else Receive (deliver one bag message)
+        S->>B: pop selected message
+        S->>L: route to handler (on_vote_request / on_append_entries_request / *_reply)
+        L->>A: UpdateTerm/persist/truncate/append + reply push
+    else DropMessage / DuplicateMessage
+        S->>B: discard / re-enqueue (no node mutation) (R2.5)
+    else Restart
+        S->>L: raft_init preserving persisted term/votedFor + restore log_len (R2.1)
+    end
+    S->>C: raft_sim_check_all() (R3)
+    alt any invariant violated
+        C-->>S: fail -> dump seed + trace + invariant (R5); halt
+    else ok
+        C-->>S: continue
+    end
+```
+
+Flow-level decisions:
+
+- **The library performs the protocol; the harness performs the interleaving.**
+  `UpdateTerm` and `DropStaleResponse` happen *inside* the handlers (R2.4); the harness
+  never pre-filters by term — that would mask the very bugs it must catch.
+- **Commit is library-gated.** The harness calls `raft_advance_commit`; the current-term
+  Figure-8 rule is enforced by `raft.c`. The harness only *applies* whatever the library
+  reports committed via `on_commit_advanced`, then checks State Machine Safety.
+- **Faults are bag-only.** Drop/Duplicate/Reorder mutate `cluster.bag[]` exclusively
+  (R2.5), mirroring `raft.tla:443`/`448` (`UNCHANGED` of all non-message vars).
+- **Restart preserves persistence.** The harness keeps `node.log[]`, the recovered
+  `log_len`, and the last-persisted `(term, votedFor)` across a `raft_init`, matching
+  `raft.tla` `Restart` (keeps `currentTerm`, `votedFor`, `log`; resets volatile state).
+
+### Invariant checker construction (observable-state-only)
+
+```mermaid
+flowchart LR
+    subgraph IN["inputs: per-node observable state"]
+        R["roles+terms (raft_snapshot)"]
+        LG["node.log[] (index, term, value)"]
+        CM["node.commit_index"]
+        AP["node.applied[]"]
+    end
+    R --> ES["Election Safety: |{i: role==LEADER, term==t}| <= 1, all t (R3.1)"]
+    LG --> LM["Log Matching: pairwise, shared (idx,term) => identical prefix (R3.2)"]
+    LG --> LC["Leader Completeness: committed entry in every higher-term leader log (R3.3)"]
+    CM --> LC
+    AP --> SM["State Machine Safety: applied[i][k] == applied[j][k] for all k (R3.4)"]
+```
+
+## Requirements Traceability
+
+| Requirement | Summary | Design elements | raft.tla anchor |
+|---|---|---|---|
+| 1.1–1.6 | Deterministic seedable driver | seeded PRNG; `cluster.bag[]`; per-node `log[]`/`commit_index`/`applied[]`; trace buffer; no threads/sockets/clock; deterministic replay | `messages`, `serverVars`, `logVars` |
+| 2.1–2.6 | Drive Next-equivalent actions via library entry points | scheduler step dispatch; `raft_send_append_entries`; `raft_advance_commit`; handler routing (UpdateTerm/DropStale inside lib); bag-only Drop/Dup; entry-points-only rule | `Next` (454), `Restart`/`Timeout`/`RequestVote`/`BecomeLeader`/`ClientRequest`/`AppendEntries`/`AdvanceCommitIndex`/`Receive`/`Duplicate`/`Drop` |
+| 3.1–3.6 | Per-step four-invariant assertion | `raft_sim_check_election_safety/log_matching/leader_completeness/state_machine_safety`; `raft_sim_check_all`; halt-on-fail; observable-state-only | Election Safety / Log Matching / Leader Completeness / State Machine Safety |
+| 4.1–4.5 | Targeted scenarios | `raft_sim_scenario_partition_heal/restart_churn/drop_dup_reorder/figure8/stale_rejoin` | partition / Restart / message faults / Figure-8 / G5 |
+| 5.1–5.4 | Failure-trace capture + replay | trace recorder; `raft_sim_dump_trace` (seed + ordered actions); replay from seed; quiet-on-success | (verification) |
+| 6.1–6.4 | CI-runnable target | `test_raft.c` conformance entry; M-seed batch + scenarios; ASAN/UBSAN clean; bounded steps/seeds | (verification) |
+| 7.1–7.4 | Deliberately-broken-build negative test | compile-time `RAFT_SIM_BREAK_*` fault switch (test build only); both-direction assertion | bug-851 G1/G3 mechanism |
+| 8.1–8.3 | Optional TLC-consumable trace export | stable machine-readable record named per `raft.tla`; off by default | `raft.tla` actions/vars |
+
+## Components and Interfaces
+
+| Component | Layer | Intent | Req coverage | Key dependencies | Contract |
+|---|---|---|---|---|---|
+| Message bag | Harness | Mirror `messages`; carry vote/AE req+rsp | 1, 2 | `raft_sim_msg_t` | State |
+| In-memory adapter callbacks | Harness | Implement the 6 ports against `node.log[]`/commit | 1, 2 | upstream callback contract (P0) | Service |
+| Action scheduler | Harness | Seeded interleaving over the Next set | 1, 2 | PRNG, entry points (P0) | Service |
+| Invariant checkers | Harness | Assert the four safety properties | 3 | observable node state | Service |
+| Trace recorder | Harness | Record + dump replayable action trace | 1, 5 | scheduler | Service |
+| Scenario drivers | Harness | Targeted adversarial cases | 4 | scheduler, bag | Service |
+| Fault switch | Test build | Deliberately-broken build | 7 | compile-time macro | Service |
+| Conformance test | Test suite | CI batch + negative test | 6, 7 | all of the above | Test |
+
+### In-memory adapter callbacks (the harness as adapter)
+
+The harness implements the upstream callbacks against per-node in-memory state, exactly
+as the `raft-log-replication` design anticipates ("bytes live in the adapter (WAL) /
+simulator (in-memory array)"). Signatures are consumed verbatim from upstream; behavior:
+
+| Callback | Harness behavior |
+|---|---|
+| `on_persist_entry(ctx, index, term, entry, len)` | Write/overwrite `node.log[index-1] = {term, value}`; grow `log_len`. Persist-before-act preserved (called before the library reports success). |
+| `on_get_entry_term(ctx, index)` | Return `node.log[index-1].term`, or `0` if `index==0` or `index > log_len` (the `term 0` = absent convention). |
+| `on_send_append_entries(ctx, peer, prev_log_index, prev_log_term, leader_commit, ...)` | Push an AE-request `raft_sim_msg_t` onto `cluster.bag[]` honoring partitions. |
+| `on_truncate_from(ctx, index)` | Set `node.log_len = index - 1` (delete `index..end`). |
+| `on_commit_advanced(ctx, commit_index)` | Set `node.commit_index = commit_index`; apply `node.log[k].value` into `node.applied[k]` for each newly committed `k`. |
+| `on_send_vote_request(ctx, peer, term)` (existing) | Push a vote-request message onto the bag. |
+
+### Message bag (mirrors `raft.tla` `messages`)
+
+```c
+typedef enum {
+    RAFT_SIM_MSG_VOTE_REQ, RAFT_SIM_MSG_VOTE_RSP,
+    RAFT_SIM_MSG_AE_REQ,   RAFT_SIM_MSG_AE_RSP,
+} raft_sim_msg_kind_t;
+
+typedef struct {
+    raft_sim_msg_kind_t kind;
+    int      src, dst;
+    uint64_t term;            /* RPC term (all kinds) */
+    /* AE request */
+    uint64_t prev_log_index, prev_log_term, leader_commit;
+    uint64_t entry_term, entry_value; int entries_len; /* 0=heartbeat, 1=one entry */
+    /* responses */
+    int      granted;         /* vote rsp */
+    int      success;         /* AE rsp */
+    uint64_t match_index;     /* AE rsp */
+} raft_sim_msg_t;
+```
+
+- `raft_sim_bag_push` / `raft_sim_bag_deliver(i)` (route message `i` to the matching
+  handler and push any reply) / `raft_sim_bag_drop(i)` / `raft_sim_bag_duplicate(i)` /
+  reorder (deliver in PRNG-chosen order). Delivery honors `node.partition` (a partitioned
+  endpoint's messages are not delivered, mirroring a dropped network edge).
+- A handler reply is itself pushed onto the bag as a `*_RSP` message, so responses are
+  first-class bag members (drop/dup/reorder apply to them too).
+
+### Action scheduler
+
+```c
+/* One step of the Next interleaving. Picks an enabled action from the PRNG,
+ * records it, applies it through the library entry points / bag, then runs
+ * raft_sim_check_all. Returns 0 on success, <0 (and dumps the trace) on the
+ * first invariant violation. */
+int raft_sim_step(raft_sim_cluster_t *c);
+
+/* Run up to max_steps (or until quiescent). Returns 0 if no invariant was
+ * violated through the whole run; <0 on the first violation. */
+int raft_sim_run(raft_sim_cluster_t *c, int max_steps);
+
+/* Seeded init: builds N nodes wired to the in-memory adapter + bag. */
+int raft_sim_conformance_init(raft_sim_cluster_t *c, int n_nodes,
+                              uint32_t election_timeout_ms, uint64_t seed);
+```
+
+The action set the scheduler chooses from (each maps to a `raft.tla` `Next` disjunct):
+`Restart`, `Timeout`, `RequestVote`(deliver vote-req), `BecomeLeader`(deliver enough
+vote-rsp), `ClientRequest`, `AppendEntries`, `AdvanceCommitIndex`, `Receive`(deliver an
+AE req/rsp incl. `UpdateTerm`/`DropStaleResponse`), `DuplicateMessage`, `DropMessage`.
+Only *enabled* actions are eligible (e.g. `ClientRequest`/`AppendEntries` require a
+leader; `Receive` requires a non-empty bag). Note that `BecomeLeader` is **not** a
+standalone library entry point: it is realized by delivering the quorum-completing
+vote-response to `raft_on_vote_reply`, which returns `RAFT_BECAME_LEADER` when that
+response carries the candidate over quorum.
+
+### Invariant checkers
+
+```c
+int raft_sim_check_election_safety(const raft_sim_cluster_t *c);      /* R3.1 */
+int raft_sim_check_log_matching(const raft_sim_cluster_t *c);         /* R3.2 */
+int raft_sim_check_leader_completeness(const raft_sim_cluster_t *c);  /* R3.3 */
+int raft_sim_check_state_machine_safety(const raft_sim_cluster_t *c); /* R3.4 */
+int raft_sim_check_all(const raft_sim_cluster_t *c);                  /* R3.5: first failure */
+```
+
+Checker semantics (observable-state-only, R3.6):
+
+- **Election Safety** — for each term `t`, count nodes with `role==LEADER && term==t`;
+  fail if any count > 1.
+- **Log Matching** — for every node pair `(i, j)` and every index `k` present in both:
+  if `log_i[k].term == log_j[k].term`, then every entry at indices `1..k` must be equal
+  `(term, value)` in both logs; fail otherwise. Note: the library collapses `raft.tla`'s
+  truncate-only conflict step and the immediately-following no-conflict append into a
+  single AppendEntries handler invocation (truncate **then** append), so the
+  Log-Matching / divergent-suffix checker authors should expect the divergent suffix to
+  be truncated *and* replaced with the leader's entry in one step rather than over two.
+- **Leader Completeness** — track the highest committed entry observed cluster-wide; for
+  every node currently `LEADER` (and across forced leader changes, the highest-term
+  leader to date), every committed `(index, term)` must be present in its log; fail
+  otherwise. (Committedness is taken from `node.commit_index` produced by the
+  library-gated `raft_advance_commit`.)
+- **State Machine Safety** — for every index `k` applied by two nodes, `applied_i[k] ==
+  applied_j[k]`; fail otherwise.
+
+### Trace recorder
+
+`raft_sim_action_t { kind, actor, msg_index, term, index }` recorded per step into
+`cluster.trace[]`; `raft_sim_dump_trace(c, fp)` prints `seed`, `n`, the failing invariant,
+and the ordered actions in a human-readable form (R5.1, R5.2). Replay = re-run
+`raft_sim_conformance_init` with the same `(n, seed)` and the same `max_steps` (R5.3);
+quiet on success unless `RAFT_SIM_VERBOSE` (R5.4).
+
+### Deliberately-broken-build fault switch (test-only)
+
+Compile-time macros gate **harness-side adapter** corruptions that emulate the bug-851
+defects without touching `raft.c`:
+
+- `RAFT_SIM_BREAK_TERM_STAMP` — `on_persist_entry` writes `term = 0` instead of the
+  supplied term (emulates G1: entry term always 0), so `on_get_entry_term` can no longer
+  distinguish `(term=A, idx=k)` from `(term=B, idx=k)`.
+- `RAFT_SIM_BREAK_TRUNCATE` — `on_truncate_from` is a no-op (emulates G3: the divergent
+  suffix is never deleted).
+
+Under either macro the targeted scenarios (Figure-8, stale-rejoin, divergent-suffix) must
+trip Log Matching / Leader Completeness / State Machine Safety; with the macros off, the
+same run is clean. This keeps the negative test entirely inside the harness build
+configuration (R7.3) — the shipped protocol and the default CI run are unaffected.
+
+> **Why adapter-side and not a `raft.c` edit:** the harness owns only `raft_sim.c`/tests.
+> Emulating G1 by having the *adapter* drop the term reproduces the exact observable
+> divergence (same-index/different-term entries become indistinguishable) that the real
+> bug caused, while honoring the spec boundary (no protocol modification). It still proves
+> the *checkers* have teeth: they catch the divergence regardless of where it originates.
+
+### Optional TLC-consumable trace export
+
+`raft_sim_export_trace(c, fp)` (off unless `RAFT_SIM_EXPORT`) emits, per step, the action
+and the resulting tracked variables (`messages` summary, per-node `currentTerm`/`state`/
+`log`/`commitIndex`) named per `raft.tla` (R8.1, R8.2), in a stable line-oriented format
+the maintainer can post-process for TLC cross-checking. Off by default (R8.3).
+
+## Data Models
+
+### Domain Model
+
+- **Bag message** `raft_sim_msg_t` — a single in-flight RPC (vote/AE, request/response),
+  the unit Drop/Duplicate/Reorder operate on.
+- **Log entry (sim)** `raft_sim_log_entry_t { uint64_t term; uint64_t value; }` — index
+  is the 1-based array position (= WAL LSN); `value` is a synthetic monotone client value
+  so State Machine Safety divergence is observable.
+- **Action record** `raft_sim_action_t` — `(kind, actor, msg_index, term, index)`, the
+  replayable trace unit.
+
+### Logical Data Model
+
+- `raft_sim_node_t` gains `log[RAFT_SIM_MAX_LOG]`, `log_len`, `commit_index`,
+  `applied[RAFT_SIM_MAX_LOG]`, and a persisted-`(term, votedFor)` shadow for Restart.
+- `raft_sim_cluster_t` gains `bag[RAFT_SIM_MAX_MSGS]` + `bag_count`, the PRNG state, the
+  `seed`, and `trace[RAFT_SIM_MAX_STEPS]` + `trace_len`.
+- Fixed-size arrays only (no allocation), bounded by `RAFT_SIM_MAX_LOG`/`MSGS`/`STEPS`,
+  keeping the harness allocation-free and ASAN-clean.
+
+### Data Contracts & Integration
+
+- **Entries handle**: one entry per AE (`entries_len ∈ {0,1}`), matching the upstream
+  `raft.tla` "≤1 entry" simplification and the log-replication wire shape — the harness
+  reuses that shape exactly.
+- **`term 0` sentinel**: honored by `on_get_entry_term` for absent indices.
+- **1-based index = LSN**: `node.log[index-1]` is the canonical mapping.
+
+## Error Handling
+
+### Error Strategy
+
+The harness is test code: on an invariant violation it dumps the seed + trace and fails
+the Unity assertion (the test aborts), **never** the library (`raft.c` returns status
+codes only — "controller never self-terminates" is preserved; the harness's halt is a
+test failure, not a library abort).
+
+### Error Categories and Responses
+
+- **Invariant violation** → `raft_sim_check_all` returns the failing invariant; the
+  scheduler dumps `(seed, n, invariant, trace)` and returns `<0`; the test fails (R3.5,
+  R5.1).
+- **Bag overflow / log overflow** (a scenario exceeded a fixed bound) → the harness
+  fails loud (a distinct, non-invariant test error) rather than silently truncating, so
+  bounds bugs are visible.
+- **Misuse** (e.g. `ClientRequest` driven on a non-leader) → the scheduler only selects
+  *enabled* actions, so the library's `-EPERM` path is exercised intentionally (a
+  targeted unit test), never as an accidental scheduler state.
+
+### Monitoring
+
+Reuse the existing per-node `trace_buf` and the library `on_trace` events
+(`"ae_recv"`, `"ae_conflict_truncate"`, `"term_adopt_ae"`, `"ae_reply_advance"`, etc.)
+plus the new harness-level action trace, so a failing run shows both the library's view
+and the scheduler's interleaving.
+
+## Testing Strategy
+
+TDD-mandatory (`testing.md`): every checker and scenario is written as a RED test in
+`test_raft.c` (with the `raft_sim.c` backing) that fails before the harness logic exists
+and passes after. All tests are `XCE_TESTING`-gated and run via `/xce-build lib`.
+
+### Unit tests (checkers in isolation)
+
+1. Each checker flags a hand-constructed violating cluster and passes a valid one:
+   two leaders same term (Election Safety); same `(idx, term)` with different prefix
+   (Log Matching); a committed entry missing from a higher-term leader (Leader
+   Completeness); divergent `applied[k]` (State Machine Safety). (R3.1–3.4)
+2. `raft_sim_check_all` returns the first failing invariant and halts (R3.5).
+
+### Scenario tests (targeted, R4)
+
+1. Partition/heal: isolate a minority during appends, heal, drain; assert all four
+   invariants hold throughout and after convergence (R4.1).
+2. Restart churn: restart nodes mid-replication preserving persisted state; assert
+   re-convergence with no violation (R4.2).
+3. Drop/dup/reorder: perturb bag delivery; assert idempotent convergence + invariants
+   (R4.3).
+4. Figure-8: a previous-term entry on a quorum, leader change; assert it is **not**
+   reported committed until a current-term entry commits (R4.4).
+5. Stale-rejoin (bug-851 G5): a term-lagging node rejoins under a higher-term leader;
+   assert it adopts the higher term from the AE/heartbeat path and does not re-elect at a
+   reused term (R4.5).
+
+### Randomized + CI (R6)
+
+1. M-seed sweep: run `raft_sim_run` for a fixed `max_steps` across M seeds; assert zero
+   violations. Bounded so the CI target fits the library test-suite budget (R6.4).
+2. Runs clean under ASAN/UBSAN (R6.3).
+
+### Negative test (R7)
+
+1. With `RAFT_SIM_BREAK_TERM_STAMP` (or `RAFT_SIM_BREAK_TRUNCATE`) set, the Figure-8 /
+   divergent-suffix scenario **fails** an invariant and the trace is dumped; with the
+   macro off, the same scenario **passes** — both directions asserted in one test, proving
+   the oracle's sensitivity (R7.1, R7.2, R7.4). The macro is confined to the test build
+   (R7.3).
+
+### Replay test (R5)
+
+1. A run that fails for seed `S` reproduces the identical failing action and invariant
+   when re-run with seed `S` and the same `max_steps` (R5.3).
--- a/.kiro/specs/raft-sim-conformance/requirements.md
+++ b/.kiro/specs/raft-sim-conformance/requirements.md
@ -0,0 +1,163 @@
+# Requirements Document
+
+## Introduction
+
+XCE's Raft consensus core lives in `src/lib/raft/raft.{c,h}` as a pure, callback-driven
+state machine. With `raft-log-replication` (the `(term, index)` log, `AppendEntries`,
+`HandleAppendEntriesRequest`, `UpdateTerm`-on-any-RPC) and `raft-commit-safety`
+(`AdvanceCommitIndex` with the Figure-8 current-term rule, `(lastLogTerm, lastLogIndex)`
+vote freshness) the library owns the full election + log + commit halves of
+`raft/raft.tla` (Ongaro canonical). What is still missing is an **independent oracle**:
+nothing continuously proves the implementation actually conforms to the four Raft safety
+invariants. Today `src/lib/raft/raft_sim.c` exercises only the election RPC path; it has
+no log, no commit, and no invariant checkers.
+
+This feature extends `raft_sim.c` into an **in-process conformance/verification
+harness** that drives N library instances purely synchronously (no threads, no sockets,
+no live cluster) through transitions equivalent to the `raft.tla` `Next` actions
+(`raft.tla:454`) and asserts the four TLA+ safety invariants on every step. The harness
+is the "use the verified spec to help" payoff: a deterministic regression oracle for the
+consensus core. Its acceptance bar is concrete — it must **reproduce a bug-851-class
+invariant divergence on a deliberately-broken build** (e.g. term stamping disabled or
+`(term, index)` conflict truncation removed) and **pass on the correct build**, and it
+must run M randomized seeds in CI with zero invariant violations.
+
+## Boundary Context
+
+- **In scope**: a seedable/deterministic action driver mirroring `raft.tla`'s message
+  bag + per-node variables; randomized and targeted scenario generators (network
+  partitions, node restarts, message drop/duplicate/reorder, the Figure-8 commit
+  scenario); per-step assertion of all four safety invariants over the N library
+  instances; failure-trace capture and reproduction (print the exact action sequence
+  that broke an invariant, replayable from the seed); a CI-runnable test target
+  alongside the existing library tests; an acceptance/negative test that a
+  deliberately-broken protocol build is caught and the correct build passes.
+  Optionally, an exported machine-readable trace format consumable by TLC for
+  cross-checking against `raft.tla`.
+- **Out of scope**: the protocol implementation itself — the log model, `AppendEntries`,
+  `AdvanceCommitIndex`, vote-freshness, and term-adoption semantics are owned by
+  `raft-log-replication` and `raft-commit-safety`; this harness only *drives and checks*
+  them. Also out: the live controller / `/xce-deploy-ha` chaotic-failover regression
+  (owned by `ha-term-safety`); the controller WAL/repl/recovery/commit-barrier
+  internals; config-change markers and the upgrade snapshot (`ha-term-safety`); any
+  change to `raft.tla` itself or a TLC model-checking harness for the spec.
+- **Adjacent expectations**: the harness consumes the `raft-log-replication` shared
+  callback contract verbatim (`on_persist_entry`, `on_get_entry_term`,
+  `on_send_append_entries`, `on_truncate_from`; 1-based index = WAL LSN; `term 0` means
+  absent) and the `raft-commit-safety` additions (`on_commit_advanced`,
+  `(lastLogTerm, lastLogIndex)` freshness). It re-defines none of these; if any of those
+  signatures change, this spec re-validates. The simulator backing log + commit state
+  lives in-memory per node (the harness is the adapter), exactly as the upstream design
+  docs anticipate (`raft-log-replication` design: "bytes live in the adapter (WAL) /
+  simulator (in-memory array)").
+
+## Requirements
+
+### Requirement 1: Deterministic seedable action driver
+
+**Objective:** As a Raft library maintainer, I want a fully deterministic, seed-driven
+in-process driver over N library instances, so that any invariant violation is exactly
+reproducible from its seed without threads, sockets, or a live cluster.
+
+#### Acceptance Criteria
+
+1. When the harness is initialized with a node count N (1 ≤ N ≤ `RAFT_SIM_MAX_NODES`) and a 64-bit seed, the Raft Conformance Harness shall construct N library instances wired to in-memory adapter callbacks and a single shared message bag, and shall produce byte-identical execution for the same (N, seed) inputs across runs.
+2. The Raft Conformance Harness shall derive every nondeterministic choice (which action to take, which node, which message to deliver/drop/duplicate, partition toggles, restart targets, client values) from a single seeded pseudo-random source, so that no wall-clock time, address values, or uninitialized memory influence the run.
+3. While a run is in progress, the Raft Conformance Harness shall maintain per-node state mirroring the `raft.tla` variables it owns (in-memory log of `(index, term, value)` entries, `commitIndex`, applied/state-machine view) and a message bag of in-flight RPC requests and responses.
+4. When a step is executed, the Raft Conformance Harness shall record the chosen action (action kind, actor node, target/message identity) into a replayable trace buffer before applying it.
+5. The Raft Conformance Harness shall execute purely synchronously: it shall not spawn threads, open sockets, perform real I/O, or read wall-clock time during a run.
+6. Where the seed and node count are fixed, when the same number of steps is executed twice, the Raft Conformance Harness shall reach an identical final per-node and message-bag state (deterministic replay).
+
+### Requirement 2: Drive transitions equivalent to the raft.tla Next actions
+
+**Objective:** As a Raft library maintainer, I want the harness to drive transitions
+equivalent to every `raft.tla` `Next` action through the actual library entry points, so
+that the implementation — not a re-modeled protocol — is what is being exercised.
+
+#### Acceptance Criteria
+
+1. When the driver selects a node-local action, the Raft Conformance Harness shall be able to apply `Restart` (re-init a node preserving its persisted `term`/`votedFor`/log per `raft.tla` `Restart`), `Timeout`/`RequestVote` (begin an election via the election entry points), `BecomeLeader` (on a quorum of granted votes), and `ClientRequest` (append a value on a leader via `raft_log_append`).
+2. When a leader is selected for replication, the Raft Conformance Harness shall emit `AppendEntries` via the library's `raft_send_append_entries` entry point (computing `prevLogIndex`/`prevLogTerm` from the library's per-peer progress), placing the resulting RPC into the message bag.
+3. When the driver advances commit, the Raft Conformance Harness shall invoke the library's `AdvanceCommitIndex` entry point (`raft-commit-safety`) and apply the resulting committed entries to that node's in-memory state machine.
+4. When a message is delivered from the bag, the Raft Conformance Harness shall route it through the matching library handler (`raft_on_vote_request`/`raft_on_vote_reply`, `raft_on_append_entries_request`/`raft_on_append_entries_reply`), including the `UpdateTerm` higher-term adoption and the stale-response drop (`DropStaleResponse`) that those handlers implement.
+5. When a message is selected for a network fault, the Raft Conformance Harness shall apply `DuplicateMessage` (re-enqueue an in-flight message) or `DropMessage` (discard it) without otherwise mutating node state, mirroring `raft.tla:443`/`raft.tla:448`.
+6. The Raft Conformance Harness shall drive these transitions **only** through the published library entry points and callbacks; it shall not reach into or re-implement the protocol decision logic (logOk, conflict truncation, current-term commit rule, freshness) that lives in `raft.c`.
+
+### Requirement 3: Per-step assertion of the four Raft safety invariants
+
+**Objective:** As a Raft library maintainer, I want all four TLA+ safety invariants
+checked after every applied action, so that the first divergence is caught at the step
+that causes it, not many steps later.
+
+#### Acceptance Criteria
+
+1. After every applied action, the Raft Conformance Harness shall assert **Election Safety**: no two distinct nodes are in the `LEADER` role for the same term.
+2. After every applied action, the Raft Conformance Harness shall assert **Log Matching**: for any two nodes whose logs contain an entry at the same index with the same term, all entries at and before that index are identical (same term and value) across those two logs.
+3. After every applied action, the Raft Conformance Harness shall assert **Leader Completeness**: every entry that has been committed by any node is present, at the same index and term, in the log of every node that is currently or subsequently a leader of a higher term.
+4. After every applied action, the Raft Conformance Harness shall assert **State Machine Safety**: no two nodes have applied different values at the same log index.
+5. If any of the four invariant checks fails, then the Raft Conformance Harness shall halt the run immediately and report the failing invariant.
+6. The Raft Conformance Harness shall compute the invariant checks only from observable per-node state (roles, terms, in-memory logs, commit indices, applied values), so that the checkers are independent of the library's internal protocol logic.
+
+### Requirement 4: Targeted adversarial scenarios
+
+**Objective:** As a Raft library maintainer, I want named targeted scenarios that exercise
+the exact conditions behind bug-851 and the canonical Raft hazards, so that known-hard
+cases are covered deterministically rather than left to chance.
+
+#### Acceptance Criteria
+
+1. The Raft Conformance Harness shall provide a network-partition scenario that isolates a subset of nodes for a number of steps and then heals it, and shall assert all four invariants throughout and after convergence.
+2. The Raft Conformance Harness shall provide a restart scenario that restarts nodes (preserving persisted `term`/`votedFor`/log) during replication, and shall assert that restarted nodes re-converge without invariant violation.
+3. The Raft Conformance Harness shall provide a drop/duplicate/reorder scenario that perturbs the message-bag delivery order and multiplicity, and shall assert idempotent convergence with all four invariants held.
+4. The Raft Conformance Harness shall provide a Figure-8 scenario (a previous-term entry replicated to a quorum but not yet committed under the current leader) and shall assert that no node reports the previous-term entry as committed until a current-term entry is committed (Leader Completeness / State Machine Safety preserved).
+5. When a stale-term node rejoins under a higher-term leader (the bug-851 G5 mechanism), the Raft Conformance Harness shall assert that the rejoining node adopts the higher term from the AppendEntries/heartbeat path and does not re-elect at a reused term (Election Safety preserved).
+
+### Requirement 5: Failure-trace capture and reproduction
+
+**Objective:** As a Raft library maintainer, I want a precise, replayable failure trace
+whenever an invariant breaks, so that I can reproduce and diagnose the exact divergence.
+
+#### Acceptance Criteria
+
+1. If an invariant check fails, then the Raft Conformance Harness shall emit the seed, the node count, the failing invariant name, and the full ordered action trace leading to the failure.
+2. The Raft Conformance Harness shall format the emitted trace so that each action is human-readable (action kind, actor node, message/target identity, relevant terms/indices).
+3. When the harness is re-run with a previously failing seed and node count, the Raft Conformance Harness shall deterministically reproduce the same failing action and invariant (replay reproducibility).
+4. When a run completes without any invariant failure, the Raft Conformance Harness shall report success quietly (no trace dump) unless verbose tracing is explicitly requested.
+
+### Requirement 6: CI-runnable conformance target
+
+**Objective:** As a Raft library maintainer, I want the conformance harness to run as a
+first-class test target in CI, so that consensus regressions are caught automatically
+before they ship.
+
+#### Acceptance Criteria
+
+1. The Raft Conformance Harness shall expose a test entry point in the existing library test suite (`src/lib/testsuite/`) that is built and run under the existing `XCE_TESTING` build (`/xce-build lib`).
+2. When the conformance test runs in CI, the Raft Conformance Harness shall execute a fixed batch of M randomized seeds (M ≥ a configured minimum) plus all targeted scenarios, and shall fail the test run if any seed or scenario violates any invariant.
+3. While running under sanitizers (ASAN/UBSAN as used by the library tests), the Raft Conformance Harness shall complete the CI batch without memory errors, leaks, or undefined behavior.
+4. The Raft Conformance Harness shall bound each run (a maximum step count and a fixed seed batch) so that the CI target completes within the existing library test-suite time budget.
+
+### Requirement 7: Deliberately-broken-build acceptance (negative test)
+
+**Objective:** As a Raft library maintainer, I want proof the oracle actually has teeth,
+so that a passing conformance run is meaningful — the harness must catch a real
+bug-851-class defect when one is present.
+
+#### Acceptance Criteria
+
+1. Where a deliberately-broken protocol build is selected by a build-time fault-injection switch (for example, term stamping disabled so entries carry term 0, or `(term, index)` conflict truncation removed), the Raft Conformance Harness shall, under that switch, reproduce a bug-851-class invariant divergence (a Log Matching, Leader Completeness, or State Machine Safety violation) within the bounded run.
+2. When the fault-injection switch is off (the correct build), the Raft Conformance Harness shall run the same scenarios and seeds with zero invariant violations.
+3. The Raft Conformance Harness shall confine the fault-injection switch to the harness/test build configuration so that it neither alters the shipped protocol implementation nor remains enabled in the default CI conformance run.
+4. The negative test shall assert both directions: the broken configuration fails (an invariant violation is detected and reported) and the correct configuration passes (no violation), so that the oracle's sensitivity is itself verified.
+
+### Requirement 8: Optional TLC-consumable trace export
+
+**Objective:** As a Raft library maintainer, I want the harness to optionally emit a
+machine-readable action/state trace, so that a harness run can be cross-checked against
+TLC's exploration of `raft.tla`.
+
+#### Acceptance Criteria
+
+1. Where trace export is enabled, the Raft Conformance Harness shall emit each step as a machine-readable record (action and the resulting values of the `raft.tla` variables the harness tracks) in a stable, documented format.
+2. Where trace export is enabled, the Raft Conformance Harness shall name actions and variables consistently with `raft/raft.tla` so the export can be mapped to spec actions without re-interpretation.
+3. The Raft Conformance Harness shall keep trace export off by default so that the standard CI conformance run incurs no export overhead.
--- a/.kiro/specs/raft-sim-conformance/spec.json
+++ b/.kiro/specs/raft-sim-conformance/spec.json
@ -0,0 +1,23 @@
+{
+  "feature_name": "raft-sim-conformance",
+  "created_at": "2026-06-21T13:00:00Z",
+  "updated_at": "2026-06-21T13:00:00Z",
+  "language": "en",
+  "phase": "tasks-generated",
+  "approvals": {
+    "requirements": {
+      "generated": true,
+      "approved": true
+    },
+    "design": {
+      "generated": true,
+      "approved": true
+    },
+    "tasks": {
+      "generated": true,
+      "approved": true
+    }
+  },
+  "ready_for_implementation": true,
+  "note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety."
+}
--- a/.kiro/specs/raft-sim-conformance/tasks.md
+++ b/.kiro/specs/raft-sim-conformance/tasks.md
@ -0,0 +1,210 @@
+# Implementation Plan
+
+> **TDD-mandatory** (`testing.md`): every checker, scheduler action, and scenario lands as
+> a RED test in `src/lib/testsuite/test_raft.c` (with the `raft_sim.c` backing) that fails
+> before the harness logic exists and passes after. Build/run library tests via
+> `/xce-build lib`. All harness code is `XCE_TESTING`-gated.
+>
+> **Spec boundary** (applies to ALL tasks): this spec owns `src/lib/raft/raft_sim.c`, the
+> `XCE_TESTING` sim section of `src/lib/raft/raft.h`, and the conformance tests in
+> `src/lib/testsuite/test_raft.c` ONLY. It does **NOT** modify the protocol
+> implementation: no task changes `raft.c` protocol logic or any protocol declaration in
+> `raft.h`, and no task edits a controller file. The harness drives the published library
+> entry points and consumes the `raft-log-replication` + `raft-commit-safety` callback
+> contract **verbatim**.
+>
+> **Depends** (whole spec): `raft-log-replication` (log model + AppendEntries entry points
+> + the four callbacks), `raft-commit-safety` (`raft_advance_commit`, `on_commit_advanced`,
+> `(lastLogTerm, lastLogIndex)` freshness). If those signatures change, re-validate per the
+> design's Revalidation Triggers.
+
+## 1. Harness types and in-memory adapter backing
+
+- [ ] 1.1 Extend the `XCE_TESTING` sim section of `raft.h` with the harness data types
+  - Add `raft_sim_msg_kind_t`, `raft_sim_msg_t` (vote/AE req+rsp union fields), `raft_sim_log_entry_t {term, value}`, `raft_sim_action_t {kind, actor, msg_index, term, index}`, and the `RAFT_SIM_MAX_LOG/MSGS/STEPS` bounds
+  - Extend `raft_sim_node_t` with `log[]`, `log_len`, `commit_index`, `applied[]`, and a persisted-`(term, votedFor)` shadow for Restart; extend `raft_sim_cluster_t` with `bag[]`+`bag_count`, PRNG state, `seed`, and `trace[]`+`trace_len`
+  - Observable completion: `raft.h` compiles under `XCE_TESTING`; no protocol declaration changed; `sizeof(raft_sim_cluster_t)` grows by the new fixed arrays
+  - _Requirements: 1.1, 1.3, 2.5_
+  - _Boundary: raft.h (XCE_TESTING section only)_
+
+- [ ] 1.2 Implement the seeded PRNG and the in-memory adapter callbacks
+  - Add a small inlined deterministic generator (e.g. SplitMix64) seeded by `cluster.seed`; add `on_persist_entry` → `node.log[]`, `on_get_entry_term` → term|0, `on_truncate_from` → shrink `log_len`, `on_commit_advanced` → set `commit_index` + apply into `applied[]`; route `on_send_vote_request`/`on_send_append_entries` to a bag push
+  - Add `raft_sim_conformance_init(c, n, timeout, seed)` wiring all six callbacks + the bag
+  - Observable completion: a test initializes a cluster, drives one `raft_log_append` on a forced leader, and reads the entry back via the in-memory log and `on_get_entry_term`
+  - _Requirements: 1.1, 1.2, 2.1_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.1_
+
+## 2. Message bag (mirrors raft.tla `messages`)
+
+- [ ] 2.1 RED+GREEN: bag push/deliver routes vote and AppendEntries RPCs through the library handlers
+  - Test: pushing a vote-req and an AE-req then delivering them invokes `raft_on_vote_request` / `raft_on_append_entries_request`, and the handler reply is pushed back as a `*_RSP` message
+  - Implement `raft_sim_bag_push` and `raft_sim_bag_deliver(i)` (route by `kind`, honor `node.partition`, push replies)
+  - Observable completion: delivered requests produce matching responses in the bag; partitioned endpoints are not delivered
+  - _Requirements: 2.2, 2.4_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.2_
+
+- [ ] 2.2 RED+GREEN: Drop / Duplicate / Reorder mutate only the bag
+  - Test: `raft_sim_bag_drop`/`_duplicate` change bag multiplicity without mutating any node's term/role/log; reorder delivers in a PRNG-chosen order
+  - Implement the three bag operations (mirroring `raft.tla:443`/`448` `UNCHANGED` of non-message vars)
+  - Observable completion: node snapshots are identical before/after a drop or duplicate of an undelivered message
+  - _Requirements: 2.5_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 2.1_
+
+## 3. The four invariant checkers (TDD against hand-built clusters)
+
+- [ ] 3.1 RED+GREEN: Election Safety checker (P)
+  - Test: a cluster with two `LEADER` nodes at the same term fails; a single-leader-per-term cluster passes
+  - Implement `raft_sim_check_election_safety` over roles+terms read from `raft_snapshot`
+  - Observable completion: both directions assert correctly
+  - _Requirements: 3.1, 3.6_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.1_
+
+- [ ] 3.2 RED+GREEN: Log Matching checker (P)
+  - Test: two logs sharing `(idx, term)` but differing earlier fail; identical-prefix logs pass
+  - Implement `raft_sim_check_log_matching` (pairwise; shared `(idx, term)` ⇒ identical `1..idx`)
+  - Observable completion: both directions assert correctly
+  - _Requirements: 3.2, 3.6_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.1_
+
+- [ ] 3.3 RED+GREEN: Leader Completeness checker (P)
+  - Test: a committed `(index, term)` absent from a higher-term leader's log fails; present passes
+  - Implement `raft_sim_check_leader_completeness` using `commit_index` + per-node logs, tracking the highest-term leader to date
+  - Observable completion: both directions assert correctly
+  - _Requirements: 3.3, 3.6_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.1_
+
+- [ ] 3.4 RED+GREEN: State Machine Safety checker (P)
+  - Test: divergent `applied[k]` across two nodes fails; identical applied prefixes pass
+  - Implement `raft_sim_check_state_machine_safety` over `applied[]`
+  - Observable completion: both directions assert correctly
+  - _Requirements: 3.4, 3.6_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.1_
+
+- [ ] 3.5 RED+GREEN: `raft_sim_check_all` returns the first failing invariant and halts
+  - Test: a cluster violating two invariants reports the first; a clean cluster returns OK
+  - Implement `raft_sim_check_all` composing 3.1–3.4 in a fixed order
+  - Observable completion: first-failure semantics verified
+  - _Requirements: 3.5_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 3.1, 3.2, 3.3, 3.4_
+
+## 4. Action scheduler and trace recorder
+
+- [ ] 4.1 RED+GREEN: trace recorder records every action before it is applied
+  - Test: after a fixed step sequence, `cluster.trace[]` holds the actions in order; `raft_sim_dump_trace` prints seed, n, and the ordered actions
+  - Implement `raft_sim_action_t` recording (R1.4) + `raft_sim_dump_trace` (R5.1, R5.2); quiet on success unless `RAFT_SIM_VERBOSE` (R5.4)
+  - Observable completion: a recorded trace round-trips to a readable dump
+  - _Requirements: 1.4, 5.1, 5.2, 5.4_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 1.1_
+
+- [ ] 4.2 RED+GREEN: `raft_sim_step` drives one Next-equivalent action through the library and checks invariants
+  - Test: stepping reaches each enabled action kind over a run; every step calls `raft_sim_check_all`; a forced violation halts with a trace dump and a `<0` return
+  - Implement the seeded scheduler dispatch (Restart, Timeout, RequestVote/BecomeLeader via vote delivery, ClientRequest, AppendEntries via `raft_send_append_entries`, AdvanceCommitIndex via `raft_advance_commit`, Receive via bag deliver, Duplicate/Drop) selecting only **enabled** actions, driving **only** library entry points
+  - Observable completion: a multi-step run touches log append, AE replication, commit advance, and message faults; invariants checked each step
+  - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.6, 3.5_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 2.2, 3.5, 4.1_
+
+- [ ] 4.3 RED+GREEN: Restart preserves persisted `term`/`votedFor`/log
+  - Test: a Restart action re-inits a node while keeping `node.log[]`, recovered `log_len`, and last-persisted `(term, votedFor)`; volatile state resets per `raft.tla` `Restart`
+  - Implement the Restart branch (feed `initial_log_len` + restored `(term, votedFor)` into `raft_init`)
+  - Observable completion: a restarted node retains its log and term; no invariant violation results from the restart alone
+  - _Requirements: 2.1_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+
+- [ ] 4.4 RED+GREEN: deterministic replay from `(n, seed, max_steps)`
+  - Test: two runs with the same `(n, seed, max_steps)` reach identical final per-node + bag state and an identical recorded trace
+  - Implement `raft_sim_run(c, max_steps)` and confirm no wall-clock/address/uninitialized input leaks into choices
+  - Observable completion: byte-identical replay verified across two runs
+  - _Requirements: 1.5, 1.6, 5.3_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+
+## 5. Targeted adversarial scenarios
+
+- [ ] 5.1 Partition/heal scenario (P)
+  - Implement `raft_sim_scenario_partition_heal`: isolate a minority during appends, heal, drain the bag; assert all four invariants throughout and after convergence
+  - Observable completion: scenario passes with zero violations on the correct build
+  - _Requirements: 4.1_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+
+- [ ] 5.2 Restart-churn scenario (P)
+  - Implement `raft_sim_scenario_restart_churn`: restart nodes mid-replication preserving persisted state; assert re-convergence with no violation
+  - Observable completion: scenario passes; restarted nodes converge to the leader's prefix
+  - _Requirements: 4.2_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.3_
+
+- [ ] 5.3 Drop/duplicate/reorder scenario (P)
+  - Implement `raft_sim_scenario_drop_dup_reorder`: perturb bag delivery order + multiplicity; assert idempotent convergence with all four invariants held
+  - Observable completion: scenario passes; duplicated AE is a no-op (already-have branch)
+  - _Requirements: 4.3_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+
+- [ ] 5.4 Figure-8 scenario
+  - Implement `raft_sim_scenario_figure8`: a previous-term entry on a quorum, a leader change; assert the entry is NOT reported committed until a current-term entry commits (Leader Completeness / State Machine Safety preserved)
+  - Observable completion: the previous-term entry's `commit_index` does not cover it until a current-term commit
+  - _Requirements: 4.4_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+
+- [ ] 5.5 Stale-rejoin scenario (bug-851 G5)
+  - Implement `raft_sim_scenario_stale_rejoin`: a term-lagging node rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does not re-elect at a reused term (Election Safety preserved)
+  - Observable completion: the rejoining node's term advances via AE; no duplicate-term leader appears
+  - _Requirements: 4.5_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+
+## 6. CI batch and randomized seed sweep
+
+- [ ] 6.1 RED+GREEN: randomized M-seed sweep with zero violations
+  - Test: run `raft_sim_run` for a fixed `max_steps` across M seeds (M ≥ configured minimum) plus all targeted scenarios; fail if any seed/scenario violates any invariant
+  - Implement the sweep harness in `test_raft.c`; bound `max_steps`, M, and array sizes so the target fits the library test-suite budget
+  - Observable completion: the full batch runs clean and within budget
+  - _Requirements: 6.1, 6.2, 6.4_
+  - _Boundary: test_raft.c_
+  - _Depends: 5.1, 5.2, 5.3, 5.4, 5.5_
+
+- [ ] 6.2 Verify the conformance batch is ASAN/UBSAN-clean
+  - Run the conformance batch under the library test sanitizer build; fix any leak/overflow/UB in the harness (fixed-size arrays, no allocation)
+  - Observable completion: `/xce-build lib` sanitizer run reports no memory errors, leaks, or UB for the conformance target
+  - _Requirements: 6.3_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 6.1_
+
+## 7. Deliberately-broken-build negative test (the oracle's teeth)
+
+- [ ] 7.1 RED+GREEN: compile-time fault switch emulates the bug-851 defects in the adapter
+  - Implement `RAFT_SIM_BREAK_TERM_STAMP` (adapter `on_persist_entry` writes term 0) and `RAFT_SIM_BREAK_TRUNCATE` (adapter `on_truncate_from` no-op), confined to the test/harness build configuration; default off
+  - Observable completion: with a macro off, scenarios are clean; the macro is not set in the default CI run and does not touch `raft.c`
+  - _Requirements: 7.1, 7.3_
+  - _Boundary: raft_sim.c_
+  - _Depends: 5.4, 5.5_
+
+- [ ] 7.2 RED+GREEN: negative test asserts both directions
+  - Test: under the broken configuration the Figure-8 / divergent-suffix scenario triggers a Log Matching / Leader Completeness / State Machine Safety violation (with trace dump); under the correct configuration the same scenario passes — both asserted in one test
+  - Observable completion: the broken build is caught and the correct build passes, proving checker sensitivity
+  - _Requirements: 7.1, 7.2, 7.4_
+  - _Boundary: test_raft.c_
+  - _Depends: 7.1_
+
+## 8. Optional TLC-consumable trace export
+
+- [ ]* 8.1 TLC-consumable trace export (deferred; off by default)
+  - Implement `raft_sim_export_trace` gated by `RAFT_SIM_EXPORT`: per step, emit the action + tracked `raft.tla` variables (`messages` summary, per-node `currentTerm`/`state`/`log`/`commitIndex`) named per `raft/raft.tla`, in a stable line-oriented format; off by default
+  - Observable completion: with `RAFT_SIM_EXPORT` set, a run emits a stable, spec-named record stream; default runs emit nothing extra
+  - _Requirements: 8.1, 8.2, 8.3_
+  - _Boundary: raft_sim.c, test_raft.c_
+  - _Depends: 4.2_
+```