myxce/.kiro/specs/raft-sim-conformance/requirements.md

Requirements Document

Introduction

XCE's Raft consensus core lives in src/lib/raft/raft.{c,h} as a pure, callback-driven state machine. With raft-log-replication (the (term, index) log, AppendEntries, HandleAppendEntriesRequest, UpdateTerm-on-any-RPC) and raft-commit-safety (AdvanceCommitIndex with the Figure-8 current-term rule, (lastLogTerm, lastLogIndex) vote freshness) the library owns the full election + log + commit halves of raft/raft.tla (Ongaro canonical). What is still missing is an independent oracle: nothing continuously proves the implementation actually conforms to the four Raft safety invariants. Today src/lib/raft/raft_sim.c exercises only the election RPC path; it has no log, no commit, and no invariant checkers.

This feature extends raft_sim.c into an in-process conformance/verification harness that drives N library instances purely synchronously (no threads, no sockets, no live cluster) through transitions equivalent to the raft.tla Next actions (raft.tla:454) and asserts the four TLA+ safety invariants on every step. The harness is the "use the verified spec to help" payoff: a deterministic regression oracle for the consensus core. Its acceptance bar is concrete — it must reproduce a bug-851-class invariant divergence on a deliberately-broken build (e.g. term stamping disabled or (term, index) conflict truncation removed) and pass on the correct build, and it must run M randomized seeds in CI with zero invariant violations.

Boundary Context

• In scope: a seedable/deterministic action driver mirroring raft.tla's message bag + per-node variables; randomized and targeted scenario generators (network partitions, node restarts, message drop/duplicate/reorder, the Figure-8 commit scenario); per-step assertion of all four safety invariants over the N library instances; failure-trace capture and reproduction (print the exact action sequence that broke an invariant, replayable from the seed); a CI-runnable test target alongside the existing library tests; an acceptance/negative test that a deliberately-broken protocol build is caught and the correct build passes. Optionally, an exported machine-readable trace format consumable by TLC for cross-checking against raft.tla.
• Out of scope: the protocol implementation itself — the log model, AppendEntries, AdvanceCommitIndex, vote-freshness, and term-adoption semantics are owned by raft-log-replication and raft-commit-safety; this harness only drives and checks them. Also out: the live controller / /xce-deploy-ha chaotic-failover regression (owned by ha-term-safety); the controller WAL/repl/recovery/commit-barrier internals; config-change markers and the upgrade snapshot (ha-term-safety); any change to raft.tla itself or a TLC model-checking harness for the spec.
• Adjacent expectations: the harness consumes the raft-log-replication shared callback contract verbatim (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from; 1-based index = WAL LSN; term 0 means absent) and the raft-commit-safety additions (on_commit_advanced, (lastLogTerm, lastLogIndex) freshness). It re-defines none of these; if any of those signatures change, this spec re-validates. The simulator backing log + commit state lives in-memory per node (the harness is the adapter), exactly as the upstream design docs anticipate (raft-log-replication design: "bytes live in the adapter (WAL) / simulator (in-memory array)").

Requirements

Requirement 1: Deterministic seedable action driver

Objective: As a Raft library maintainer, I want a fully deterministic, seed-driven in-process driver over N library instances, so that any invariant violation is exactly reproducible from its seed without threads, sockets, or a live cluster.

Acceptance Criteria

1. When the harness is initialized with a node count N (1 ≤ N ≤ RAFT_SIM_MAX_NODES) and a 64-bit seed, the Raft Conformance Harness shall construct N library instances wired to in-memory adapter callbacks and a single shared message bag, and shall produce byte-identical execution for the same (N, seed) inputs across runs.
2. The Raft Conformance Harness shall derive every nondeterministic choice (which action to take, which node, which message to deliver/drop/duplicate, partition toggles, restart targets, client values) from a single seeded pseudo-random source, so that no wall-clock time, address values, or uninitialized memory influence the run.
3. While a run is in progress, the Raft Conformance Harness shall maintain per-node state mirroring the raft.tla variables it owns (in-memory log of (index, term, value) entries, commitIndex, applied/state-machine view) and a message bag of in-flight RPC requests and responses.
4. When a step is executed, the Raft Conformance Harness shall record the chosen action (action kind, actor node, target/message identity) into a replayable trace buffer before applying it.
5. The Raft Conformance Harness shall execute purely synchronously: it shall not spawn threads, open sockets, perform real I/O, or read wall-clock time during a run.
6. Where the seed and node count are fixed, when the same number of steps is executed twice, the Raft Conformance Harness shall reach an identical final per-node and message-bag state (deterministic replay).

Requirement 2: Drive transitions equivalent to the raft.tla Next actions

Objective: As a Raft library maintainer, I want the harness to drive transitions equivalent to every raft.tla Next action through the actual library entry points, so that the implementation — not a re-modeled protocol — is what is being exercised.

Acceptance Criteria

1. When the driver selects a node-local action, the Raft Conformance Harness shall be able to apply Restart (re-init a node preserving its persisted term/votedFor/log per raft.tla Restart), Timeout/RequestVote (begin an election via the election entry points), BecomeLeader (on a quorum of granted votes), and ClientRequest (append a value on a leader via raft_log_append).
2. When a leader is selected for replication, the Raft Conformance Harness shall emit AppendEntries via the library's raft_send_append_entries entry point (computing prevLogIndex/prevLogTerm from the library's per-peer progress), placing the resulting RPC into the message bag.
3. When the driver advances commit, the Raft Conformance Harness shall invoke the library's AdvanceCommitIndex entry point (raft-commit-safety) and apply the resulting committed entries to that node's in-memory state machine.
4. When a message is delivered from the bag, the Raft Conformance Harness shall route it through the matching library handler (raft_on_vote_request/raft_on_vote_reply, raft_on_append_entries_request/raft_on_append_entries_reply), including the UpdateTerm higher-term adoption and the stale-response drop (DropStaleResponse) that those handlers implement.
5. When a message is selected for a network fault, the Raft Conformance Harness shall apply DuplicateMessage (re-enqueue an in-flight message) or DropMessage (discard it) without otherwise mutating node state, mirroring raft.tla:443/raft.tla:448.
6. The Raft Conformance Harness shall drive these transitions only through the published library entry points and callbacks; it shall not reach into or re-implement the protocol decision logic (logOk, conflict truncation, current-term commit rule, freshness) that lives in raft.c.

Requirement 3: Per-step assertion of the four Raft safety invariants

Objective: As a Raft library maintainer, I want all four TLA+ safety invariants checked after every applied action, so that the first divergence is caught at the step that causes it, not many steps later.

Acceptance Criteria

1. After every applied action, the Raft Conformance Harness shall assert Election Safety: no two distinct nodes are in the LEADER role for the same term.
2. After every applied action, the Raft Conformance Harness shall assert Log Matching: for any two nodes whose logs contain an entry at the same index with the same term, all entries at and before that index are identical (same term and value) across those two logs.
3. After every applied action, the Raft Conformance Harness shall assert Leader Completeness: every entry that has been committed by any node is present, at the same index and term, in the log of every node that is currently or subsequently a leader of a higher term.
4. After every applied action, the Raft Conformance Harness shall assert State Machine Safety: no two nodes have applied different values at the same log index.
5. If any of the four invariant checks fails, then the Raft Conformance Harness shall halt the run immediately and report the failing invariant.
6. The Raft Conformance Harness shall compute the invariant checks only from observable per-node state (roles, terms, in-memory logs, commit indices, applied values), so that the checkers are independent of the library's internal protocol logic.

Requirement 4: Targeted adversarial scenarios

Objective: As a Raft library maintainer, I want named targeted scenarios that exercise the exact conditions behind bug-851 and the canonical Raft hazards, so that known-hard cases are covered deterministically rather than left to chance.

Acceptance Criteria

1. The Raft Conformance Harness shall provide a network-partition scenario that isolates a subset of nodes for a number of steps and then heals it, and shall assert all four invariants throughout and after convergence.
2. The Raft Conformance Harness shall provide a restart scenario that restarts nodes (preserving persisted term/votedFor/log) during replication, and shall assert that restarted nodes re-converge without invariant violation.
3. The Raft Conformance Harness shall provide a drop/duplicate/reorder scenario that perturbs the message-bag delivery order and multiplicity, and shall assert idempotent convergence with all four invariants held.
4. The Raft Conformance Harness shall provide a Figure-8 scenario (a previous-term entry replicated to a quorum but not yet committed under the current leader) and shall assert that no node reports the previous-term entry as committed until a current-term entry is committed (Leader Completeness / State Machine Safety preserved).
5. When a stale-term node rejoins under a higher-term leader (the bug-851 G5 mechanism), the Raft Conformance Harness shall assert that the rejoining node adopts the higher term from the AppendEntries/heartbeat path and does not re-elect at a reused term (Election Safety preserved).

Requirement 5: Failure-trace capture and reproduction

Objective: As a Raft library maintainer, I want a precise, replayable failure trace whenever an invariant breaks, so that I can reproduce and diagnose the exact divergence.

Acceptance Criteria

1. If an invariant check fails, then the Raft Conformance Harness shall emit the seed, the node count, the failing invariant name, and the full ordered action trace leading to the failure.
2. The Raft Conformance Harness shall format the emitted trace so that each action is human-readable (action kind, actor node, message/target identity, relevant terms/indices).
3. When the harness is re-run with a previously failing seed and node count, the Raft Conformance Harness shall deterministically reproduce the same failing action and invariant (replay reproducibility).
4. When a run completes without any invariant failure, the Raft Conformance Harness shall report success quietly (no trace dump) unless verbose tracing is explicitly requested.

Requirement 6: CI-runnable conformance target

Objective: As a Raft library maintainer, I want the conformance harness to run as a first-class test target in CI, so that consensus regressions are caught automatically before they ship.

Acceptance Criteria

1. The Raft Conformance Harness shall expose a test entry point in the existing library test suite (src/lib/testsuite/) that is built and run under the existing XCE_TESTING build (/xce-build lib).
2. When the conformance test runs in CI, the Raft Conformance Harness shall execute a fixed batch of M randomized seeds (M ≥ a configured minimum) plus all targeted scenarios, and shall fail the test run if any seed or scenario violates any invariant.
3. While running under sanitizers (ASAN/UBSAN as used by the library tests), the Raft Conformance Harness shall complete the CI batch without memory errors, leaks, or undefined behavior.
4. The Raft Conformance Harness shall bound each run (a maximum step count and a fixed seed batch) so that the CI target completes within the existing library test-suite time budget.

Requirement 7: Deliberately-broken-build acceptance (negative test)

Objective: As a Raft library maintainer, I want proof the oracle actually has teeth, so that a passing conformance run is meaningful — the harness must catch a real bug-851-class defect when one is present.

Acceptance Criteria

1. Where a deliberately-broken protocol build is selected by a build-time fault-injection switch (for example, term stamping disabled so entries carry term 0, or (term, index) conflict truncation removed), the Raft Conformance Harness shall, under that switch, reproduce a bug-851-class invariant divergence (a Log Matching, Leader Completeness, or State Machine Safety violation) within the bounded run.
2. When the fault-injection switch is off (the correct build), the Raft Conformance Harness shall run the same scenarios and seeds with zero invariant violations.
3. The Raft Conformance Harness shall confine the fault-injection switch to the harness/test build configuration so that it neither alters the shipped protocol implementation nor remains enabled in the default CI conformance run.
4. The negative test shall assert both directions: the broken configuration fails (an invariant violation is detected and reported) and the correct configuration passes (no violation), so that the oracle's sensitivity is itself verified.

Requirement 8: Optional TLC-consumable trace export

Objective: As a Raft library maintainer, I want the harness to optionally emit a machine-readable action/state trace, so that a harness run can be cross-checked against TLC's exploration of raft.tla.

Acceptance Criteria

1. Where trace export is enabled, the Raft Conformance Harness shall emit each step as a machine-readable record (action and the resulting values of the raft.tla variables the harness tracks) in a stable, documented format.
2. Where trace export is enabled, the Raft Conformance Harness shall name actions and variables consistently with raft/raft.tla so the export can be mapped to spec actions without re-interpretation.
3. The Raft Conformance Harness shall keep trace export off by default so that the standard CI conformance run incurs no export overhead.