# Gap Analysis — raft-sim-conformance (2026-06-24)

Analyzed per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Scope: a single
feature — an in-process, deterministic, seed-driven harness that drives N `src/lib/raft`
library instances through `raft.tla`-`Next`-equivalent transitions and asserts the four
Raft safety invariants every step (the "use the verified spec as an oracle" payoff;
acceptance bar = catch a bug-851-class divergence on a deliberately-broken build, pass on
the correct build).

## 1. Current State Investigation

### Authoritative spec
- `raft/raft.tla` — Diego Ongaro's canonical Raft (abstract spec; ends `raft.tla:471`).
  All `Next` actions present and located: `Restart` (167), `Timeout` (178),
  `RequestVote` (190), `AppendEntries` (204), `BecomeLeader` (229), `ClientRequest` (246),
  `AdvanceCommitIndex` (259), `Receive` (421) with `UpdateTerm` (406) and
  `DropStaleResponse` (415), `DuplicateMessage` (443), `DropMessage` (448),
  `Next` disjunction (454). The `.tla` file carries NO embedded invariant `THEOREM` block;
  the four safety invariants (Election Safety, Log Matching, Leader Completeness, State
  Machine Safety) are tracked prose-side in `src/controller/CLAUDE.md`. **No TLC config
  (`*.cfg`), no `tla2tools`/TLC tooling exists in-repo** — the spec is a reference
  document, not a model-checked artifact today.

### How the C library is checked against raft.tla today
- **It is not, automatically.** Conformance is enforced by hand: `src/controller/CLAUDE.md`
  holds the TLA+ variable→code and action→code mapping plus a manual "Protocol
  Verification Checklist." There is no trace generation, no invariant checker that runs
  per-step, and no randomized simulation that asserts the four invariants.
- `src/controller/raft/` does **not** exist; the library lives at `src/lib/raft/`
  (`raft.c` 33 KB, `raft.h` 30 KB, `raft_sim.c` 18 KB). Per `src/controller/CLAUDE.md`,
  the controller's `raft.c` was election-only; the full election+log+commit protocol now
  lives in the **library** `src/lib/raft` (deps `raft-log-replication` +
  `raft-commit-safety` landed — callbacks present and exercised in `test_raft.c`).

### Existing harness (`src/lib/raft/raft_sim.c`, `XCE_TESTING`-gated)
Already present (the election + partial log layer it was grown to for the dependency
specs), all consuming published library entry points verbatim:
- `raft_sim_init`, `raft_sim_deliver_pending` (vote RPC routing via
  `raft_on_vote_request`/`_reply`), `raft_sim_deliver_append_entries` (AE routing via
  `raft_on_append_entries_request`/`_reply`), `raft_sim_start_all_elections`,
  `raft_sim_fire_timeout`, `raft_sim_partition`, `raft_sim_advance_commit`
  (`raft_advance_commit`), `raft_sim_restart_node` (models Restart, preserves persisted
  term/votedFor/log/commit_index), and **one** invariant checker:
  `raft_sim_check_log_matching` (`raft_sim.c:424`).
- In-memory adapter callbacks already back the four log callbacks + `on_commit_advanced`
  into per-node `raft_sim_node_t` (`log[]`, `log_count`, `commit_index`,
  `commit_advanced_count`, `trace_buf`). The library holds no entry bytes — the simulator
  owns storage, exactly as the dependency design anticipated.
- `raft.h` `XCE_TESTING` section (`raft.h:432+`) already defines `raft_sim_rpc_t`,
  `raft_sim_entry_t`, `raft_sim_ae_t`, `raft_sim_node_t`, `raft_sim_cluster_t`, and bounds
  `RAFT_SIM_MAX_NODES=8`, `RAFT_SIM_MAX_LOG=64`, `RAFT_SIM_ENTRY_LEN=16`,
  `RAFT_SIM_MAX_AE=64`.
- `test_raft.c` (7011 lines) already exercises election + AE + commit + a
  hand-built partition/catch-up convergence path and both directions of
  `raft_sim_check_log_matching` (pass-on-converged / fail-on-divergence).

### All library entry points the harness needs already exist in `raft.h`
`raft_log_append`, `raft_send_append_entries`, `raft_advance_commit`,
`raft_on_append_entries_request`/`_reply`, `raft_on_vote_request`/`_reply`, `raft_init`,
`raft_start_election`, `raft_on_election_timeout`, `raft_snapshot` — all declared.
**Constraint:** `raft_snapshot_t` (`raft.h:69-76`) exposes term/role/voted_for/votes only —
NOT log or commit_index. So Election Safety can read from `raft_snapshot`, but Log
Matching / Leader Completeness / State Machine Safety MUST read the harness-owned
in-memory `log[]`/`commit_index`/`applied[]` (Req 3.6 "observable per-node state only" is
satisfied by the in-memory mirror, not by reaching into `raft.c`).

### Spec phase state
`spec.json` = `tasks-generated`, all 3 gates auto-approved (`-y`), `ready_for_implementation`.
`tasks.md` has 8 task groups (1.1–8.1); **every task box is unchecked `[ ]`** — none of
the conformance machinery is implemented. The brief/requirements/design are complete and
internally consistent.

## 2. Requirement-to-Asset Map (gaps tagged Missing / Unknown / Constraint)

| Req | Need | Existing asset | Gap |
|-----|------|----------------|-----|
| 1.1–1.6 Deterministic seeded driver | Seeded PRNG (SplitMix64), `(N,seed)` byte-identical replay, message bag, action trace buffer, no threads/sockets/wall-clock | `raft_sim_init` (no seed/PRNG); synchronous already | **Missing**: PRNG, `seed`, `raft_sim_conformance_init`, bag (`bag[]`/`bag_count`), trace buffer, `raft_sim_run` replay. Constraint: must extend the existing `raft_sim_cluster_t`/`_node_t`, not rewrite. |
| 2.1–2.6 Drive `raft.tla` Next actions via library entry points | `Restart`/`Timeout`/`RequestVote`/`BecomeLeader`/`ClientRequest`/`AppendEntries`/`AdvanceCommitIndex`/`Receive`(UpdateTerm+DropStaleResponse)/`Duplicate`/`Drop` through published API | Vote+AE delivery, restart, advance-commit, partition all wired verbatim to library handlers | **Missing**: unified `raft_sim_step` scheduler selecting only *enabled* actions; `ClientRequest` via `raft_log_append`; bag-based `Duplicate`/`Drop` (current delivery drains a queue, no per-message drop/dup/reorder). UpdateTerm/DropStaleResponse are already inside the library handlers (consumed, not re-modeled). |
| 3.1–3.6 Per-step assertion of all four invariants | Election Safety, Log Matching, Leader Completeness, State Machine Safety; first-failure halt; observable-state-only | Only `raft_sim_check_log_matching` exists | **Missing**: `raft_sim_check_election_safety`, `_leader_completeness`, `_state_machine_safety`, `_check_all` (first-failure composer); `applied[]` per-node state machine view. |
| 4.1–4.5 Targeted adversarial scenarios | partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin (bug-851 G5) | A hand-built partition/catch-up test exists in `test_raft.c` (not a reusable named scenario) | **Missing**: 5 named `raft_sim_scenario_*` entry points asserting all four invariants throughout. |
| 5.1–5.4 Failure-trace capture/replay | seed+N+failing-invariant+ordered action trace dump; quiet-on-success; replay reproducibility | per-node `trace_buf` (string events only, not structured actions) | **Missing**: structured `raft_sim_action_t` trace + `raft_sim_dump_trace`; replay tie-in to `raft_sim_run`. |
| 6.1–6.4 CI-runnable target | M-seed sweep + all scenarios in `src/lib/testsuite`, ASAN/UBSAN-clean, bounded runtime | `test_raft.c` already builds/runs under `XCE_TESTING` + sanitizers | **Missing**: the sweep harness + bounds; mechanics (test entry point, sanitizer build) already exist. Low-risk. |
| 7.1–7.4 Deliberately-broken-build negative test | compile-time fault switches (`RAFT_SIM_BREAK_TERM_STAMP`, `RAFT_SIM_BREAK_TRUNCATE`) confined to adapter; both-directions assert | none | **Missing**. **Constraint**: switches must live in the *adapter* (`raft_sim.c` callbacks), never `raft.c` — the boundary forbids touching protocol code. This is the acceptance teeth. |
| 8.1–8.3 Optional TLC-consumable trace export | spec-named machine-readable record stream, off by default | none | **Missing** (optional `[ ]*`, deferred). **Unknown/Research**: no TLC tooling in-repo to actually consume it — export format can be written but cross-checking against `raft.tla` via TLC is unproven. |

### Cross-cutting findings
- **Currently CHECKED against raft.tla (in code):** Log Matching only (one checker,
  `raft_sim.c:424`), and only on hand-driven clusters — not per-step, not over randomized
  runs. Election Safety / Leader Completeness / State Machine Safety are **UNCHECKED in
  code** (asserted only by prose checklist). Of the `Next` actions, Restart / Timeout /
  RequestVote / AppendEntries / AdvanceCommitIndex / Receive are *driven* by the existing
  sim; `ClientRequest`, per-message `Duplicate`/`Drop`/reorder, and a unified
  enabled-action scheduler are **not yet driven**.
- **raft.tla does NOT model durability** (restores `log` intact on Restart,
  `raft.tla:165-174`) — so the harness, like the spec, cannot and should not assert the
  commit-durability invariant (that rests on `ha-commit-durability` tests). This correctly
  scopes the harness to the four safety invariants only. Confirmed against memory.

## 3. Implementation Approach Options

### Option A — Extend `raft_sim.c` / `test_raft.c` in place (spec's chosen path)
Add PRNG, bag, action trace, three new checkers + composer, `raft_sim_step`/`_run`, five
scenarios, fault switches — all inside the existing `XCE_TESTING` sim section and test
file. Reuse `raft_sim_node_t`/`_cluster_t` (extend with `bag[]`, `applied[]`, `seed`,
`trace[]`).
- ✅ Matches the spec boundary exactly (owns `raft_sim.c` + `raft.h` XCE_TESTING + `test_raft.c`); reuses the verbatim-library-entry-point pattern already proven by the dependency specs; no protocol change; sanitizer/CI plumbing already in place.
- ❌ `raft_sim.c` grows well past the existing 18 KB (toward the 800-line guideline ceiling); `test_raft.c` is already 7011 lines.

### Option B — New `raft_conformance.c` / `test_raft_conformance.c` modules
Put the harness (PRNG, bag, scheduler, checkers, scenarios, fault switches) in dedicated
files; keep `raft_sim.c` as the thin election/AE adapter.
- ✅ Honors "many small files"; isolates the oracle from the dependency-era sim; easier to read/maintain; keeps `test_raft.c` from ballooning further.
- ❌ Deviates from the spec's stated file boundary ("owns `raft_sim.c` … ONLY"); needs a meson build wiring change and a boundary-note update; minor duplication of the in-memory adapter.

### Option C — Hybrid (recommended)
Keep the in-memory adapter callbacks, bag, PRNG, and the **four checkers** in
`raft_sim.c`/`raft.h` (they are genuinely "simulator" primitives and reuse existing node
state). Lift the **scheduler + five scenarios + M-seed sweep + negative test** into a
new `test_raft_conformance.c` (or a clearly-fenced section) so `test_raft.c` does not grow
unbounded. Fault switches stay in `raft_sim.c` (adapter) per boundary.
- ✅ Best file-size/cohesion balance; preserves the "harness drives only published entry points" boundary; checkers stay reusable; the heavy scenario/sweep code is isolated.
- ❌ Splits ownership across two test files — needs a one-line boundary amendment in `tasks.md`/`design.md` (Allowed Dependencies already permit a new test file under `src/lib/testsuite`).

## 4. Effort & Risk

- **Effort: L (1–2 weeks).** Eight task groups; the hard parts are the seeded scheduler
  with enabled-action selection (4.2), deterministic byte-identical replay (4.4/1.6), the
  three new invariant checkers (3.1/3.3/3.4) and the bug-851 Figure-8 / stale-rejoin
  scenarios (5.4/5.5). Substantial scaffolding already exists (adapter callbacks,
  partition, restart, Log Matching, AE/vote delivery), which pulls it down from XL.
- **Risk: Medium.** Mitigants: all library entry points exist and are stable; deps landed;
  no protocol change; synchronous/no-I/O design removes concurrency risk; the negative
  test self-validates checker sensitivity. Residual risk: (a) determinism leaks (pointer
  values / iteration order in action selection) — must be audited (Req 1.2); (b) the
  Figure-8 and stale-rejoin scenarios must be precise enough to actually exercise the
  bug-851 mechanism (else the oracle's "teeth" are blunt); (c) keeping fixed-size bounds
  (`RAFT_SIM_MAX_*`) large enough for randomized runs without blowing the CI time/ASAN
  budget. The optional TLC export (Req 8) is **Unknown** — no in-repo TLC to validate it.

## 5. Recommendations for Design Phase

- **Preferred approach: Option C (hybrid)** — checkers + bag + PRNG + fault switches in
  `raft_sim.c`/`raft.h`; scheduler + scenarios + sweep + negative test in a new
  `src/lib/testsuite/test_raft_conformance.c`. Amend the `tasks.md`/`design.md` file
  boundary with that one new test file (design "Allowed Dependencies" already permits it).
  If strict adherence to the stated single-file boundary is required, fall back to
  Option A and accept the file-size growth (flag a `clang-format`/size exception).
- **Key decisions to lock:** (1) PRNG = SplitMix64 inlined, single `cluster.seed` source —
  forbid any wall-clock/address input (Req 1.2); (2) invariant checkers read ONLY
  harness-owned in-memory state (`log[]`/`commit_index`/`applied[]`) + `raft_snapshot`
  (role/term) — never `raft.c` internals (Req 3.6); (3) `ClientRequest` drives
  `raft_log_append` on a forced leader; `Duplicate`/`Drop`/reorder mutate ONLY the bag
  (Req 2.5); (4) fault switches confined to adapter callbacks, default-off, never set in
  the default CI run (Req 7.3).
- **Research items to carry forward:** (a) **TLC consumability (Req 8)** — no
  `tla2tools`/`.cfg` exists in-repo; decide whether to add TLC tooling or keep Req 8 as a
  documented-format-only deferral. (b) confirm `RAFT_SIM_MAX_LOG=64` / step bounds are
  large enough for randomized convergence yet inside the ASAN/UBSAN CI budget (Req 6.4).
  (c) verify the existing per-node `applied[]`/state-machine view needs adding (not present
  in current `raft_sim_node_t`) for State Machine Safety (Req 3.4).
- **Dependency watch:** harness consumes `raft-log-replication` + `raft-commit-safety`
  callback signatures verbatim; per design "Revalidation Triggers," any change to
  `on_persist_entry`/`on_get_entry_term`/`on_send_append_entries`/`on_truncate_from`/
  `on_commit_advanced` or the `(lastLogTerm,lastLogIndex)` freshness contract re-validates
  this spec.