myxce/.kiro/specs/raft-sim-conformance/research.md

Gap Analysis — raft-sim-conformance (2026-06-24)

Analyzed per .claude/skills/kiro-validate-gap/rules/gap-analysis.md. Scope: a single feature — an in-process, deterministic, seed-driven harness that drives N src/lib/raft library instances through raft.tla-Next-equivalent transitions and asserts the four Raft safety invariants every step (the "use the verified spec as an oracle" payoff; acceptance bar = catch a bug-851-class divergence on a deliberately-broken build, pass on the correct build).

1. Current State Investigation

Authoritative spec

• raft/raft.tla — Diego Ongaro's canonical Raft (abstract spec; ends raft.tla:471). All Next actions present and located: Restart (167), Timeout (178), RequestVote (190), AppendEntries (204), BecomeLeader (229), ClientRequest (246), AdvanceCommitIndex (259), Receive (421) with UpdateTerm (406) and DropStaleResponse (415), DuplicateMessage (443), DropMessage (448), Next disjunction (454). The .tla file carries NO embedded invariant THEOREM block; the four safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) are tracked prose-side in src/controller/CLAUDE.md. No TLC config (*.cfg), no tla2tools/TLC tooling exists in-repo — the spec is a reference document, not a model-checked artifact today.

How the C library is checked against raft.tla today

• It is not, automatically. Conformance is enforced by hand: src/controller/CLAUDE.md holds the TLA+ variable→code and action→code mapping plus a manual "Protocol Verification Checklist." There is no trace generation, no invariant checker that runs per-step, and no randomized simulation that asserts the four invariants.
• src/controller/raft/ does not exist; the library lives at src/lib/raft/ (raft.c 33 KB, raft.h 30 KB, raft_sim.c 18 KB). Per src/controller/CLAUDE.md, the controller's raft.c was election-only; the full election+log+commit protocol now lives in the library src/lib/raft (deps raft-log-replication + raft-commit-safety landed — callbacks present and exercised in test_raft.c).

Existing harness (src/lib/raft/raft_sim.c, XCE_TESTING-gated)

Already present (the election + partial log layer it was grown to for the dependency specs), all consuming published library entry points verbatim:

• raft_sim_init, raft_sim_deliver_pending (vote RPC routing via raft_on_vote_request/_reply), raft_sim_deliver_append_entries (AE routing via raft_on_append_entries_request/_reply), raft_sim_start_all_elections, raft_sim_fire_timeout, raft_sim_partition, raft_sim_advance_commit (raft_advance_commit), raft_sim_restart_node (models Restart, preserves persisted term/votedFor/log/commit_index), and one invariant checker: raft_sim_check_log_matching (raft_sim.c:424).
• In-memory adapter callbacks already back the four log callbacks + on_commit_advanced into per-node raft_sim_node_t (log[], log_count, commit_index, commit_advanced_count, trace_buf). The library holds no entry bytes — the simulator owns storage, exactly as the dependency design anticipated.
• raft.h XCE_TESTING section (raft.h:432+) already defines raft_sim_rpc_t, raft_sim_entry_t, raft_sim_ae_t, raft_sim_node_t, raft_sim_cluster_t, and bounds RAFT_SIM_MAX_NODES=8, RAFT_SIM_MAX_LOG=64, RAFT_SIM_ENTRY_LEN=16, RAFT_SIM_MAX_AE=64.
• test_raft.c (7011 lines) already exercises election + AE + commit + a hand-built partition/catch-up convergence path and both directions of raft_sim_check_log_matching (pass-on-converged / fail-on-divergence).

All library entry points the harness needs already exist in raft.h

raft_log_append, raft_send_append_entries, raft_advance_commit, raft_on_append_entries_request/_reply, raft_on_vote_request/_reply, raft_init, raft_start_election, raft_on_election_timeout, raft_snapshot — all declared. Constraint: raft_snapshot_t (raft.h:69-76) exposes term/role/voted_for/votes only — NOT log or commit_index. So Election Safety can read from raft_snapshot, but Log Matching / Leader Completeness / State Machine Safety MUST read the harness-owned in-memory log[]/commit_index/applied[] (Req 3.6 "observable per-node state only" is satisfied by the in-memory mirror, not by reaching into raft.c).

Spec phase state

spec.json = tasks-generated, all 3 gates auto-approved (-y), ready_for_implementation. tasks.md has 8 task groups (1.1–8.1); every task box is unchecked [ ] — none of the conformance machinery is implemented. The brief/requirements/design are complete and internally consistent.

2. Requirement-to-Asset Map (gaps tagged Missing / Unknown / Constraint)

Req	Need	Existing asset	Gap
1.1–1.6 Deterministic seeded driver	Seeded PRNG (SplitMix64), (N,seed) byte-identical replay, message bag, action trace buffer, no threads/sockets/wall-clock	raft_sim_init (no seed/PRNG); synchronous already	Missing: PRNG, seed, raft_sim_conformance_init, bag (bag[]/bag_count), trace buffer, raft_sim_run replay. Constraint: must extend the existing raft_sim_cluster_t/_node_t, not rewrite.
2.1–2.6 Drive raft.tla Next actions via library entry points	Restart/Timeout/RequestVote/BecomeLeader/ClientRequest/AppendEntries/AdvanceCommitIndex/Receive(UpdateTerm+DropStaleResponse)/Duplicate/Drop through published API	Vote+AE delivery, restart, advance-commit, partition all wired verbatim to library handlers	Missing: unified raft_sim_step scheduler selecting only enabled actions; ClientRequest via raft_log_append; bag-based Duplicate/Drop (current delivery drains a queue, no per-message drop/dup/reorder). UpdateTerm/DropStaleResponse are already inside the library handlers (consumed, not re-modeled).
3.1–3.6 Per-step assertion of all four invariants	Election Safety, Log Matching, Leader Completeness, State Machine Safety; first-failure halt; observable-state-only	Only raft_sim_check_log_matching exists	Missing: raft_sim_check_election_safety, _leader_completeness, _state_machine_safety, _check_all (first-failure composer); applied[] per-node state machine view.
4.1–4.5 Targeted adversarial scenarios	partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin (bug-851 G5)	A hand-built partition/catch-up test exists in test_raft.c (not a reusable named scenario)	Missing: 5 named raft_sim_scenario_* entry points asserting all four invariants throughout.
5.1–5.4 Failure-trace capture/replay	seed+N+failing-invariant+ordered action trace dump; quiet-on-success; replay reproducibility	per-node trace_buf (string events only, not structured actions)	Missing: structured raft_sim_action_t trace + raft_sim_dump_trace; replay tie-in to raft_sim_run.
6.1–6.4 CI-runnable target	M-seed sweep + all scenarios in src/lib/testsuite, ASAN/UBSAN-clean, bounded runtime	test_raft.c already builds/runs under XCE_TESTING + sanitizers	Missing: the sweep harness + bounds; mechanics (test entry point, sanitizer build) already exist. Low-risk.
7.1–7.4 Deliberately-broken-build negative test	compile-time fault switches (RAFT_SIM_BREAK_TERM_STAMP, RAFT_SIM_BREAK_TRUNCATE) confined to adapter; both-directions assert	none	Missing. Constraint: switches must live in the adapter (raft_sim.c callbacks), never raft.c — the boundary forbids touching protocol code. This is the acceptance teeth.
8.1–8.3 Optional TLC-consumable trace export	spec-named machine-readable record stream, off by default	none	Missing (optional [ ]*, deferred). Unknown/Research: no TLC tooling in-repo to actually consume it — export format can be written but cross-checking against raft.tla via TLC is unproven.

Cross-cutting findings

• Currently CHECKED against raft.tla (in code): Log Matching only (one checker, raft_sim.c:424), and only on hand-driven clusters — not per-step, not over randomized runs. Election Safety / Leader Completeness / State Machine Safety are UNCHECKED in code (asserted only by prose checklist). Of the Next actions, Restart / Timeout / RequestVote / AppendEntries / AdvanceCommitIndex / Receive are driven by the existing sim; ClientRequest, per-message Duplicate/Drop/reorder, and a unified enabled-action scheduler are not yet driven.
• raft.tla does NOT model durability (restores log intact on Restart, raft.tla:165-174) — so the harness, like the spec, cannot and should not assert the commit-durability invariant (that rests on ha-commit-durability tests). This correctly scopes the harness to the four safety invariants only. Confirmed against memory.

3. Implementation Approach Options

Option A — Extend raft_sim.c / test_raft.c in place (spec's chosen path)

Add PRNG, bag, action trace, three new checkers + composer, raft_sim_step/_run, five scenarios, fault switches — all inside the existing XCE_TESTING sim section and test file. Reuse raft_sim_node_t/_cluster_t (extend with bag[], applied[], seed, trace[]).

• ✅ Matches the spec boundary exactly (owns raft_sim.c + raft.h XCE_TESTING + test_raft.c); reuses the verbatim-library-entry-point pattern already proven by the dependency specs; no protocol change; sanitizer/CI plumbing already in place.
• ❌ raft_sim.c grows well past the existing 18 KB (toward the 800-line guideline ceiling); test_raft.c is already 7011 lines.

Option B — New raft_conformance.c / test_raft_conformance.c modules

Put the harness (PRNG, bag, scheduler, checkers, scenarios, fault switches) in dedicated files; keep raft_sim.c as the thin election/AE adapter.

• ✅ Honors "many small files"; isolates the oracle from the dependency-era sim; easier to read/maintain; keeps test_raft.c from ballooning further.
• ❌ Deviates from the spec's stated file boundary ("owns raft_sim.c … ONLY"); needs a meson build wiring change and a boundary-note update; minor duplication of the in-memory adapter.

Option C — Hybrid (recommended)

Keep the in-memory adapter callbacks, bag, PRNG, and the four checkers in raft_sim.c/raft.h (they are genuinely "simulator" primitives and reuse existing node state). Lift the scheduler + five scenarios + M-seed sweep + negative test into a new test_raft_conformance.c (or a clearly-fenced section) so test_raft.c does not grow unbounded. Fault switches stay in raft_sim.c (adapter) per boundary.

• ✅ Best file-size/cohesion balance; preserves the "harness drives only published entry points" boundary; checkers stay reusable; the heavy scenario/sweep code is isolated.
• ❌ Splits ownership across two test files — needs a one-line boundary amendment in tasks.md/design.md (Allowed Dependencies already permit a new test file under src/lib/testsuite).

4. Effort & Risk

• Effort: L (1–2 weeks). Eight task groups; the hard parts are the seeded scheduler with enabled-action selection (4.2), deterministic byte-identical replay (4.4/1.6), the three new invariant checkers (3.1/3.3/3.4) and the bug-851 Figure-8 / stale-rejoin scenarios (5.4/5.5). Substantial scaffolding already exists (adapter callbacks, partition, restart, Log Matching, AE/vote delivery), which pulls it down from XL.
• Risk: Medium. Mitigants: all library entry points exist and are stable; deps landed; no protocol change; synchronous/no-I/O design removes concurrency risk; the negative test self-validates checker sensitivity. Residual risk: (a) determinism leaks (pointer values / iteration order in action selection) — must be audited (Req 1.2); (b) the Figure-8 and stale-rejoin scenarios must be precise enough to actually exercise the bug-851 mechanism (else the oracle's "teeth" are blunt); (c) keeping fixed-size bounds (RAFT_SIM_MAX_*) large enough for randomized runs without blowing the CI time/ASAN budget. The optional TLC export (Req 8) is Unknown — no in-repo TLC to validate it.

5. Recommendations for Design Phase

• Preferred approach: Option C (hybrid) — checkers + bag + PRNG + fault switches in raft_sim.c/raft.h; scheduler + scenarios + sweep + negative test in a new src/lib/testsuite/test_raft_conformance.c. Amend the tasks.md/design.md file boundary with that one new test file (design "Allowed Dependencies" already permits it). If strict adherence to the stated single-file boundary is required, fall back to Option A and accept the file-size growth (flag a clang-format/size exception).
• Key decisions to lock: (1) PRNG = SplitMix64 inlined, single cluster.seed source — forbid any wall-clock/address input (Req 1.2); (2) invariant checkers read ONLY harness-owned in-memory state (log[]/commit_index/applied[]) + raft_snapshot (role/term) — never raft.c internals (Req 3.6); (3) ClientRequest drives raft_log_append on a forced leader; Duplicate/Drop/reorder mutate ONLY the bag (Req 2.5); (4) fault switches confined to adapter callbacks, default-off, never set in the default CI run (Req 7.3).
• Research items to carry forward: (a) TLC consumability (Req 8) — no tla2tools/.cfg exists in-repo; decide whether to add TLC tooling or keep Req 8 as a documented-format-only deferral. (b) confirm RAFT_SIM_MAX_LOG=64 / step bounds are large enough for randomized convergence yet inside the ASAN/UBSAN CI budget (Req 6.4). (c) verify the existing per-node applied[]/state-machine view needs adding (not present in current raft_sim_node_t) for State Machine Safety (Req 3.4).
• Dependency watch: harness consumes raft-log-replication + raft-commit-safety callback signatures verbatim; per design "Revalidation Triggers," any change to on_persist_entry/on_get_entry_term/on_send_append_entries/on_truncate_from/ on_commit_advanced or the (lastLogTerm,lastLogIndex) freshness contract re-validates this spec.