Rongsong Shen
05febbc9a7
/kiro-validate-gap gap-focused sweep. Per-spec gap analysis written to each spec's research.md (3 new, 2 appended), grounded in file:line evidence: - config-upgrade-snapshot (new): real open gap — transport + (term,lsn) marker + COW substrate exist, but the entity-bearing snapshot PAYLOAD does not (16-byte LSN sentinel). HARD-blocked on entity-bearing-snapshots. Closes bug-857. Effort L (XL incl. blocker) / Risk High. - raft-commit-safety (new): CLOSED — G4 Figure-8 commit (raft.c:733) + G6 vote freshness (raft.c:157) done, full sim coverage; the "(0,0) stopgap / broken build" concern is stale (build green, real vote tuple at cluster.c:435). - raft-sim-conformance (new): mostly open — raft.tla is reference-only; only 1 of 4 invariants (Log Matching) checked in code. Missing PRNG/replay, message-fault bag, scheduler, 3 invariant checkers, fault switches, scenarios. Effort L / Risk Medium. - ha-repl-hardening (append): RESCOPE/superseded — 31/33 EARS closed; Req 2.3 + Req 6 obsolete (CTRL_PRIMARY_COMMITTING removed). Residual: docs only. - ha-convergence-recovery (append): done bar Task 8 (live E2E); bug-1028 post-failover availability resolved 2026-06-23. Skipped ha-elect-ctx-lifecycle (no requirements.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
165 lines
14 KiB
Markdown
165 lines
14 KiB
Markdown
# Gap Analysis — raft-sim-conformance (2026-06-24)
|
||
|
||
Analyzed per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Scope: a single
|
||
feature — an in-process, deterministic, seed-driven harness that drives N `src/lib/raft`
|
||
library instances through `raft.tla`-`Next`-equivalent transitions and asserts the four
|
||
Raft safety invariants every step (the "use the verified spec as an oracle" payoff;
|
||
acceptance bar = catch a bug-851-class divergence on a deliberately-broken build, pass on
|
||
the correct build).
|
||
|
||
## 1. Current State Investigation
|
||
|
||
### Authoritative spec
|
||
- `raft/raft.tla` — Diego Ongaro's canonical Raft (abstract spec; ends `raft.tla:471`).
|
||
All `Next` actions present and located: `Restart` (167), `Timeout` (178),
|
||
`RequestVote` (190), `AppendEntries` (204), `BecomeLeader` (229), `ClientRequest` (246),
|
||
`AdvanceCommitIndex` (259), `Receive` (421) with `UpdateTerm` (406) and
|
||
`DropStaleResponse` (415), `DuplicateMessage` (443), `DropMessage` (448),
|
||
`Next` disjunction (454). The `.tla` file carries NO embedded invariant `THEOREM` block;
|
||
the four safety invariants (Election Safety, Log Matching, Leader Completeness, State
|
||
Machine Safety) are tracked prose-side in `src/controller/CLAUDE.md`. **No TLC config
|
||
(`*.cfg`), no `tla2tools`/TLC tooling exists in-repo** — the spec is a reference
|
||
document, not a model-checked artifact today.
|
||
|
||
### How the C library is checked against raft.tla today
|
||
- **It is not, automatically.** Conformance is enforced by hand: `src/controller/CLAUDE.md`
|
||
holds the TLA+ variable→code and action→code mapping plus a manual "Protocol
|
||
Verification Checklist." There is no trace generation, no invariant checker that runs
|
||
per-step, and no randomized simulation that asserts the four invariants.
|
||
- `src/controller/raft/` does **not** exist; the library lives at `src/lib/raft/`
|
||
(`raft.c` 33 KB, `raft.h` 30 KB, `raft_sim.c` 18 KB). Per `src/controller/CLAUDE.md`,
|
||
the controller's `raft.c` was election-only; the full election+log+commit protocol now
|
||
lives in the **library** `src/lib/raft` (deps `raft-log-replication` +
|
||
`raft-commit-safety` landed — callbacks present and exercised in `test_raft.c`).
|
||
|
||
### Existing harness (`src/lib/raft/raft_sim.c`, `XCE_TESTING`-gated)
|
||
Already present (the election + partial log layer it was grown to for the dependency
|
||
specs), all consuming published library entry points verbatim:
|
||
- `raft_sim_init`, `raft_sim_deliver_pending` (vote RPC routing via
|
||
`raft_on_vote_request`/`_reply`), `raft_sim_deliver_append_entries` (AE routing via
|
||
`raft_on_append_entries_request`/`_reply`), `raft_sim_start_all_elections`,
|
||
`raft_sim_fire_timeout`, `raft_sim_partition`, `raft_sim_advance_commit`
|
||
(`raft_advance_commit`), `raft_sim_restart_node` (models Restart, preserves persisted
|
||
term/votedFor/log/commit_index), and **one** invariant checker:
|
||
`raft_sim_check_log_matching` (`raft_sim.c:424`).
|
||
- In-memory adapter callbacks already back the four log callbacks + `on_commit_advanced`
|
||
into per-node `raft_sim_node_t` (`log[]`, `log_count`, `commit_index`,
|
||
`commit_advanced_count`, `trace_buf`). The library holds no entry bytes — the simulator
|
||
owns storage, exactly as the dependency design anticipated.
|
||
- `raft.h` `XCE_TESTING` section (`raft.h:432+`) already defines `raft_sim_rpc_t`,
|
||
`raft_sim_entry_t`, `raft_sim_ae_t`, `raft_sim_node_t`, `raft_sim_cluster_t`, and bounds
|
||
`RAFT_SIM_MAX_NODES=8`, `RAFT_SIM_MAX_LOG=64`, `RAFT_SIM_ENTRY_LEN=16`,
|
||
`RAFT_SIM_MAX_AE=64`.
|
||
- `test_raft.c` (7011 lines) already exercises election + AE + commit + a
|
||
hand-built partition/catch-up convergence path and both directions of
|
||
`raft_sim_check_log_matching` (pass-on-converged / fail-on-divergence).
|
||
|
||
### All library entry points the harness needs already exist in `raft.h`
|
||
`raft_log_append`, `raft_send_append_entries`, `raft_advance_commit`,
|
||
`raft_on_append_entries_request`/`_reply`, `raft_on_vote_request`/`_reply`, `raft_init`,
|
||
`raft_start_election`, `raft_on_election_timeout`, `raft_snapshot` — all declared.
|
||
**Constraint:** `raft_snapshot_t` (`raft.h:69-76`) exposes term/role/voted_for/votes only —
|
||
NOT log or commit_index. So Election Safety can read from `raft_snapshot`, but Log
|
||
Matching / Leader Completeness / State Machine Safety MUST read the harness-owned
|
||
in-memory `log[]`/`commit_index`/`applied[]` (Req 3.6 "observable per-node state only" is
|
||
satisfied by the in-memory mirror, not by reaching into `raft.c`).
|
||
|
||
### Spec phase state
|
||
`spec.json` = `tasks-generated`, all 3 gates auto-approved (`-y`), `ready_for_implementation`.
|
||
`tasks.md` has 8 task groups (1.1–8.1); **every task box is unchecked `[ ]`** — none of
|
||
the conformance machinery is implemented. The brief/requirements/design are complete and
|
||
internally consistent.
|
||
|
||
## 2. Requirement-to-Asset Map (gaps tagged Missing / Unknown / Constraint)
|
||
|
||
| Req | Need | Existing asset | Gap |
|
||
|-----|------|----------------|-----|
|
||
| 1.1–1.6 Deterministic seeded driver | Seeded PRNG (SplitMix64), `(N,seed)` byte-identical replay, message bag, action trace buffer, no threads/sockets/wall-clock | `raft_sim_init` (no seed/PRNG); synchronous already | **Missing**: PRNG, `seed`, `raft_sim_conformance_init`, bag (`bag[]`/`bag_count`), trace buffer, `raft_sim_run` replay. Constraint: must extend the existing `raft_sim_cluster_t`/`_node_t`, not rewrite. |
|
||
| 2.1–2.6 Drive `raft.tla` Next actions via library entry points | `Restart`/`Timeout`/`RequestVote`/`BecomeLeader`/`ClientRequest`/`AppendEntries`/`AdvanceCommitIndex`/`Receive`(UpdateTerm+DropStaleResponse)/`Duplicate`/`Drop` through published API | Vote+AE delivery, restart, advance-commit, partition all wired verbatim to library handlers | **Missing**: unified `raft_sim_step` scheduler selecting only *enabled* actions; `ClientRequest` via `raft_log_append`; bag-based `Duplicate`/`Drop` (current delivery drains a queue, no per-message drop/dup/reorder). UpdateTerm/DropStaleResponse are already inside the library handlers (consumed, not re-modeled). |
|
||
| 3.1–3.6 Per-step assertion of all four invariants | Election Safety, Log Matching, Leader Completeness, State Machine Safety; first-failure halt; observable-state-only | Only `raft_sim_check_log_matching` exists | **Missing**: `raft_sim_check_election_safety`, `_leader_completeness`, `_state_machine_safety`, `_check_all` (first-failure composer); `applied[]` per-node state machine view. |
|
||
| 4.1–4.5 Targeted adversarial scenarios | partition/heal, restart-churn, drop/dup/reorder, Figure-8, stale-rejoin (bug-851 G5) | A hand-built partition/catch-up test exists in `test_raft.c` (not a reusable named scenario) | **Missing**: 5 named `raft_sim_scenario_*` entry points asserting all four invariants throughout. |
|
||
| 5.1–5.4 Failure-trace capture/replay | seed+N+failing-invariant+ordered action trace dump; quiet-on-success; replay reproducibility | per-node `trace_buf` (string events only, not structured actions) | **Missing**: structured `raft_sim_action_t` trace + `raft_sim_dump_trace`; replay tie-in to `raft_sim_run`. |
|
||
| 6.1–6.4 CI-runnable target | M-seed sweep + all scenarios in `src/lib/testsuite`, ASAN/UBSAN-clean, bounded runtime | `test_raft.c` already builds/runs under `XCE_TESTING` + sanitizers | **Missing**: the sweep harness + bounds; mechanics (test entry point, sanitizer build) already exist. Low-risk. |
|
||
| 7.1–7.4 Deliberately-broken-build negative test | compile-time fault switches (`RAFT_SIM_BREAK_TERM_STAMP`, `RAFT_SIM_BREAK_TRUNCATE`) confined to adapter; both-directions assert | none | **Missing**. **Constraint**: switches must live in the *adapter* (`raft_sim.c` callbacks), never `raft.c` — the boundary forbids touching protocol code. This is the acceptance teeth. |
|
||
| 8.1–8.3 Optional TLC-consumable trace export | spec-named machine-readable record stream, off by default | none | **Missing** (optional `[ ]*`, deferred). **Unknown/Research**: no TLC tooling in-repo to actually consume it — export format can be written but cross-checking against `raft.tla` via TLC is unproven. |
|
||
|
||
### Cross-cutting findings
|
||
- **Currently CHECKED against raft.tla (in code):** Log Matching only (one checker,
|
||
`raft_sim.c:424`), and only on hand-driven clusters — not per-step, not over randomized
|
||
runs. Election Safety / Leader Completeness / State Machine Safety are **UNCHECKED in
|
||
code** (asserted only by prose checklist). Of the `Next` actions, Restart / Timeout /
|
||
RequestVote / AppendEntries / AdvanceCommitIndex / Receive are *driven* by the existing
|
||
sim; `ClientRequest`, per-message `Duplicate`/`Drop`/reorder, and a unified
|
||
enabled-action scheduler are **not yet driven**.
|
||
- **raft.tla does NOT model durability** (restores `log` intact on Restart,
|
||
`raft.tla:165-174`) — so the harness, like the spec, cannot and should not assert the
|
||
commit-durability invariant (that rests on `ha-commit-durability` tests). This correctly
|
||
scopes the harness to the four safety invariants only. Confirmed against memory.
|
||
|
||
## 3. Implementation Approach Options
|
||
|
||
### Option A — Extend `raft_sim.c` / `test_raft.c` in place (spec's chosen path)
|
||
Add PRNG, bag, action trace, three new checkers + composer, `raft_sim_step`/`_run`, five
|
||
scenarios, fault switches — all inside the existing `XCE_TESTING` sim section and test
|
||
file. Reuse `raft_sim_node_t`/`_cluster_t` (extend with `bag[]`, `applied[]`, `seed`,
|
||
`trace[]`).
|
||
- ✅ Matches the spec boundary exactly (owns `raft_sim.c` + `raft.h` XCE_TESTING + `test_raft.c`); reuses the verbatim-library-entry-point pattern already proven by the dependency specs; no protocol change; sanitizer/CI plumbing already in place.
|
||
- ❌ `raft_sim.c` grows well past the existing 18 KB (toward the 800-line guideline ceiling); `test_raft.c` is already 7011 lines.
|
||
|
||
### Option B — New `raft_conformance.c` / `test_raft_conformance.c` modules
|
||
Put the harness (PRNG, bag, scheduler, checkers, scenarios, fault switches) in dedicated
|
||
files; keep `raft_sim.c` as the thin election/AE adapter.
|
||
- ✅ Honors "many small files"; isolates the oracle from the dependency-era sim; easier to read/maintain; keeps `test_raft.c` from ballooning further.
|
||
- ❌ Deviates from the spec's stated file boundary ("owns `raft_sim.c` … ONLY"); needs a meson build wiring change and a boundary-note update; minor duplication of the in-memory adapter.
|
||
|
||
### Option C — Hybrid (recommended)
|
||
Keep the in-memory adapter callbacks, bag, PRNG, and the **four checkers** in
|
||
`raft_sim.c`/`raft.h` (they are genuinely "simulator" primitives and reuse existing node
|
||
state). Lift the **scheduler + five scenarios + M-seed sweep + negative test** into a
|
||
new `test_raft_conformance.c` (or a clearly-fenced section) so `test_raft.c` does not grow
|
||
unbounded. Fault switches stay in `raft_sim.c` (adapter) per boundary.
|
||
- ✅ Best file-size/cohesion balance; preserves the "harness drives only published entry points" boundary; checkers stay reusable; the heavy scenario/sweep code is isolated.
|
||
- ❌ Splits ownership across two test files — needs a one-line boundary amendment in `tasks.md`/`design.md` (Allowed Dependencies already permit a new test file under `src/lib/testsuite`).
|
||
|
||
## 4. Effort & Risk
|
||
|
||
- **Effort: L (1–2 weeks).** Eight task groups; the hard parts are the seeded scheduler
|
||
with enabled-action selection (4.2), deterministic byte-identical replay (4.4/1.6), the
|
||
three new invariant checkers (3.1/3.3/3.4) and the bug-851 Figure-8 / stale-rejoin
|
||
scenarios (5.4/5.5). Substantial scaffolding already exists (adapter callbacks,
|
||
partition, restart, Log Matching, AE/vote delivery), which pulls it down from XL.
|
||
- **Risk: Medium.** Mitigants: all library entry points exist and are stable; deps landed;
|
||
no protocol change; synchronous/no-I/O design removes concurrency risk; the negative
|
||
test self-validates checker sensitivity. Residual risk: (a) determinism leaks (pointer
|
||
values / iteration order in action selection) — must be audited (Req 1.2); (b) the
|
||
Figure-8 and stale-rejoin scenarios must be precise enough to actually exercise the
|
||
bug-851 mechanism (else the oracle's "teeth" are blunt); (c) keeping fixed-size bounds
|
||
(`RAFT_SIM_MAX_*`) large enough for randomized runs without blowing the CI time/ASAN
|
||
budget. The optional TLC export (Req 8) is **Unknown** — no in-repo TLC to validate it.
|
||
|
||
## 5. Recommendations for Design Phase
|
||
|
||
- **Preferred approach: Option C (hybrid)** — checkers + bag + PRNG + fault switches in
|
||
`raft_sim.c`/`raft.h`; scheduler + scenarios + sweep + negative test in a new
|
||
`src/lib/testsuite/test_raft_conformance.c`. Amend the `tasks.md`/`design.md` file
|
||
boundary with that one new test file (design "Allowed Dependencies" already permits it).
|
||
If strict adherence to the stated single-file boundary is required, fall back to
|
||
Option A and accept the file-size growth (flag a `clang-format`/size exception).
|
||
- **Key decisions to lock:** (1) PRNG = SplitMix64 inlined, single `cluster.seed` source —
|
||
forbid any wall-clock/address input (Req 1.2); (2) invariant checkers read ONLY
|
||
harness-owned in-memory state (`log[]`/`commit_index`/`applied[]`) + `raft_snapshot`
|
||
(role/term) — never `raft.c` internals (Req 3.6); (3) `ClientRequest` drives
|
||
`raft_log_append` on a forced leader; `Duplicate`/`Drop`/reorder mutate ONLY the bag
|
||
(Req 2.5); (4) fault switches confined to adapter callbacks, default-off, never set in
|
||
the default CI run (Req 7.3).
|
||
- **Research items to carry forward:** (a) **TLC consumability (Req 8)** — no
|
||
`tla2tools`/`.cfg` exists in-repo; decide whether to add TLC tooling or keep Req 8 as a
|
||
documented-format-only deferral. (b) confirm `RAFT_SIM_MAX_LOG=64` / step bounds are
|
||
large enough for randomized convergence yet inside the ASAN/UBSAN CI budget (Req 6.4).
|
||
(c) verify the existing per-node `applied[]`/state-machine view needs adding (not present
|
||
in current `raft_sim_node_t`) for State Machine Safety (Req 3.4).
|
||
- **Dependency watch:** harness consumes `raft-log-replication` + `raft-commit-safety`
|
||
callback signatures verbatim; per design "Revalidation Triggers," any change to
|
||
`on_persist_entry`/`on_get_entry_term`/`on_send_append_entries`/`on_truncate_from`/
|
||
`on_commit_advanced` or the `(lastLogTerm,lastLogIndex)` freshness contract re-validates
|
||
this spec.
|