myxce/.kiro/specs/raft-log-replication/brief.md

Brief: raft-log-replication

Problem

src/lib/raft/raft.{c,h} is an election-only Raft library (terms, votes, leader election, PreVote) with a clean callback interface and a deterministic simulator (raft_sim.c). It has no log, no AppendEntries, no commit. The replicated log is hand-rolled in the controller (persist.c WAL, repl.c catch-up streaming, ha.c, ha_convergence.c) and violates the Raft log invariants of raft/raft.tla, which root-caused bug-851 (silent committed-prefix divergence under chaotic failover). Authoritative gap analysis: .kiro/specs/ha-term-safety/TLA-gap-analysis.md.

This spec closes the four log-half gaps by making the library own the log, strictly following raft.tla:

• G1 — per-entry term is dead (WAL hdr.term always 0; p->term never set from ha->term).
• G2 — replication matches LSN contiguity only; no prevLogTerm check (TLA+ AppendEntries/HandleAppendEntriesRequest logOk).
• G3 — no (term,lsn) conflict truncation (TLA+ conflict branch: delete the divergent entry + all after).
• G5 — a node adopts a higher term only on vote RPCs; the AppendEntries/heartbeat path never advances currentTerm (TLA+ UpdateTerm fires on any higher-term RPC). Confirmed live: a rejoined standby reported term:2 under a term-3 primary → term reuse → two leaders per term.

Approach

Extend the library to own the replicated log, strictly mirroring the raft.tla actions, keeping the core pure / synchronous / deterministic (no I/O, no threads) so it stays simulator-verifiable. All side effects go through new callbacks the controller adapter implements over the existing WAL/repl transport.

TLA+ actions implemented (raft.tla line refs)

• ClientRequest(i,v) (246) → raft_log_append(...): append [term ↦ currentTerm, value].
• AppendEntries(i,j) (204) → emit via on_send_append_entries with prevLogIndex, prevLogTerm, ≤1+ entries, leaderCommit. Per-peer nextIndex/matchIndex (TLA+ leaderVars).
• HandleAppendEntriesRequest(i,j,m) (327) → raft_on_append_entries_request(...): logOk = prevLogIndex==0 ∨ log[prevLogIndex].term==prevLogTerm; reject (stale term / ¬logOk), candidate→follower on equal term, accept (already-have / conflict-truncate-one / append).
• HandleAppendEntriesResponse(i,j,m) (393) → raft_on_append_entries_reply(...): success → advance nextIndex/matchIndex; fail → decrement nextIndex.
• UpdateTerm(i,j,m) (406) → applied on every incoming RPC including AppendEntries/heartbeat (closes G5): higher term ⇒ adopt term, clear vote, step down to follower.

Authoritative shared contract (downstream specs MUST reference this verbatim)

Log index model: the library log index is 1-based (TLA+) and maps 1:1 to the controller WAL LSN. Per-entry term is uint64_t. New REQUIRED callbacks (pure; controller implements I/O):

• int on_persist_entry(void *ctx, uint64_t index, uint64_t term, const void *entry, size_t len) — durably append/overwrite at index.
• uint64_t on_get_entry_term(void *ctx, uint64_t index) — term of local entry at index (0 if absent).
• int on_send_append_entries(void *ctx, int peer_idx, uint64_t prev_log_index, uint64_t prev_log_term, uint64_t leader_commit, /* entries handle */ ...) — transmit.
• int on_truncate_from(void *ctx, uint64_t index) — delete local entries at index..end. New library entry points: raft_log_append, raft_on_append_entries_request, raft_on_append_entries_reply. Existing election callbacks/behaviour unchanged.

Scope

• In: library log model with per-entry term; ClientRequest append; AppendEntries send (per-peer next/match index); HandleAppendEntriesRequest accept/reject/conflict-truncate/append by (term,index); UpdateTerm on every RPC; controller adapter that maps callbacks onto the existing WAL/repl transport (WAL hdr.term stamped from ha->term, carried in the repl envelope, read back on recovery); simulator coverage asserting Log Matching.
• Out: AdvanceCommitIndex / commit-index safety + (term,lsn) vote freshness → raft-commit-safety. The full conformance harness → raft-sim-conformance. Config-change markers + upgrade snapshot → ha-term-safety. Async I/O, fan-out scheduling, the sync commit barrier, snapshot transport → controller callback adapters (not the pure core).

Preserve as policy (do NOT canonicalize away)

PreVote, priority leadership, persist-before-ACK durability + synchronous commit barrier (consumed in raft-commit-safety), "controller never self-terminates" (library returns statuses; controller owns liveness/fencing).

Verification

Extend raft_sim.c: drive AppendEntries across partitions, message drop/dup/reorder, and restarts; assert the Log Matching invariant (same (index,term) ⇒ identical prefix) and that a follower truncates a divergent suffix. Replace the controller's LSN-only reconcile with (term,index) matching delegated to the library.

Dependencies

raft-library (election core, implemented [x]). Feeds raft-commit-safety, raft-sim-conformance, and ha-term-safety integration. Closes bug-851 G1/G2/G3/G5.