myxce/.kiro/specs/raft-log-replication/tasks.md

# Implementation Plan

> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/lib/testsuite/test_raft.c` (and the simulator backing in `raft_sim.c`) that fails before the `raft.c` change and passes after. Build/run library tests via `/xce-build lib`.
>
> **Spec boundary** (applies to ALL tasks): this spec owns the `src/lib/raft` log additions (per-entry term in the library model, `ClientRequest`/`AppendEntries`/`Handle*`/`UpdateTerm`-on-AE) and the `(term, index)` reconcile seam (the four callbacks). It does **NOT** own commit-index advance, `(lastLogTerm,lastLogIndex)` vote freshness, config markers/upgrade snapshot, or the controller's WAL/repl/recovery internals — those are `raft-commit-safety` and `ha-term-safety`. No task here edits any controller file; the controller adapter is described, not built.

## 1. Header contract: log model, callbacks, entry points

- [x] 1.1 Extend `raft.h` with the log model state and config restore field
  - Add `uint64_t log_len;`, `uint64_t next_index[RAFT_MAX_PEERS];`, `uint64_t match_index[RAFT_MAX_PEERS];` to `raft_state_t`
  - Add `uint64_t initial_log_len;` to `raft_config_t` (restored WAL length at boot, default 0; library never reads storage)
  - Observable completion: `raft.h` compiles; `sizeof(raft_state_t)` grows by the new fixed fields; no entry-byte storage added
  - _Requirements: 1.1, 1.2, 3.1_
  - _Boundary: raft.h_

- [x] 1.2 Add the four new REQUIRED callbacks to `raft_callbacks_t` (verbatim contract)
  - Declare `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from` with the exact signatures from design.md / brief
  - Document each as REQUIRED and the `on_get_entry_term` → `0`-means-absent convention
  - Observable completion: the four function-pointer members exist in `raft_callbacks_t` with the verbatim signatures; header doc comments match the Authoritative shared contract
  - _Requirements: 7.1, 7.2, 7.3, 7.4_
  - _Boundary: raft.h_
  - _Depends: 1.1_

- [x] 1.3 Declare the three new entry points and `raft_append_result_t`
  - NOTE (impl 2026-06-21): `on_send_append_entries`'s design "entries handle `...`" varargs placeholder was concretized to an explicit `int n_entries` (0=heartbeat, 1=one entry at `prev_log_index+1`; adapter sources the bytes since the library holds none). design.md §"Seam — raft_callbacks_t additions" updated to match. Downstream `raft-commit-safety` / `raft-sim-conformance` consume this concrete shape.
  - Declare `raft_append_result_t`; declare `raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`, `raft_on_append_entries_reply` with the design.md signatures
  - Observable completion: header declares all new entry points; a stub TU including `raft.h` compiles
  - _Requirements: 2.1, 3.3, 4.7, 5.1_
  - _Boundary: raft.h_
  - _Depends: 1.2_

## 2. Init validation and leader-progress initialization

- [x] 2.1 RED+GREEN: `raft_init` requires the four new callbacks
  - Test: `raft_init` returns `-EINVAL` when any of the four new callbacks is NULL; returns 0 when all present
  - Implement the NULL checks in `raft_init` (mirroring the existing required-callback checks); restore `s->log_len = cfg->initial_log_len`
  - Observable completion: new `-EINVAL` test cases pass; existing election init tests still green
  - _Requirements: 7.1, 7.2, 7.3, 7.4, 1.1_
  - _Boundary: raft.c, test_raft.c_
  - _Depends: 1.3_

- [x] 2.2 RED+GREEN: leader progress initialized on becoming leader
  - Test: after a node becomes leader, `next_index[j] == log_len + 1` and `match_index[j] == 0` for every peer (raft.tla BecomeLeader)
  - Implement the `next_index`/`match_index` initialization on the win path (`raft_start_election` single-node + the multi-node become-leader transition)
  - Observable completion: assertion on `next_index`/`match_index` after election passes; no regression in election tests
  - _Requirements: 3.1_
  - _Boundary: raft.c, test_raft.c_
  - _Depends: 2.1_

## 3. ClientRequest — raft_log_append (raft.tla:246)

- [x] 3.1 RED: append tests (leader stamps term, follower rejected)
  - Test: leader `raft_log_append` assigns index `log_len+1`, stamps `currentTerm`, calls `on_persist_entry` exactly once with `(index, term, entry)`, returns the index; follower call returns `-EPERM` with no log change and no callback
  - Observable completion: tests compile and FAIL (no implementation yet)
  - _Requirements: 2.1, 2.2, 2.3, 2.4_
  - _Boundary: test_raft.c_
  - _Depends: 2.2_

- [x] 3.2 GREEN: implement `raft_log_append`
  - Role guard (`-EPERM` if not leader); persist-before-act (`on_persist_entry` before incrementing `log_len`); set `*out_index`; do NOT advance any commit index
  - Observable completion: 3.1 tests pass; `log_len` only advances on a successful persist (no phantom entry on persist failure)
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_
  - _Boundary: raft.c_
  - _Depends: 3.1_

## 4. AppendEntries emission — raft_send_append_entries (raft.tla:204)

- [x] 4.1 RED: emit tests (prevLogIndex/prevLogTerm, ≤1 entry, leader-only)
  - Test: for peer `j`, emit computes `prevLogIndex = next_index[j]-1` and `prevLogTerm` via `on_get_entry_term` (0 when `prevLogIndex==0`), sends ≤1 entry, passes `leader_commit` through opaquely; a non-leader emits nothing
  - Observable completion: tests compile and FAIL
  - _Requirements: 3.2, 3.3, 3.4, 3.5_
  - _Boundary: test_raft.c_
  - _Depends: 2.2_

- [x] 4.2 GREEN: implement `raft_send_append_entries`
  - Compute prev index/term; `lastEntry = Min(log_len, next_index[j])`; one entry or heartbeat; call `on_send_append_entries` with the opaque `leader_commit`; leader-only guard
  - Observable completion: 4.1 tests pass; heartbeat (0 entries) emitted when `next_index[j] > log_len`
  - _Requirements: 3.2, 3.3, 3.4, 3.5_
  - _Boundary: raft.c_
  - _Depends: 4.1_

## 5. UpdateTerm-on-AppendEntries (raft.tla:406, closes G5)

- [x] 5.1 RED: higher-term AE/heartbeat adopts term before logOk
  - Test: a follower and a leader at term T receiving an AE (and separately a 0-entry heartbeat) at term T+1 adopt T+1, clear `voted_for`, call `on_persist(T+1,-1)`, and (was-leader case) fire `on_become_follower`; assert this happens even when the local log would fail `logOk`
  - Observable completion: tests compile and FAIL
  - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
  - _Boundary: test_raft.c_
  - _Depends: 1.3_

- [x] 5.2 GREEN: term-adoption block at the top of the request handler
  - Implement the `UpdateTerm` block as the FIRST step of `raft_on_append_entries_request`, before `logOk`; persist-before-process; fire `on_become_follower` if was leader; no change on equal/lower term
  - Observable completion: 5.1 tests pass; the block is unmistakably before `logOk` in `raft.c`
  - _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
  - _Boundary: raft.c_
  - _Depends: 5.1_

## 6. HandleAppendEntriesRequest — logOk / reject / step-down / accept (raft.tla:327, closes G2/G3)

- [x] 6.1 RED: logOk + reject + candidate step-down tests
  - Test: `logOk` true when `prevLogIndex==0` or `on_get_entry_term(prevLogIndex)==prevLogTerm`; reject (success=0, matchIndex=0) on stale term and on `follower && !logOk`; candidate at equal term transitions to follower
  - Observable completion: tests compile and FAIL
  - _Requirements: 4.1, 4.2, 4.3_
  - _Boundary: test_raft.c_
  - _Depends: 5.2_

- [x] 6.2 RED: accept branch tests (already-have / conflict-truncate / append)
  - Test: already-have (empty entries or matching term) → success, no log rewrite, `matchIndex = prevLogIndex + entries_len`; conflict (`local[index].term != entry.term`) → `on_truncate_from(index)` then `on_persist_entry(index, entry)`, success `matchIndex=index`; append (`log_len == prevLogIndex`) → `on_persist_entry(index)`, success
  - Observable completion: tests compile and FAIL
  - _Requirements: 4.4, 4.5, 4.6, 4.7_
  - _Boundary: test_raft.c_
  - _Depends: 6.1_

- [x] 6.3 GREEN: implement `raft_on_append_entries_request` (logOk + all branches)
  - Implement (after the 5.2 UpdateTerm block): `logOk` via `on_get_entry_term`; reject/candidate-step-down/accept; the three accept sub-cases exactly per raft.tla:351-388; fill `raft_append_result_t`; never advance commit (`leader_commit` ignored); never perform transport I/O
  - Observable completion: 6.1 and 6.2 tests pass; `on_truncate_from` is always followed by re-persist of the leader entry on the conflict branch
  - _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7_
  - _Boundary: raft.c_
  - _Depends: 6.2_

## 7. HandleAppendEntriesResponse — raft_on_append_entries_reply (raft.tla:393)

- [x] 7.1 RED: reply advance/decrement/stale-drop tests
  - Test: success sets `next_index[peer]=matchIndex+1`, `match_index[peer]=reply.matchIndex`; failure sets `next_index[peer]=Max(next_index[peer]-1,1)` (floor at 1), `match_index` unchanged; a reply with `reply_term != currentTerm` is ignored
  - Observable completion: tests compile and FAIL
  - _Requirements: 5.1, 5.2, 5.3_
  - _Boundary: test_raft.c_
  - _Depends: 2.2_

- [x] 7.2 GREEN: implement `raft_on_append_entries_reply`
  - Stale-term drop; success/failure branches; `next_index` floored at 1; do NOT recompute or advance any commit index from `match_index`
  - Observable completion: 7.1 tests pass
  - _Requirements: 5.1, 5.2, 5.3, 5.4_
  - _Boundary: raft.c_
  - _Depends: 7.1_

## 8. Simulator backing: in-memory log + AppendEntries routing

- [x] 8.1 Add an in-memory per-node log and wire the four callbacks in `raft_sim.c`
  - NOTE (impl 2026-06-21): pulled forward and landed together with 2.1 — task 2.1's stricter `raft_init` (four callbacks now REQUIRED) makes `raft_sim_init` return -EINVAL, so ~15 `test_sim_*`/`test_proto_*` tests go red the instant 2.1 lands. 8.1's callback wiring only needs the header signatures (not 6.3/7.2), so it compiles standalone; co-committing 2.1+8.1 keeps the suite green at every commit. AE delivery (8.2) still deferred.
  - Add a per-`raft_sim_node_t` in-memory entry array (`{term, bytes}` per index); implement `on_persist_entry` (append/overwrite at index), `on_get_entry_term` (term or 0), `on_truncate_from` (drop index..end), and buffer `on_send_append_entries` into a new AE RPC queue
  - Observable completion: a sim node can append, read a term back, and truncate deterministically; `raft_sim_init` wires the four callbacks for every node
  - _Requirements: 8.2, 7.5_
  - _Boundary: raft_sim.c_
  - _Depends: 6.3, 7.2_

- [x] 8.2 Add synchronous AppendEntries delivery honoring partitions
  - NOTE (impl 2026-06-21): `raft_sim_ae_t` now carries `leader_term` captured at SEND time (in `sim_send_append_entries`) and delivery uses `ae->leader_term`, NOT the live source `state.term`. Required so reorder (9.3) and stale-term churn (9.4) deliver the sending-epoch term — using the live term would bypass UpdateTerm/stale-drop and defeat the G5 regression. Caught in review.
  - Add `raft_sim_deliver_append_entries`: for each buffered AE, call `raft_on_append_entries_request` on the (non-partitioned) destination, then route the reply back via `raft_on_append_entries_reply`; skip partitioned nodes (mirrors the existing vote-RPC routing)
  - Observable completion: driving one append + delivery replicates the entry to a connected follower; partitioned follower receives nothing
  - _Requirements: 8.1_
  - _Boundary: raft_sim.c_
  - _Depends: 8.1_

- [x] 8.3 Add a Log Matching assertion helper
  - Add a helper asserting: for any two nodes and any index `i`, if both hold an entry at `i` with equal term, their prefixes through `i` are byte-identical
  - Observable completion: the helper returns pass on a converged cluster and fails on a hand-divergent one (self-test)
  - _Requirements: 8.4_
  - _Boundary: raft_sim.c_
  - _Depends: 8.1_

## 9. Simulator invariant tests (Log Matching, truncation, faults, G5 regression)

- [x] 9.1 Replication + partition catch-up Log Matching test
  - Test: leader appends 3 entries, replicates to a quorum, assert Log Matching; then isolate a follower across more appends, heal, assert `nextIndex` walks back, the follower catches up, and Log Matching holds
  - Observable completion: test passes end-to-end via the sim
  - _Requirements: 8.1, 8.3, 8.4_
  - _Boundary: test_raft.c, raft_sim.c_
  - _Depends: 8.2, 8.3_

- [x] 9.2 Divergent-suffix truncation test (G3)
  - Test: give a follower a manufactured divergent tail (same indices, different term), replicate from a higher-term leader; assert the follower truncates the divergent suffix (`on_truncate_from`) and converges to the leader's prefix; Log Matching holds
  - Observable completion: test passes; follower's post-converge prefix equals the leader's
  - _Requirements: 8.5_
  - _Boundary: test_raft.c, raft_sim.c_
  - _Depends: 9.1_

- [x] 9.3 Message drop/dup/reorder convergence test
  - Test: drop, duplicate, and reorder AE requests/replies in the sim queue; assert idempotent convergence (a duplicated AE is a no-op via the already-have branch) and Log Matching after faults clear
  - Observable completion: test passes despite injected faults
  - _Requirements: 8.3_
  - _Boundary: test_raft.c, raft_sim.c_
  - _Depends: 9.1_

- [x] 9.4 Chaotic-churn G5 regression test
  - Test: a stale-term node (term lagging) rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does NOT re-elect at a reused term; Log Matching holds across the churn
  - Observable completion: test passes; the rejoining node's term equals the leader's after the first heartbeat, and no second leader appears at the reused term
  - _Requirements: 8.6_
  - _Boundary: test_raft.c, raft_sim.c_
  - _Depends: 9.1_

## 10. Verification

- [x] 10.1 Full library suite green under ASAN
  - Run `/xce-build lib`; all existing election tests plus the new log-replication and simulator tests pass under ASAN with zero leaks
  - Observable completion: test binary exits 0; ASAN clean; no regression in the election-core tests
  - _Requirements: 1.1, 2.1, 3.2, 4.3, 5.1, 6.1, 7.1, 8.4_
  - _Boundary: src/lib/raft, src/lib/testsuite_
  - _Depends: 9.2, 9.3, 9.4_

## Implementation Notes

- **Validation (2026-06-21): GO.** Clean ASAN build green from scratch (`xce-build.sh lib`, Fail 0); 8/8 requirement sections covered (40/41 acceptance criteria); boundary ALIGNED (no controller files; no commit advance / vote freshness / transport I/O / entry-bytes in library state); out-of-scope items confirmed absent.
- **Req 8.3 "node restart while replication proceeds" — CLOSED (2026-06-21):** added `raft_sim_restart_node` (durable log[]/(term,voted_for) survive; volatile consensus state cleared via raft_init → FOLLOWER) + `test_sim_9_5_node_restart_resumes_replication` (restart a follower mid-replication with a partial log; assert log_count preserved — load-bearing: would fail if the log were wiped — role/term restored, replication resumes, Log Matching holds). drop/dup/reorder remain covered by 9.3. All 8/8 sections now fully covered.
- **Adapter invariant (from validation LOW note):** the AE accept already-have branch treats `on_get_entry_term(index) == entry_term` as a match, which is also true when both are 0. A correct leader never sends `entry_term == 0` for a real entry (`raft_log_append` stamps `s->term >= 1`), so this is safe in-library — but the controller adapter (raft-commit-safety) MUST guarantee it never delivers `entry_term == 0` with `entries_len == 1`.