myxce/.kiro/specs/raft-log-replication/tasks.md

Implementation Plan

TDD-mandatory (testing.md): every implementation sub-task is preceded by a RED test in src/lib/testsuite/test_raft.c (and the simulator backing in raft_sim.c) that fails before the raft.c change and passes after. Build/run library tests via /xce-build lib.

Spec boundary (applies to ALL tasks): this spec owns the src/lib/raft log additions (per-entry term in the library model, ClientRequest/AppendEntries/Handle*/UpdateTerm-on-AE) and the (term, index) reconcile seam (the four callbacks). It does NOT own commit-index advance, (lastLogTerm,lastLogIndex) vote freshness, config markers/upgrade snapshot, or the controller's WAL/repl/recovery internals — those are raft-commit-safety and ha-term-safety. No task here edits any controller file; the controller adapter is described, not built.

1. Header contract: log model, callbacks, entry points

• 1.1 Extend raft.h with the log model state and config restore field

  • Add uint64_t log_len;, uint64_t next_index[RAFT_MAX_PEERS];, uint64_t match_index[RAFT_MAX_PEERS]; to raft_state_t
  • Add uint64_t initial_log_len; to raft_config_t (restored WAL length at boot, default 0; library never reads storage)
  • Observable completion: raft.h compiles; sizeof(raft_state_t) grows by the new fixed fields; no entry-byte storage added
  • Requirements: 1.1, 1.2, 3.1
  • Boundary: raft.h

• 1.2 Add the four new REQUIRED callbacks to raft_callbacks_t (verbatim contract)

  • Declare on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from with the exact signatures from design.md / brief
  • Document each as REQUIRED and the on_get_entry_term → 0-means-absent convention
  • Observable completion: the four function-pointer members exist in raft_callbacks_t with the verbatim signatures; header doc comments match the Authoritative shared contract
  • Requirements: 7.1, 7.2, 7.3, 7.4
  • Boundary: raft.h
  • Depends: 1.1

• 1.3 Declare the three new entry points and raft_append_result_t

  • NOTE (impl 2026-06-21): on_send_append_entries's design "entries handle ..." varargs placeholder was concretized to an explicit int n_entries (0=heartbeat, 1=one entry at prev_log_index+1; adapter sources the bytes since the library holds none). design.md §"Seam — raft_callbacks_t additions" updated to match. Downstream raft-commit-safety / raft-sim-conformance consume this concrete shape.
  • Declare raft_append_result_t; declare raft_log_append, raft_send_append_entries, raft_on_append_entries_request, raft_on_append_entries_reply with the design.md signatures
  • Observable completion: header declares all new entry points; a stub TU including raft.h compiles
  • Requirements: 2.1, 3.3, 4.7, 5.1
  • Boundary: raft.h
  • Depends: 1.2

2. Init validation and leader-progress initialization

• 2.1 RED+GREEN: raft_init requires the four new callbacks

  • Test: raft_init returns -EINVAL when any of the four new callbacks is NULL; returns 0 when all present
  • Implement the NULL checks in raft_init (mirroring the existing required-callback checks); restore s->log_len = cfg->initial_log_len
  • Observable completion: new -EINVAL test cases pass; existing election init tests still green
  • Requirements: 7.1, 7.2, 7.3, 7.4, 1.1
  • Boundary: raft.c, test_raft.c
  • Depends: 1.3

• 2.2 RED+GREEN: leader progress initialized on becoming leader

  • Test: after a node becomes leader, next_index[j] == log_len + 1 and match_index[j] == 0 for every peer (raft.tla BecomeLeader)
  • Implement the next_index/match_index initialization on the win path (raft_start_election single-node + the multi-node become-leader transition)
  • Observable completion: assertion on next_index/match_index after election passes; no regression in election tests
  • Requirements: 3.1
  • Boundary: raft.c, test_raft.c
  • Depends: 2.1

3. ClientRequest — raft_log_append (raft.tla:246)

• 3.1 RED: append tests (leader stamps term, follower rejected)

  • Test: leader raft_log_append assigns index log_len+1, stamps currentTerm, calls on_persist_entry exactly once with (index, term, entry), returns the index; follower call returns -EPERM with no log change and no callback
  • Observable completion: tests compile and FAIL (no implementation yet)
  • Requirements: 2.1, 2.2, 2.3, 2.4
  • Boundary: test_raft.c
  • Depends: 2.2

• 3.2 GREEN: implement raft_log_append

  • Role guard (-EPERM if not leader); persist-before-act (on_persist_entry before incrementing log_len); set *out_index; do NOT advance any commit index
  • Observable completion: 3.1 tests pass; log_len only advances on a successful persist (no phantom entry on persist failure)
  • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5
  • Boundary: raft.c
  • Depends: 3.1

4. AppendEntries emission — raft_send_append_entries (raft.tla:204)

• 4.1 RED: emit tests (prevLogIndex/prevLogTerm, ≤1 entry, leader-only)

  • Test: for peer j, emit computes prevLogIndex = next_index[j]-1 and prevLogTerm via on_get_entry_term (0 when prevLogIndex==0), sends ≤1 entry, passes leader_commit through opaquely; a non-leader emits nothing
  • Observable completion: tests compile and FAIL
  • Requirements: 3.2, 3.3, 3.4, 3.5
  • Boundary: test_raft.c
  • Depends: 2.2

• 4.2 GREEN: implement raft_send_append_entries

  • Compute prev index/term; lastEntry = Min(log_len, next_index[j]); one entry or heartbeat; call on_send_append_entries with the opaque leader_commit; leader-only guard
  • Observable completion: 4.1 tests pass; heartbeat (0 entries) emitted when next_index[j] > log_len
  • Requirements: 3.2, 3.3, 3.4, 3.5
  • Boundary: raft.c
  • Depends: 4.1

5. UpdateTerm-on-AppendEntries (raft.tla:406, closes G5)

• 5.1 RED: higher-term AE/heartbeat adopts term before logOk

  • Test: a follower and a leader at term T receiving an AE (and separately a 0-entry heartbeat) at term T+1 adopt T+1, clear voted_for, call on_persist(T+1,-1), and (was-leader case) fire on_become_follower; assert this happens even when the local log would fail logOk
  • Observable completion: tests compile and FAIL
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5
  • Boundary: test_raft.c
  • Depends: 1.3

• 5.2 GREEN: term-adoption block at the top of the request handler

  • Implement the UpdateTerm block as the FIRST step of raft_on_append_entries_request, before logOk; persist-before-process; fire on_become_follower if was leader; no change on equal/lower term
  • Observable completion: 5.1 tests pass; the block is unmistakably before logOk in raft.c
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5
  • Boundary: raft.c
  • Depends: 5.1

6. HandleAppendEntriesRequest — logOk / reject / step-down / accept (raft.tla:327, closes G2/G3)

• 6.1 RED: logOk + reject + candidate step-down tests

  • Test: logOk true when prevLogIndex==0 or on_get_entry_term(prevLogIndex)==prevLogTerm; reject (success=0, matchIndex=0) on stale term and on follower && !logOk; candidate at equal term transitions to follower
  • Observable completion: tests compile and FAIL
  • Requirements: 4.1, 4.2, 4.3
  • Boundary: test_raft.c
  • Depends: 5.2

• 6.2 RED: accept branch tests (already-have / conflict-truncate / append)

  • Test: already-have (empty entries or matching term) → success, no log rewrite, matchIndex = prevLogIndex + entries_len; conflict (local[index].term != entry.term) → on_truncate_from(index) then on_persist_entry(index, entry), success matchIndex=index; append (log_len == prevLogIndex) → on_persist_entry(index), success
  • Observable completion: tests compile and FAIL
  • Requirements: 4.4, 4.5, 4.6, 4.7
  • Boundary: test_raft.c
  • Depends: 6.1

• 6.3 GREEN: implement raft_on_append_entries_request (logOk + all branches)

  • Implement (after the 5.2 UpdateTerm block): logOk via on_get_entry_term; reject/candidate-step-down/accept; the three accept sub-cases exactly per raft.tla:351-388; fill raft_append_result_t; never advance commit (leader_commit ignored); never perform transport I/O
  • Observable completion: 6.1 and 6.2 tests pass; on_truncate_from is always followed by re-persist of the leader entry on the conflict branch
  • Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7
  • Boundary: raft.c
  • Depends: 6.2

7. HandleAppendEntriesResponse — raft_on_append_entries_reply (raft.tla:393)

• 7.1 RED: reply advance/decrement/stale-drop tests

  • Test: success sets next_index[peer]=matchIndex+1, match_index[peer]=reply.matchIndex; failure sets next_index[peer]=Max(next_index[peer]-1,1) (floor at 1), match_index unchanged; a reply with reply_term != currentTerm is ignored
  • Observable completion: tests compile and FAIL
  • Requirements: 5.1, 5.2, 5.3
  • Boundary: test_raft.c
  • Depends: 2.2

• 7.2 GREEN: implement raft_on_append_entries_reply

  • Stale-term drop; success/failure branches; next_index floored at 1; do NOT recompute or advance any commit index from match_index
  • Observable completion: 7.1 tests pass
  • Requirements: 5.1, 5.2, 5.3, 5.4
  • Boundary: raft.c
  • Depends: 7.1

8. Simulator backing: in-memory log + AppendEntries routing

• 8.1 Add an in-memory per-node log and wire the four callbacks in raft_sim.c

  • NOTE (impl 2026-06-21): pulled forward and landed together with 2.1 — task 2.1's stricter raft_init (four callbacks now REQUIRED) makes raft_sim_init return -EINVAL, so ~15 test_sim_*/test_proto_* tests go red the instant 2.1 lands. 8.1's callback wiring only needs the header signatures (not 6.3/7.2), so it compiles standalone; co-committing 2.1+8.1 keeps the suite green at every commit. AE delivery (8.2) still deferred.
  • Add a per-raft_sim_node_t in-memory entry array ({term, bytes} per index); implement on_persist_entry (append/overwrite at index), on_get_entry_term (term or 0), on_truncate_from (drop index..end), and buffer on_send_append_entries into a new AE RPC queue
  • Observable completion: a sim node can append, read a term back, and truncate deterministically; raft_sim_init wires the four callbacks for every node
  • Requirements: 8.2, 7.5
  • Boundary: raft_sim.c
  • Depends: 6.3, 7.2

• 8.2 Add synchronous AppendEntries delivery honoring partitions

  • Add raft_sim_deliver_append_entries: for each buffered AE, call raft_on_append_entries_request on the (non-partitioned) destination, then route the reply back via raft_on_append_entries_reply; skip partitioned nodes (mirrors the existing vote-RPC routing)
  • Observable completion: driving one append + delivery replicates the entry to a connected follower; partitioned follower receives nothing
  • Requirements: 8.1
  • Boundary: raft_sim.c
  • Depends: 8.1

• 8.3 Add a Log Matching assertion helper

  • Add a helper asserting: for any two nodes and any index i, if both hold an entry at i with equal term, their prefixes through i are byte-identical
  • Observable completion: the helper returns pass on a converged cluster and fails on a hand-divergent one (self-test)
  • Requirements: 8.4
  • Boundary: raft_sim.c
  • Depends: 8.1

9. Simulator invariant tests (Log Matching, truncation, faults, G5 regression)

• 9.1 Replication + partition catch-up Log Matching test

  • Test: leader appends 3 entries, replicates to a quorum, assert Log Matching; then isolate a follower across more appends, heal, assert nextIndex walks back, the follower catches up, and Log Matching holds
  • Observable completion: test passes end-to-end via the sim
  • Requirements: 8.1, 8.3, 8.4
  • Boundary: test_raft.c, raft_sim.c
  • Depends: 8.2, 8.3

• 9.2 Divergent-suffix truncation test (G3)

  • Test: give a follower a manufactured divergent tail (same indices, different term), replicate from a higher-term leader; assert the follower truncates the divergent suffix (on_truncate_from) and converges to the leader's prefix; Log Matching holds
  • Observable completion: test passes; follower's post-converge prefix equals the leader's
  • Requirements: 8.5
  • Boundary: test_raft.c, raft_sim.c
  • Depends: 9.1

• 9.3 Message drop/dup/reorder convergence test

  • Test: drop, duplicate, and reorder AE requests/replies in the sim queue; assert idempotent convergence (a duplicated AE is a no-op via the already-have branch) and Log Matching after faults clear
  • Observable completion: test passes despite injected faults
  • Requirements: 8.3
  • Boundary: test_raft.c, raft_sim.c
  • Depends: 9.1

• 9.4 Chaotic-churn G5 regression test

  • Test: a stale-term node (term lagging) rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does NOT re-elect at a reused term; Log Matching holds across the churn
  • Observable completion: test passes; the rejoining node's term equals the leader's after the first heartbeat, and no second leader appears at the reused term
  • Requirements: 8.6
  • Boundary: test_raft.c, raft_sim.c
  • Depends: 9.1

10. Verification

• 10.1 Full library suite green under ASAN
  • Run /xce-build lib; all existing election tests plus the new log-replication and simulator tests pass under ASAN with zero leaks
  • Observable completion: test binary exits 0; ASAN clean; no regression in the election-core tests
  • Requirements: 1.1, 2.1, 3.2, 4.3, 5.1, 6.1, 7.1, 8.4
  • Boundary: src/lib/raft, src/lib/testsuite
  • Depends: 9.2, 9.3, 9.4