Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
41 lines
5.1 KiB
Markdown
41 lines
5.1 KiB
Markdown
# Brief: raft-log-replication
|
||
|
||
## Problem
|
||
`src/lib/raft/raft.{c,h}` is an **election-only** Raft library (terms, votes, leader election, PreVote) with a clean callback interface and a deterministic simulator (`raft_sim.c`). It has **no log, no AppendEntries, no commit**. The replicated log is hand-rolled in the controller (`persist.c` WAL, `repl.c` catch-up streaming, `ha.c`, `ha_convergence.c`) and **violates the Raft log invariants of `raft/raft.tla`**, which root-caused **bug-851** (silent committed-prefix divergence under chaotic failover). Authoritative gap analysis: `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`.
|
||
|
||
This spec closes the four **log-half** gaps by making the library own the log, strictly following `raft.tla`:
|
||
- **G1** — per-entry term is dead (WAL `hdr.term` always 0; `p->term` never set from `ha->term`).
|
||
- **G2** — replication matches LSN contiguity only; no `prevLogTerm` check (TLA+ `AppendEntries`/`HandleAppendEntriesRequest` `logOk`).
|
||
- **G3** — no `(term,lsn)` conflict truncation (TLA+ conflict branch: delete the divergent entry + all after).
|
||
- **G5** — a node adopts a higher term only on **vote** RPCs; the AppendEntries/heartbeat path never advances `currentTerm` (TLA+ `UpdateTerm` fires on **any** higher-term RPC). Confirmed live: a rejoined standby reported `term:2` under a term-3 primary → term reuse → two leaders per term.
|
||
|
||
## Approach
|
||
Extend the library to own the replicated log, **strictly mirroring the `raft.tla` actions**, keeping the core **pure / synchronous / deterministic** (no I/O, no threads) so it stays simulator-verifiable. All side effects go through new callbacks the controller adapter implements over the existing WAL/repl transport.
|
||
|
||
### TLA+ actions implemented (raft.tla line refs)
|
||
- `ClientRequest(i,v)` (246) → `raft_log_append(...)`: append `[term ↦ currentTerm, value]`.
|
||
- `AppendEntries(i,j)` (204) → emit via `on_send_append_entries` with `prevLogIndex`, `prevLogTerm`, ≤1+ entries, `leaderCommit`. Per-peer `nextIndex`/`matchIndex` (TLA+ leaderVars).
|
||
- `HandleAppendEntriesRequest(i,j,m)` (327) → `raft_on_append_entries_request(...)`: `logOk` = `prevLogIndex==0 ∨ log[prevLogIndex].term==prevLogTerm`; reject (stale term / `¬logOk`), candidate→follower on equal term, accept (already-have / conflict-truncate-one / append).
|
||
- `HandleAppendEntriesResponse(i,j,m)` (393) → `raft_on_append_entries_reply(...)`: success → advance `nextIndex`/`matchIndex`; fail → decrement `nextIndex`.
|
||
- `UpdateTerm(i,j,m)` (406) → applied on **every** incoming RPC including AppendEntries/heartbeat (closes G5): higher term ⇒ adopt term, clear vote, step down to follower.
|
||
|
||
### Authoritative shared contract (downstream specs MUST reference this verbatim)
|
||
Log index model: the library log index is 1-based (TLA+) and maps 1:1 to the controller WAL **LSN**. Per-entry `term` is `uint64_t`. New REQUIRED callbacks (pure; controller implements I/O):
|
||
- `int on_persist_entry(void *ctx, uint64_t index, uint64_t term, const void *entry, size_t len)` — durably append/overwrite at index.
|
||
- `uint64_t on_get_entry_term(void *ctx, uint64_t index)` — term of local entry at index (0 if absent).
|
||
- `int on_send_append_entries(void *ctx, int peer_idx, uint64_t prev_log_index, uint64_t prev_log_term, uint64_t leader_commit, /* entries handle */ ...)` — transmit.
|
||
- `int on_truncate_from(void *ctx, uint64_t index)` — delete local entries at `index..end`.
|
||
New library entry points: `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`. Existing election callbacks/behaviour unchanged.
|
||
|
||
## Scope
|
||
- **In**: library log model with per-entry term; `ClientRequest` append; AppendEntries send (per-peer next/match index); `HandleAppendEntriesRequest` accept/reject/conflict-truncate/append by `(term,index)`; `UpdateTerm` on every RPC; controller adapter that maps callbacks onto the existing WAL/repl transport (WAL `hdr.term` stamped from `ha->term`, carried in the repl envelope, read back on recovery); simulator coverage asserting **Log Matching**.
|
||
- **Out**: `AdvanceCommitIndex` / commit-index safety + `(term,lsn)` vote freshness → `raft-commit-safety`. The full conformance harness → `raft-sim-conformance`. Config-change markers + upgrade snapshot → `ha-term-safety`. Async I/O, fan-out scheduling, the sync commit barrier, snapshot transport → controller callback adapters (not the pure core).
|
||
|
||
## Preserve as policy (do NOT canonicalize away)
|
||
PreVote, priority leadership, persist-before-ACK durability + synchronous commit barrier (consumed in `raft-commit-safety`), "controller never self-terminates" (library returns statuses; controller owns liveness/fencing).
|
||
|
||
## Verification
|
||
Extend `raft_sim.c`: drive AppendEntries across partitions, message drop/dup/reorder, and restarts; assert the **Log Matching** invariant (same `(index,term)` ⇒ identical prefix) and that a follower truncates a divergent suffix. Replace the controller's LSN-only reconcile with `(term,index)` matching delegated to the library.
|
||
|
||
## Dependencies
|
||
`raft-library` (election core, implemented `[x]`). Feeds `raft-commit-safety`, `raft-sim-conformance`, and `ha-term-safety` integration. Closes bug-851 G1/G2/G3/G5.
|