Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
62 lines
8.2 KiB
Markdown
62 lines
8.2 KiB
Markdown
# Gap Analysis: XCE HA implementation vs `raft/raft.tla`
|
||
|
||
Reference spec: `raft/raft.tla` (Diego Ongaro's canonical Raft TLA⁺, 471 lines).
|
||
Implementation: `src/lib/raft/raft.c` (election library) + the controller log/replication/commit layer (`persist.c`, `repl.c`, `ha.c`, `ha_convergence.c`, `config_repl.c`).
|
||
Date: 2026-06-21. Grounds the `ha-term-safety` spec and bug-851.
|
||
|
||
## Headline
|
||
|
||
`raft.c` implements **only the election half** of the TLA⁺ spec — `currentTerm`, `votedFor`, `state`, `RequestVote`, `BecomeLeader`, `UpdateTerm` (on vote RPCs), `Timeout`. It has **no `log`, no `AppendEntries`, no `AdvanceCommitIndex`, no `commitIndex`**. The replicated log and its replication/commit/reconcile are implemented **separately** in the controller (WAL = `log`, catch-up streaming = `AppendEntries`, commit barrier = `AdvanceCommitIndex`, `ha_convergence.c` = conflict reconcile) and **do not follow the TLA⁺ log protocol**. Every bug-851 defect lives in that split: the election module is faithful, but the log layer omits the term-coupling the TLA⁺ spec uses to guarantee Log Matching and Leader Completeness.
|
||
|
||
## Action-by-action conformance
|
||
|
||
| TLA⁺ action / invariant | Implementation site | Status |
|
||
|---|---|---|
|
||
| `Timeout` (term++, clear vote, self-vote) | `raft_start_election` (raft.c:61) — `s->term++`, `voted_for=self`, persist before RPC | **Conformant** |
|
||
| `RequestVote` freshness `logOk` = `mlastLogTerm > LastTerm ∨ (= ∧ mlastLogIndex ≥ Len)` | `on_vote_eligible` gate (raft.c:171) → controller LSN floor | **GAP G6** — freshness is **LSN-only**; never compares `lastLogTerm` (the WAL has no usable term) |
|
||
| `HandleRequestVoteRequest` grant rules (≤ one vote/term, persist) | `raft_on_vote_request` (raft.c:110): stale-term deny, dup-vote deny, persist-before-grant | **Conformant** (Election Safety primitives correct) |
|
||
| `BecomeLeader` (votesGranted ∈ Quorum) | `raft_on_vote_reply` step 5 (raft.c:373) `votes_granted ≥ n/2+1` | **Conformant** |
|
||
| `UpdateTerm` — **any** RPC with higher term advances `currentTerm`, steps down | `raft_on_vote_request` step 2 + `raft_on_vote_reply` step 1 (vote RPCs **only**) | **GAP G5** — term adoption is wired to **vote RPCs only**; the AppendEntries/heartbeat-equivalent path does **not** route the leader's term into raft state |
|
||
| `ClientRequest` — append `[term ↦ currentTerm, value]` | `persist_wal_write*` (persist.c) — `hdr.term = p->term`, `p->term` never set | **GAP G1** — entry term is **always 0**; the log carries no term |
|
||
| `AppendEntries` / `HandleAppendEntriesRequest` `logOk` = `mprevLogTerm = log[prevLogIndex].term` | WAL catch-up stream + `persist_wal_write_replicated` contiguity guard (bug-850) | **GAP G2** — replication matches **LSN contiguity only**; no `prevLogTerm` check |
|
||
| `HandleAppendEntriesRequest` conflict branch — delete entry whose `term ≠` leader's + all after | `ha_convergence.c` rejoin reconcile (truncates above committed boundary) | **GAP G3** — truncates by committed-boundary heuristic, **not by `(term,lsn)` conflict**; cannot detect same-LSN/different-term divergence |
|
||
| `AdvanceCommitIndex` — commit `Max(agreeIndexes)` **only if** `log[Max].term = currentTerm` (Figure-8 safety) | commit barrier + `ha_compute_safe_committed_lsn` | **GAP G4** — current-term check is unenforceable (`term=0`); the own-term `WAL_NOOP` gate is a **partial** mitigation, but the safe-committed computation itself is term-blind |
|
||
| `Restart` — keep `currentTerm`, `votedFor`, `log` | recover WAL + `ha-term.bin`; `raft_init` restores `(term, voted_for, voted_for_term)` | **Mostly conformant**, but feeds G5: a restarted node's raft term is only as fresh as what reached `ha-term.bin`; with G5 it never advanced from heartbeats, so it recovers **stale** |
|
||
| Election Safety (≤ one leader/term) | emergent from the above | **VIOLATED in practice** — see below |
|
||
| Log Matching (same idx+term ⇒ identical prefix) | emergent from the log layer | **VIOLATED** (bug-851) — G1/G2/G3 |
|
||
| Leader Completeness (committed entry survives in all future leaders) | emergent | **VIOLATED** (bug-851) — G4 commits term-blind |
|
||
| State Machine Safety (same idx applied ⇒ same command) | emergent | **VIOLATED** (bug-851 repro: divergent jobs at same committed LSN) |
|
||
|
||
Extensions beyond the TLA⁺ spec (not gaps): PreVote (`raft_begin_pre_vote`, disruption avoidance), single-node short-circuit, even-cluster-size rejection, priority/jitter staggering.
|
||
|
||
## How the gaps compose into bug-851 (observed)
|
||
|
||
1. **G5** — a standby never adopts the leader's term from heartbeats/replication (confirmed live: rejoined ctrl-1 reported `term: 2` while serving under a term-3 primary). So its raft `currentTerm` lags.
|
||
2. On failover that lagged node runs `raft_start_election`, computing the new term as **stale + 1** → it **reuses a term** a prior, now-dead leader already used (observed: term 3 led by ctrl-3 then ctrl-1; term 4 by ctrl-2 then ctrl-3). Election Safety is bypassed not by a double-vote in one live term, but by **term-number reuse across the restart churn** that the vote-only term plumbing can't see.
|
||
3. **G1** — each such leader appends its own-term no-op + jobs with `hdr.term = 0` at overlapping LSNs (both wrote `lsn=45`).
|
||
4. **G2/G3** — replication/reconcile compare by LSN only, so `(term=3, lsn=45)` and `(term=4, lsn=45)` look identical (both type+term 0) and the divergent suffix is never truncated.
|
||
5. **G4** — `committed` advances term-blind "from 1 granted voter" over entries a majority never held.
|
||
|
||
Result: all members report `committedLsn=54` over divergent content at LSN 45–54 — a silent committed-prefix divergence (State Machine Safety violation).
|
||
|
||
## Configuration is a second, parallel violation
|
||
|
||
Config commits through a **separate** index (`committed_config_tag`) with **no term** and tag-based anti-entropy (`config_repl.c`) — it is entirely outside both the TLA⁺ log and `raft.c`. It is exposed to the same term-reuse divergence, plus a state-materialization hazard: a config upgrade does config-only push + restart + **per-node WAL replay under the new config**, with no snapshot to pin an identical post-upgrade state (the `lost_and_found` re-homing is the tell that replay is config-dependent). The TLA⁺ spec assumes config changes are log entries; XCE's are neither log entries nor snapshot-anchored.
|
||
|
||
## Mapping gaps → `ha-term-safety` requirements
|
||
|
||
| Gap | Requirement that closes it |
|
||
|---|---|
|
||
| G1 entry term always 0 | R1 (per-entry WAL term recording) |
|
||
| G5 term not adopted on replication path; term reuse | R2 (monotonic, durably-established term) |
|
||
| G6 vote freshness LSN-only | R2.4 + R1 (term available to vote gate) |
|
||
| G2 no `prevLogTerm` match | R3.1 (`(term,lsn)` comparison) |
|
||
| G3 no conflict truncation | R3.2 (truncate divergent `(term,lsn)` suffix) |
|
||
| G4 commit term-blind | R4.1/R4.5 (commit only `(term,lsn)`-quorum-corroborated; own-term) |
|
||
| Config outside log / no term | R7 (config marker in unified log) |
|
||
| Config-upgrade state re-derive | R8 (full snapshot at change boundary) |
|
||
| All of the above | R6 (chaotic-failover regression test) |
|
||
|
||
## Note on scope vs `raft.c`
|
||
|
||
**DECISION (2026-06-21):** rather than patch the gaps in the controller's split architecture, the chosen direction is to **give `raft.c` a real log and route AppendEntries/commit through it — strict `raft.tla` conformance.** This is the `Raft TLA+ Conformance` series on the roadmap (`raft-log-replication` → `raft-commit-safety` → `raft-sim-conformance`, specs generated 2026-06-21). The library owns the protocol decisions (term on entries, `UpdateTerm` on every RPC, `(term,index)` matching+truncation, current-term commit, `(lastLogTerm,lastLogIndex)` freshness); the controller becomes a **thin callback adapter** (WAL = log, repl = transport, the sync commit barrier triggered via `on_commit_advanced`). `ha-term-safety` is reframed accordingly: its protocol invariants R1–R6 are **realized by the library specs** (not re-implemented in `ha_convergence.c`/`ha_compute_safe_committed_lsn`), and it retains only the controller-side wiring, the config marker + upgrade snapshot (R7/R8), and the live chaotic-failover regression. When `ha-term-safety` design is generated, map each requirement onto the library callbacks rather than controller-internal logic.
|