myxce/.kiro/specs/ha-term-safety/TLA-gap-analysis.md

Gap Analysis: XCE HA implementation vs raft/raft.tla

Reference spec: raft/raft.tla (Diego Ongaro's canonical Raft TLA⁺, 471 lines). Implementation: src/lib/raft/raft.c (election library) + the controller log/replication/commit layer (persist.c, repl.c, ha.c, ha_convergence.c, config_repl.c). Date: 2026-06-21. Grounds the ha-term-safety spec and bug-851.

Headline

raft.c implements only the election half of the TLA⁺ spec — currentTerm, votedFor, state, RequestVote, BecomeLeader, UpdateTerm (on vote RPCs), Timeout. It has no log, no AppendEntries, no AdvanceCommitIndex, no commitIndex. The replicated log and its replication/commit/reconcile are implemented separately in the controller (WAL = log, catch-up streaming = AppendEntries, commit barrier = AdvanceCommitIndex, ha_convergence.c = conflict reconcile) and do not follow the TLA⁺ log protocol. Every bug-851 defect lives in that split: the election module is faithful, but the log layer omits the term-coupling the TLA⁺ spec uses to guarantee Log Matching and Leader Completeness.

Action-by-action conformance

TLA⁺ action / invariant	Implementation site	Status
Timeout (term++, clear vote, self-vote)	raft_start_election (raft.c:61) — s->term++, voted_for=self, persist before RPC	Conformant
RequestVote freshness logOk = mlastLogTerm > LastTerm ∨ (= ∧ mlastLogIndex ≥ Len)	on_vote_eligible gate (raft.c:171) → controller LSN floor	GAP G6 — freshness is LSN-only; never compares lastLogTerm (the WAL has no usable term)
HandleRequestVoteRequest grant rules (≤ one vote/term, persist)	raft_on_vote_request (raft.c:110): stale-term deny, dup-vote deny, persist-before-grant	Conformant (Election Safety primitives correct)
BecomeLeader (votesGranted ∈ Quorum)	raft_on_vote_reply step 5 (raft.c:373) votes_granted ≥ n/2+1	Conformant
UpdateTerm — any RPC with higher term advances currentTerm, steps down	raft_on_vote_request step 2 + raft_on_vote_reply step 1 (vote RPCs only)	GAP G5 — term adoption is wired to vote RPCs only; the AppendEntries/heartbeat-equivalent path does not route the leader's term into raft state
ClientRequest — append [term ↦ currentTerm, value]	persist_wal_write* (persist.c) — hdr.term = p->term, p->term never set	GAP G1 — entry term is always 0; the log carries no term
AppendEntries / HandleAppendEntriesRequest logOk = mprevLogTerm = log[prevLogIndex].term	WAL catch-up stream + persist_wal_write_replicated contiguity guard (bug-850)	GAP G2 — replication matches LSN contiguity only; no prevLogTerm check
HandleAppendEntriesRequest conflict branch — delete entry whose term ≠ leader's + all after	ha_convergence.c rejoin reconcile (truncates above committed boundary)	GAP G3 — truncates by committed-boundary heuristic, not by (term,lsn) conflict; cannot detect same-LSN/different-term divergence
AdvanceCommitIndex — commit Max(agreeIndexes) only if log[Max].term = currentTerm (Figure-8 safety)	commit barrier + ha_compute_safe_committed_lsn	GAP G4 — current-term check is unenforceable (term=0); the own-term WAL_NOOP gate is a partial mitigation, but the safe-committed computation itself is term-blind
Restart — keep currentTerm, votedFor, log	recover WAL + ha-term.bin; raft_init restores (term, voted_for, voted_for_term)	Mostly conformant, but feeds G5: a restarted node's raft term is only as fresh as what reached ha-term.bin; with G5 it never advanced from heartbeats, so it recovers stale
Election Safety (≤ one leader/term)	emergent from the above	VIOLATED in practice — see below
Log Matching (same idx+term ⇒ identical prefix)	emergent from the log layer	VIOLATED (bug-851) — G1/G2/G3
Leader Completeness (committed entry survives in all future leaders)	emergent	VIOLATED (bug-851) — G4 commits term-blind
State Machine Safety (same idx applied ⇒ same command)	emergent	VIOLATED (bug-851 repro: divergent jobs at same committed LSN)

Extensions beyond the TLA⁺ spec (not gaps): PreVote (raft_begin_pre_vote, disruption avoidance), single-node short-circuit, even-cluster-size rejection, priority/jitter staggering.

How the gaps compose into bug-851 (observed)

1. G5 — a standby never adopts the leader's term from heartbeats/replication (confirmed live: rejoined ctrl-1 reported term: 2 while serving under a term-3 primary). So its raft currentTerm lags.
2. On failover that lagged node runs raft_start_election, computing the new term as stale + 1 → it reuses a term a prior, now-dead leader already used (observed: term 3 led by ctrl-3 then ctrl-1; term 4 by ctrl-2 then ctrl-3). Election Safety is bypassed not by a double-vote in one live term, but by term-number reuse across the restart churn that the vote-only term plumbing can't see.
3. G1 — each such leader appends its own-term no-op + jobs with hdr.term = 0 at overlapping LSNs (both wrote lsn=45).
4. G2/G3 — replication/reconcile compare by LSN only, so (term=3, lsn=45) and (term=4, lsn=45) look identical (both type+term 0) and the divergent suffix is never truncated.
5. G4 — committed advances term-blind "from 1 granted voter" over entries a majority never held.

Result: all members report committedLsn=54 over divergent content at LSN 45–54 — a silent committed-prefix divergence (State Machine Safety violation).

Configuration is a second, parallel violation

Config commits through a separate index (committed_config_tag) with no term and tag-based anti-entropy (config_repl.c) — it is entirely outside both the TLA⁺ log and raft.c. It is exposed to the same term-reuse divergence, plus a state-materialization hazard: a config upgrade does config-only push + restart + per-node WAL replay under the new config, with no snapshot to pin an identical post-upgrade state (the lost_and_found re-homing is the tell that replay is config-dependent). The TLA⁺ spec assumes config changes are log entries; XCE's are neither log entries nor snapshot-anchored.

Mapping gaps → ha-term-safety requirements

Gap	Requirement that closes it
G1 entry term always 0	R1 (per-entry WAL term recording)
G5 term not adopted on replication path; term reuse	R2 (monotonic, durably-established term)
G6 vote freshness LSN-only	R2.4 + R1 (term available to vote gate)
G2 no prevLogTerm match	R3.1 ((term,lsn) comparison)
G3 no conflict truncation	R3.2 (truncate divergent (term,lsn) suffix)
G4 commit term-blind	R4.1/R4.5 (commit only (term,lsn)-quorum-corroborated; own-term)
Config outside log / no term	R7 (config marker in unified log)
Config-upgrade state re-derive	R8 (full snapshot at change boundary)
All of the above	R6 (chaotic-failover regression test)

Note on scope vs raft.c

DECISION (2026-06-21): rather than patch the gaps in the controller's split architecture, the chosen direction is to give raft.c a real log and route AppendEntries/commit through it — strict raft.tla conformance. This is the Raft TLA+ Conformance series on the roadmap (raft-log-replication → raft-commit-safety → raft-sim-conformance, specs generated 2026-06-21). The library owns the protocol decisions (term on entries, UpdateTerm on every RPC, (term,index) matching+truncation, current-term commit, (lastLogTerm,lastLogIndex) freshness); the controller becomes a thin callback adapter (WAL = log, repl = transport, the sync commit barrier triggered via on_commit_advanced). ha-term-safety is reframed accordingly: its protocol invariants R1–R6 are realized by the library specs (not re-implemented in ha_convergence.c/ha_compute_safe_committed_lsn), and it retains only the controller-side wiring, the config marker + upgrade snapshot (R7/R8), and the live chaotic-failover regression. When ha-term-safety design is generated, map each requirement onto the library callbacks rather than controller-internal logic.