checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions

View file

@ -688,6 +688,23 @@ int persist_wal_write_replicated(persist_t* p, uint64_t lsn, uint32_t type,
return 0;
}
/* Contiguity guard (bug-850): REFUSE a forward GAP. current_lsn is the
* next-expected LSN, so a replicated entry with lsn > current_lsn would
* leave [current_lsn .. lsn-1] missing AND advance current_lsn past that
* hole — after which the idempotency guard above silently DROPS the later
* in-order catch-up of the prefix (the failover prefix-loss mechanism).
* Append only the contiguous next entry (lsn == current_lsn); reject a gap
* without writing or acking so the ACK-driven catch-up continuation
* re-streams from current_lsn and the log fills in order. This is Raft
* log-matching: append entry N only when the log already holds N-1. A
* behind standby that keeps receiving live broadcasts simply drops the
* out-of-order ones until catch-up backfills the gap (no loss). -EAGAIN is
* the "retry via catch-up" signal the apply path treats as a quiet skip. */
if (lsn > p->current_lsn) {
pthread_mutex_unlock(&p->lock);
return -EAGAIN;
}
/* Open first segment lazily. open_new_segment names the file from
* current_lsn — for a fresh standby receiving lsn 1 first, current_lsn is
* 1 (reserve-lsn-0), so the segment is seg-...001.wal. */