checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions

View file

@ -440,8 +440,23 @@ static void test_fact_hook_append_is_retained_and_nonblocking(void)
"the uncommitted fact must be pending (lsn > committed_lsn), "
"never confirmed early");
/* ...and a covering ACK later commits the retained entry — engine
* accounting advances committed_lsn over it, no redelivery needed */
/* ha-commit-durability 2.2: the primary's self-contribution to the commit
* majority is now capped at durable_lsn, which the demand-driven
* group-commit flusher advances asynchronously after fdatasync. Force the
* fact durable here (synchronous fdatasync) so the covering ACK below
* deterministically completes the majority (durable self + standby ACK);
* a bare synchronous read would otherwise race the flusher and observe
* durable_lsn == 0 → self-cap 0 → committed_lsn unchanged. This mirrors the
* durable_lsn priming done for the in-memory stub fixtures in
* barrier_setup_vcpus. */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, persist_flush(g_fp),
"fact must fdatasync (durable) before the covering ACK");
TEST_ASSERT_TRUE_MESSAGE(persist_durable_lsn(g_fp) >= fact_lsn,
"durable_lsn must cover the fact after persist_flush");
/* ...and a covering ACK then commits the now-durable retained entry —
* repl_on_ack drives commit_on_ack synchronously, advancing committed_lsn
* over it, no redelivery needed */
repl_on_ack(&g_fp->repl, &g_fep_a, fact_lsn);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
fact_lsn, atomic_load(&g_fp->committed_lsn),