checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions

View file

@ -317,6 +317,31 @@ skip_election:;
if (role == CTRL_STANDBY || role == CTRL_INIT) {
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_STANDBY,
memory_order_release);
/* bug-849: a follower that REJOINS an existing cluster skips its own
* election (a primary already exists), so the election-result path
* that calls cluster_init_phase2 — which builds resources, runs
* persist_recover, and (bug-848) advertises the recovered
* applied_lsn — NEVER runs on this path. The node then comes up on
* an un-recovered WAL (current_lsn=1, applied_lsn=0), the primary
* re-streams the whole log over its consistent records, and the
* committed prefix is lost. Run phase2 HERE too (idempotent via
* phase2_done, same once-per-process contract as the election
* callers) so EVERY boot path recovers its own WAL and advertises
* its real position BEFORE cluster_run starts accepting replication.
* Recovery is decoupled from the primary's config push: a restarted
* node already loaded its config from disk pre-election. */
if (!c.ha.phase2_done) {
int prc = cluster_init_phase2(&c);
if (prc != 0) {
WARN("myapp_run: cluster_init_phase2 on standby rejoin "
"failed (rc=%d) — WAL not recovered; falling back to "
"election-path recovery", prc);
} else {
c.ha.phase2_done = 1;
}
}
INFO("myapp_run: CTRL_STANDBY — waiting for primary to connect and push config");
/* Arm heartbeat watchdog even when we skipped the election.
* If the known primary never connects within heartbeat_timeout_ms,