checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
|
|
@ -317,6 +317,31 @@ skip_election:;
|
|||
if (role == CTRL_STANDBY || role == CTRL_INIT) {
|
||||
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_STANDBY,
|
||||
memory_order_release);
|
||||
|
||||
/* bug-849: a follower that REJOINS an existing cluster skips its own
|
||||
* election (a primary already exists), so the election-result path
|
||||
* that calls cluster_init_phase2 — which builds resources, runs
|
||||
* persist_recover, and (bug-848) advertises the recovered
|
||||
* applied_lsn — NEVER runs on this path. The node then comes up on
|
||||
* an un-recovered WAL (current_lsn=1, applied_lsn=0), the primary
|
||||
* re-streams the whole log over its consistent records, and the
|
||||
* committed prefix is lost. Run phase2 HERE too (idempotent via
|
||||
* phase2_done, same once-per-process contract as the election
|
||||
* callers) so EVERY boot path recovers its own WAL and advertises
|
||||
* its real position BEFORE cluster_run starts accepting replication.
|
||||
* Recovery is decoupled from the primary's config push: a restarted
|
||||
* node already loaded its config from disk pre-election. */
|
||||
if (!c.ha.phase2_done) {
|
||||
int prc = cluster_init_phase2(&c);
|
||||
if (prc != 0) {
|
||||
WARN("myapp_run: cluster_init_phase2 on standby rejoin "
|
||||
"failed (rc=%d) — WAL not recovered; falling back to "
|
||||
"election-path recovery", prc);
|
||||
} else {
|
||||
c.ha.phase2_done = 1;
|
||||
}
|
||||
}
|
||||
|
||||
INFO("myapp_run: CTRL_STANDBY — waiting for primary to connect and push config");
|
||||
/* Arm heartbeat watchdog even when we skipped the election.
|
||||
* If the known primary never connects within heartbeat_timeout_ms,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue