wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

49
src/controller/ha.c
View file

@ -1570,6 +1570,16 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
int prc = persist_wal_write_replicated(p, e->lsn, (uint32_t)e->entry_type,
e->raw_payload,
(uint32_t)e->raw_payload_len);
if (prc == -EAGAIN) {
/* Contiguity guard (bug-850): this entry is ahead of our log tail
* (a forward gap) — e.g. a live broadcast that arrived before the
* in-order catch-up of the prefix. NOT an error: drop it quietly and do
* NOT ack, so the catch-up continuation re-streams from current_lsn and
* the log fills in order. */
TRACE("ha_apply_single_entry: lsn=%" PRIu64 " ahead of log tail — "
"dropped (will arrive via in-order catch-up)", e->lsn);
return prc;
}
if (prc != 0) {
ERROR("ha_apply_single_entry: persist_wal_write_replicated failed for "
"lsn=%" PRIu64 " type=%u rc=%d — not applying/acking",
@ -1636,11 +1646,14 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
int rc = ha_apply_single_entry(p, sub);
if (rc != 0) {
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed "
"for batch index %d lsn=%" PRIu64 " type=%u rc=%d "
"— skipping entry",
i, sub->lsn, (unsigned)sub->entry_type, rc);
/* Task 3.2: no ACK for failed entries */
/* -EAGAIN = contiguity gap (bug-850): entry ahead of our tail,
* dropped quietly — not an error (catch-up will backfill). */
if (rc != -EAGAIN)
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed "
"for batch index %d lsn=%" PRIu64 " type=%u rc=%d "
"— skipping entry",
i, sub->lsn, (unsigned)sub->entry_type, rc);
/* Task 3.2: no ACK for failed/dropped entries */
continue;
}
@ -1662,9 +1675,12 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
/* ── Single-entry mode (task 2.1 backward-compatible path) ────────────── */
int rc = ha_apply_single_entry(p, entry);
if (rc != 0) {
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed for "
"lsn=%" PRIu64 " type=%u rc=%d — skipping entry",
entry->lsn, (unsigned)entry->entry_type, rc);
/* -EAGAIN = contiguity gap (bug-850): dropped quietly, backfilled by
* the in-order catch-up; any other rc is a real failure. */
if (rc != -EAGAIN)
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed for "
"lsn=%" PRIu64 " type=%u rc=%d — skipping entry",
entry->lsn, (unsigned)entry->entry_type, rc);
/* Task 3.2: no ACK on apply failure */
return rc;
}
@ -2600,6 +2616,23 @@ uint64_t ha_candidate_advertise_lsn(persist_t *p)
return persist_durable_floor(p);
}
/* bug-848: see ha.h. After local WAL recovery, advertise the recovered durable
* tip for catch-up so the primary streams only the missing suffix instead of
* re-streaming + overwriting consistent records. Monotonic: never lowers a
* higher applied_lsn (e.g. if catch-up already advanced it). */
void ha_sync_applied_lsn_from_recovery(ha_state_t *ha, persist_t *p)
{
if (!ha || !p)
return;
uint64_t recovered = persist_durable_floor(p);
uint64_t cur = atomic_load_explicit(&ha->applied_lsn,
memory_order_acquire);
if (recovered > cur)
atomic_store_explicit(&ha->applied_lsn, recovered,
memory_order_release);
}
/* ── Vote Handler (Req 2.2–2.6, 6.1, 6.2, 6.4, 9.1) ────────────────────────
*
* handle_request_vote: process an incoming OP_requestVote RPC.