lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent standby catch-up wedge after a WAL_CHECKPOINT boundary (bug-1475)

Browse source 
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.

Three-layer root cause, all now closed:

1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
   prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
   entries (a local-only, never-replicated marker that still consumes
   a real LSN slot). Now advances across them too, while still never
   shipping a catchup entry for the checkpoint itself.

2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
   ha->raft.log_len from persist_durable_floor() before consuming an
   AE, to correct for persist_write_checkpoint_marker advancing
   current_lsn without going through raft_log_append/record_entry_term
   (pre-existing gap, bug-1466, not itself fixed here).

3. The actual root cause, found via real instrumentation after layers
   1-2 alone still left a regression test red: record_entry_term's
   array-growth realloc zero-filled new slots, silently reclassifying
   a checkpoint's deliberately-never-recorded gap slot from "absent"
   to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
   recorded -- a time-dependent read where a permanent fact was
   assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
   growth now fills gap slots with instead of zero, so entry_term_locked
   can distinguish "never explicitly written" from "explicitly
   recorded as legacy term 0" regardless of later array growth.

The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).

Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
This commit is contained in:
Rongsong Shen 2026-07-28 04:48:53 +00:00
parent 7d31fce27a
commit ce8df99037
5 changed files with 650 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

85
src/controller/persist.c
View file

@ -499,14 +499,37 @@ static void* fsync_worker(void* arg)
* WAL term adapter (ha-term-safety task 1.3, R1.1/R1.5/R1.6/R1.7/R1.8) *
* ------------------------------------------------------------------ */
/* Sentinel stored in a term_index[] slot that has never been explicitly
* written by record_entry_term — distinct from a genuinely-recorded raw term
* of 0 (a legacy pre-term-safety entry, which maps to WAL_LEGACY_TERM at
* read-back). Reserved so growth (below) can leave a gap slot reading back as
* truly ABSENT forever, not just until the NEXT lsn happens to be recorded
* (bug-1475 follow-up: a WAL_CHECKPOINT slot is deliberately never recorded —
* see persist_write_checkpoint_marker's comment — but before this fix, the
* geometric-growth realloc for ANY later lsn zero-filled the checkpoint's own
* gap slot, and entry_term_locked's raw==0 rule then read that zero-fill back
* as WAL_LEGACY_TERM, not absent — silently and retroactively flipping a
* deliberately-unrecorded LSN's term the instant any later entry was
* appended. Confirmed via live instrumentation: persist_entry_term(checkpoint
* lsn) read back 0 (absent) immediately after the checkpoint, then 1
* (WAL_LEGACY_TERM) once a later real entry was appended and grew the array
* past it — a time-of-check/time-of-use divergence between the sender (which
* scans/streams AFTER later entries exist) and a receiver that has not yet
* appended past the same checkpoint, causing logOk's prev_log_term equality
* check to spuriously mismatch (0 != 1) and permanently reject catch-up). */
#define TERM_INDEX_UNSET UINT64_MAX
/* Record an entry's RAW hdr.term at its 1-based LSN in the in-memory per-LSN
* term index (R1.5). Grows the index on demand; LSNs replay/append contiguously
* so the array is dense (slot i == LSN i+1). Any gap left by a non-contiguous
* LSN (should not happen on the contiguous WAL) is zero-filled by realloc-then-
* memset, which read back as "absent" until/unless written. The raw term
* (including a legacy 0) is stored verbatim; the legacy-0 -> WAL_LEGACY_TERM
* mapping is applied only at READ-BACK (persist_entry_term), keeping "present
* legacy-0" distinguishable from "absent".
* LSN (e.g. a deliberately-skipped WAL_CHECKPOINT slot) is filled with the
* TERM_INDEX_UNSET sentinel by realloc-then-memset, which reads back as
* "absent" PERMANENTLY — regardless of whether a later LSN is subsequently
* recorded and grows term_index_count past it (bug-1475 follow-up; see
* TERM_INDEX_UNSET's comment). The raw term (including a legacy 0) is stored
* verbatim; the legacy-0 -> WAL_LEGACY_TERM mapping is applied only at
* READ-BACK (persist_entry_term), keeping "present legacy-0" distinguishable
* from "absent".
*
* Callers: the live append path (holds p->lock) and the single-threaded
* recovery path (replay_wal — no other thread touches the index before the node
@ -531,8 +554,11 @@ static void record_entry_term(persist_t* p, uint64_t lsn, uint64_t term)
"back as absent", lsn, p->term_index_cap, new_cap);
return;
}
/* Zero-fill the newly grown region so any unwritten slot reads absent. */
memset(grown + p->term_index_cap, 0,
/* Fill the newly grown region with the UNSET sentinel (all-1s byte
* pattern, so memset can fill it directly) so any never-explicitly-
* written slot reads absent FOREVER, not just until term_index_count
* happens to grow past it (bug-1475 follow-up). */
memset(grown + p->term_index_cap, 0xFF,
(new_cap - p->term_index_cap) * sizeof(uint64_t));
p->term_index = grown;
p->term_index_cap = new_cap;
@ -556,6 +582,11 @@ static uint64_t entry_term_locked(const persist_t* p, uint64_t lsn)
return 0;
if (lsn <= p->term_index_count && p->term_index != NULL) {
uint64_t raw = p->term_index[lsn - 1];
/* bug-1475 follow-up: a slot filled by growth but never explicitly
* recorded (e.g. a skipped WAL_CHECKPOINT lsn) stays absent forever —
* see TERM_INDEX_UNSET. */
if (raw == TERM_INDEX_UNSET)
return 0;
return (raw == 0) ? WAL_LEGACY_TERM : raw;
}
return 0;
@ -566,8 +597,11 @@ static uint64_t entry_term_locked(const persist_t* p, uint64_t lsn)
* the on-disk entries above keep_lsn are removed, keeping the term index
* consistent with the WAL: a later persist_entry_term on a truncated LSN must
* return 0. The backing allocation is left as-is (only the live count moves);
* the zeroed-on-grow contract means a later re-append at those LSNs overwrites
* cleanly. Caller must hold p->lock. */
* the TERM_INDEX_UNSET-sentinel-on-grow contract (bug-1475 follow-up — grown
* slots are filled with TERM_INDEX_UNSET, not zero, since this fix) means a
* later re-append at those LSNs still overwrites cleanly: record_entry_term
* always writes the slot explicitly, regardless of what fill value (zero or
* sentinel) it previously held. Caller must hold p->lock. */
static void truncate_term_index_locked(persist_t* p, uint64_t keep_lsn)
{
if (!p)
@ -624,15 +658,44 @@ uint64_t persist_entry_term_locked(const persist_t* p, uint64_t lsn)
* check against this member ever references an LSN below the checkpoint
* again (the checkpoint is the new log-start boundary), and the WAL bytes
* below it were compacted away, so there is no data to reconstruct a full
* per-LSN history from even if it were needed. */
* per-LSN history from even if it were needed.
*
* bug-1475 Finding 2 (coordinator review, 2026-07-27/28): checkpoint_term==0
* on the wire is UNAMBIGUOUS today, not a legacy default to fall back to
* WAL_LEGACY_TERM. ha_snapshot_do_send (ha.c) — the only sender — computes
* this value via persist_entry_term(leader, checkpoint_lsn), whose public
* contract already collapses "absent" and "present raw-0 legacy" into
* DIFFERENT return values (0 vs WAL_LEGACY_TERM respectively — see
* entry_term_locked). A wire value of exactly 0 can therefore only mean
* "this sender's checkpoint LSN is genuinely, permanently absent" (the
* expected, designed state for ANY real checkpoint — persist_write_
* checkpoint_marker never records a term for its own entry). Before this
* fix, record_entry_term(p, checkpoint_lsn, 0) stored a RAW 0 — an EXPLICIT
* write, not a growth-fill, so it never got the TERM_INDEX_UNSET sentinel —
* and entry_term_locked's legacy-0 rule then read it back as
* WAL_LEGACY_TERM(1) forever, permanently diverging from the sender's own
* permanently-absent(0) value for the SAME LSN: a subsequent AppendEntries
* whose prev_log_index==checkpoint_lsn (the entry immediately following it)
* then mismatched logOk (0 != 1) forever — the identical wedge class
* bug-1475 fixed for WAL-streaming catch-up, reachable instead via
* HA_REPL_SEND_SNAPSHOT (dispatched whenever a standby has fallen too far
* behind for incremental streaming — a live, common path). Recording the
* SAME TERM_INDEX_UNSET sentinel here (instead of raw 0) keeps a genuinely-
* absent checkpoint boundary reading absent on the receiver too, matching
* the sender — while a genuinely non-zero checkpoint_term (a real historical
* term, however it arrived on the wire) is still recorded verbatim and still
* correctly overwrites any stale local term at this LSN, unchanged. */
void persist_reconcile_checkpoint_term(persist_t* p, uint64_t checkpoint_lsn,
uint64_t checkpoint_term)
{
if (!p || checkpoint_lsn == 0)
return;
uint64_t stored_term =
(checkpoint_term == 0) ? TERM_INDEX_UNSET : checkpoint_term;
pthread_mutex_lock(&p->lock);
record_entry_term(p, checkpoint_lsn, checkpoint_term);
record_entry_term(p, checkpoint_lsn, stored_term);
pthread_mutex_unlock(&p->lock);
}