lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent standby catch-up wedge after a WAL_CHECKPOINT boundary (bug-1475)

Browse source 
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.

Three-layer root cause, all now closed:

1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
   prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
   entries (a local-only, never-replicated marker that still consumes
   a real LSN slot). Now advances across them too, while still never
   shipping a catchup entry for the checkpoint itself.

2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
   ha->raft.log_len from persist_durable_floor() before consuming an
   AE, to correct for persist_write_checkpoint_marker advancing
   current_lsn without going through raft_log_append/record_entry_term
   (pre-existing gap, bug-1466, not itself fixed here).

3. The actual root cause, found via real instrumentation after layers
   1-2 alone still left a regression test red: record_entry_term's
   array-growth realloc zero-filled new slots, silently reclassifying
   a checkpoint's deliberately-never-recorded gap slot from "absent"
   to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
   recorded -- a time-dependent read where a permanent fact was
   assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
   growth now fills gap slots with instead of zero, so entry_term_locked
   can distinguish "never explicitly written" from "explicitly
   recorded as legacy term 0" regardless of later array growth.

The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).

Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
This commit is contained in:
Rongsong Shen 2026-07-28 04:48:53 +00:00
parent 7d31fce27a
commit ce8df99037
5 changed files with 650 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

55
src/controller/config_repl.c
View file

@ -2356,19 +2356,50 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
* never sent on the live broadcast path either
* (persist_wal_write_replicated rejects type==WAL_CHECKPOINT with
* -EINVAL). Skip it entirely -- do not build/send a catchup entry
* and do not advance prev_streamed_lsn/prev_streamed_term, so the
* NEXT real entry's AppendEntries prev tuple chains to the last
* REAL entry before this gap, exactly matching what a standby that
* never records a checkpoint marker in its own log expects.
* Without this, a standby receiving one gets stuck retrying the
* same LSN forever (the receive-side guard correctly refuses it). */
if (hdr.type == WAL_CHECKPOINT) {
free(locked_pbuf);
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
continue;
}
* for it.
*
* bug-1475 (fix): DO fall through to the SAME unconditional
* prev-tracker advance every other scanned entry gets below
* (the "for EVERY scanned entry" logic a few lines down) instead
* of special-casing a skip here. A standby's own current_lsn/
* durable_floor ALSO advances past this exact LSN the moment it
* independently applies the SAME committed WAL_CONFIG_MARKER (any
* member that commits a marker immediately writes its own local
* checkpoint at the marker's own LSN+1, deterministically the
* SAME lsn on every member that has caught up through the
* marker) -- so the checkpoint's LSN is a real, held position on
* BOTH sides, not a gap. The OLD behavior (never advancing the
* tracker across it) under-counted the sender's prev_log_index by
* one for every checkpoint encountered, permanently diverging it
* from the receiver's own (now correctly co-tracked -- see
* ha_apply_term_entry_via_library's bug-1475 fix) log_len: the
* first post-checkpoint entry would spuriously "succeed" at the
* WRONG index, after which every following entry's prev_log_index
* permanently exceeded the receiver's true log length --
* ae_reject_logmismatch forever, with the primary re-streaming
* the identical doomed range every heartbeat. Falling through
* here (falling to the same code path as every other entry)
* fixes the divergence at its source.
*
* bug-1475 follow-up: this tracker advance alone was NOT
* sufficient — persist_entry_term(p, checkpoint_lsn) (the term
* this tracker reads for prev_log_term) turned out to be
* TIME-DEPENDENT before a further persist.c fix: it correctly
* read absent (0) right after the checkpoint, but silently
* flipped to WAL_LEGACY_TERM (1) the instant this leader
* appended any LATER real entry (which has already happened by
* the time this streamer runs), because persist.c's term-index
* array growth zero-filled the checkpoint's never-explicitly-
* recorded gap slot and read that zero-fill back as present. A
* receiver that had not yet appended past its own checkpoint
* still (correctly) read 0 for the same LSN, so logOk's
* prev_log_term equality check mismatched (0 != 1) and rejected
* forever. Fixed with a TERM_INDEX_UNSET sentinel in persist.c
* (record_entry_term/entry_term_locked) that keeps a
* never-recorded slot reading absent permanently, independent of
* later growth — see TERM_INDEX_UNSET's comment there. */
if (hdr.lsn > standby_lsn) {
if (hdr.lsn > standby_lsn && hdr.type != WAL_CHECKPOINT) {
wal_catchup_entry_t *entry = calloc(1, sizeof(*entry));
if (!entry) {
free(locked_pbuf);