fix(ha): close permanent standby catch-up wedge after a WAL_CHECKPOINT boundary (bug-1475)

A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.

Three-layer root cause, all now closed:

1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
   prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
   entries (a local-only, never-replicated marker that still consumes
   a real LSN slot). Now advances across them too, while still never
   shipping a catchup entry for the checkpoint itself.

2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
   ha->raft.log_len from persist_durable_floor() before consuming an
   AE, to correct for persist_write_checkpoint_marker advancing
   current_lsn without going through raft_log_append/record_entry_term
   (pre-existing gap, bug-1466, not itself fixed here).

3. The actual root cause, found via real instrumentation after layers
   1-2 alone still left a regression test red: record_entry_term's
   array-growth realloc zero-filled new slots, silently reclassifying
   a checkpoint's deliberately-never-recorded gap slot from "absent"
   to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
   recorded -- a time-dependent read where a permanent fact was
   assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
   growth now fills gap slots with instead of zero, so entry_term_locked
   can distinguish "never explicitly written" from "explicitly
   recorded as legacy term 0" regardless of later array growth.

The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).

Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
This commit is contained in:
Rongsong Shen 2026-07-28 04:48:53 +00:00
parent 7d31fce27a
commit ce8df99037
5 changed files with 650 additions and 45 deletions

View file

@ -660,12 +660,18 @@ uint64_t persist_entry_term_locked(const persist_t* p, uint64_t lsn);
* checkpoint against this member again, so only the checkpoint LSN's own
* term needs reconciling — not the full compacted history below it.
*
* `checkpoint_term == 0` means a legacy sender (pre-task-3.2 peer) never
* stamped the wire field; treat that as WAL_LEGACY_TERM per the existing
* legacy-0 convention (record_entry_term stores the raw value verbatim; the
* legacy-0 -> WAL_LEGACY_TERM mapping is applied uniformly at read-back by
* persist_entry_term/entry_term_locked, so this call is consistent with that
* convention without needing to special-case it here).
* `checkpoint_term == 0` is unambiguous and means the checkpoint LSN is
* genuinely/permanently absent from the term index, never "legacy term 0,
* present" (bug-1475 Finding 2: the sole sender, ha_snapshot_do_send, always
* computes this value via persist_entry_term, whose own contract already
* returns the distinct WAL_LEGACY_TERM for any present-but-legacy-0 entry —
* so a wire value of exactly 0 can only mean absent). This call substitutes
* TERM_INDEX_UNSET for a wire 0 before recording, instead of the raw value,
* so entry_term_locked reads the checkpoint LSN back as absent on the
* receiver too, matching the sender — recording the raw 0 here (the old
* behavior) let it read back as WAL_LEGACY_TERM instead, permanently
* diverging from the sender's own absent(0) and wedging the next
* AppendEntries whose prevLogIndex lands on this LSN.
*
* Takes p->lock internally (callers must NOT already hold it). Safe on a
* NULL p or checkpoint_lsn == 0 (no-op). */