fix(ha): close permanent standby catch-up wedge after a WAL_CHECKPOINT boundary (bug-1475)
A restarted standby could get permanently stuck unable to catch up, with the primary re-streaming the identical doomed AppendEntries range every heartbeat with zero progress -- discovered via a live 3-node config-upgrade-snapshot soak. Liveness-only; no safety invariant (Election Safety, Leader Completeness, State Machine Safety, Commit Durability) was ever violated. Three-layer root cause, all now closed: 1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT entries (a local-only, never-replicated marker that still consumes a real LSN slot). Now advances across them too, while still never shipping a catchup entry for the checkpoint itself. 2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs ha->raft.log_len from persist_durable_floor() before consuming an AE, to correct for persist_write_checkpoint_marker advancing current_lsn without going through raft_log_append/record_entry_term (pre-existing gap, bug-1466, not itself fixed here). 3. The actual root cause, found via real instrumentation after layers 1-2 alone still left a regression test red: record_entry_term's array-growth realloc zero-filled new slots, silently reclassifying a checkpoint's deliberately-never-recorded gap slot from "absent" to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got recorded -- a time-dependent read where a permanent fact was assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that growth now fills gap slots with instead of zero, so entry_term_locked can distinguish "never explicitly written" from "explicitly recorded as legacy term 0" regardless of later array growth. The same determinism fix exposed a second, independent instance of the identical wedge via the snapshot-install catch-up path instead of WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a wire checkpoint_term of 0 as a raw, explicit write (bypassing the growth-fill sentinel entirely), so a snapshot-installed standby read the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader now correctly read its own checkpoint LSN as absent(0) -- the same permanent logOk mismatch, a different code path. Fixed by substituting TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a wire 0 is unambiguous (persist_entry_term's own contract never returns raw 0 for a present entry, so 0 can only mean genuine absence). Independently reviewed across three rounds: sentinel mechanism verified via real RED->GREEN revert-and-rebuild; a documentation claim that config_repl.c/ha.c were "comment-only" corrected; the snapshot-path gap independently traced, confirmed real via its own RED->GREEN cycle, and fixed; two stale doc comments (persist.h's public contract, ha.c's call-site rationale) still describing the disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035 tests, 0 failures.
This commit is contained in:
parent
7d31fce27a
commit
ce8df99037
5 changed files with 650 additions and 45 deletions
|
|
@ -660,12 +660,18 @@ uint64_t persist_entry_term_locked(const persist_t* p, uint64_t lsn);
|
|||
* checkpoint against this member again, so only the checkpoint LSN's own
|
||||
* term needs reconciling — not the full compacted history below it.
|
||||
*
|
||||
* `checkpoint_term == 0` means a legacy sender (pre-task-3.2 peer) never
|
||||
* stamped the wire field; treat that as WAL_LEGACY_TERM per the existing
|
||||
* legacy-0 convention (record_entry_term stores the raw value verbatim; the
|
||||
* legacy-0 -> WAL_LEGACY_TERM mapping is applied uniformly at read-back by
|
||||
* persist_entry_term/entry_term_locked, so this call is consistent with that
|
||||
* convention without needing to special-case it here).
|
||||
* `checkpoint_term == 0` is unambiguous and means the checkpoint LSN is
|
||||
* genuinely/permanently absent from the term index, never "legacy term 0,
|
||||
* present" (bug-1475 Finding 2: the sole sender, ha_snapshot_do_send, always
|
||||
* computes this value via persist_entry_term, whose own contract already
|
||||
* returns the distinct WAL_LEGACY_TERM for any present-but-legacy-0 entry —
|
||||
* so a wire value of exactly 0 can only mean absent). This call substitutes
|
||||
* TERM_INDEX_UNSET for a wire 0 before recording, instead of the raw value,
|
||||
* so entry_term_locked reads the checkpoint LSN back as absent on the
|
||||
* receiver too, matching the sender — recording the raw 0 here (the old
|
||||
* behavior) let it read back as WAL_LEGACY_TERM instead, permanently
|
||||
* diverging from the sender's own absent(0) and wedging the next
|
||||
* AppendEntries whose prevLogIndex lands on this LSN.
|
||||
*
|
||||
* Takes p->lock internally (callers must NOT already hold it). Safe on a
|
||||
* NULL p or checkpoint_lsn == 0 (no-op). */
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue