forked from lei.chen/myxce
Rongsong Shen
ce8df99037
A restarted standby could get permanently stuck unable to catch up, with the primary re-streaming the identical doomed AppendEntries range every heartbeat with zero progress -- discovered via a live 3-node config-upgrade-snapshot soak. Liveness-only; no safety invariant (Election Safety, Leader Completeness, State Machine Safety, Commit Durability) was ever violated. Three-layer root cause, all now closed: 1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT entries (a local-only, never-replicated marker that still consumes a real LSN slot). Now advances across them too, while still never shipping a catchup entry for the checkpoint itself. 2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs ha->raft.log_len from persist_durable_floor() before consuming an AE, to correct for persist_write_checkpoint_marker advancing current_lsn without going through raft_log_append/record_entry_term (pre-existing gap, bug-1466, not itself fixed here). 3. The actual root cause, found via real instrumentation after layers 1-2 alone still left a regression test red: record_entry_term's array-growth realloc zero-filled new slots, silently reclassifying a checkpoint's deliberately-never-recorded gap slot from "absent" to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got recorded -- a time-dependent read where a permanent fact was assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that growth now fills gap slots with instead of zero, so entry_term_locked can distinguish "never explicitly written" from "explicitly recorded as legacy term 0" regardless of later array growth. The same determinism fix exposed a second, independent instance of the identical wedge via the snapshot-install catch-up path instead of WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a wire checkpoint_term of 0 as a raw, explicit write (bypassing the growth-fill sentinel entirely), so a snapshot-installed standby read the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader now correctly read its own checkpoint LSN as absent(0) -- the same permanent logOk mismatch, a different code path. Fixed by substituting TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a wire 0 is unambiguous (persist_entry_term's own contract never returns raw 0 for a present entry, so 0 can only mean genuine absence). Independently reviewed across three rounds: sentinel mechanism verified via real RED->GREEN revert-and-rebuild; a documentation claim that config_repl.c/ha.c were "comment-only" corrected; the snapshot-path gap independently traced, confirmed real via its own RED->GREEN cycle, and fixed; two stale doc comments (persist.h's public contract, ha.c's call-site rationale) still describing the disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035 tests, 0 failures. |
||
|---|---|---|
| .. | ||
| include | ||
| testsuite | ||
| apiserver.c | ||
| CLAUDE.md | ||
| cluster.c | ||
| commit.c | ||
| config_repl.c | ||
| config_repl.h | ||
| ctrl_metrics.c | ||
| ctrlstatus.c | ||
| dataex.c | ||
| egress.c | ||
| ha.c | ||
| ha.h | ||
| ha_convergence.c | ||
| job.c | ||
| join.c | ||
| main.c | ||
| meson.build | ||
| misc.c | ||
| myapp.c | ||
| myconf.c | ||
| opget.c | ||
| opupd.c | ||
| persist.c | ||
| queue.c | ||
| repl.c | ||
| reply.c | ||
| resources.c | ||
| sched.c | ||
| slowlane.c | ||
| startup_validation.c | ||
| task.c | ||
| worker.c | ||