lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha-repl-hardening): snapshot fallback off-by-one — empty standby must stream, not snapshot

Browse source 
After reserve-lsn-0 (oldest retained WAL entry = lsn 1 on a fresh cluster), a
freshly-joined empty standby (applied_lsn=0) hit the snapshot-fallback branch
'standby_lsn < oldest_wal_lsn' (0 < 1) and was sent a non-existent snapshot
instead of streaming — so the own-term NOOP (lsn 1) never replicated, committed_lsn
stayed 0, the own-term gate (writable_lsn=1) stayed closed, and standbys kept
empty WALs. Live evidence:
  ha_dispatch_repl_action_for_standby: standby[0] lsn=0 < oldest_wal_lsn=1 -> requesting snapshot transfer

The snapshot fallback should fire only when the standby's NEXT-needed entry
(standby_lsn+1) has been GC'd below the oldest retained entry. Fix the boundary
to 'standby_lsn + 1 < oldest_wal_lsn' in ha_evaluate_repl_action (and the parallel
dead repl_needs_snapshot predicate). An empty standby (next-needed=1, oldest=1)
now STREAMs lsn 1 -> acks -> committed_lsn 0->1 -> own-term gate opens -> standbys
receive the WAL. Snapshot still fires for genuinely GC'd-behind standbys
(standby_lsn+1 < oldest). No underflow (oldest=0 -> 1<0 false). Pre-reserve-lsn-0
this off-by-one was masked by oldest_wal_lsn=0 (0<0 false).

TDD: test_empty_standby_fresh_cluster_streams_not_snapshot (RED->GREEN);
snapshot + heartbeat pins strengthened to assert SEND_SNAPSHOT only for genuinely
GC'd standbys and STREAM_WAL at the empty/boundary case. Full controller suite
green under ASAN; commit engine / streamer / apply path / wire untouched.
This commit is contained in:
Rongsong Shen 2026-06-14 12:10:40 +00:00
parent 7519fa96dd
commit 007bf0c403
4 changed files with 120 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

15
src/controller/ha.c
View file

@ -1251,9 +1251,9 @@ void ha_nop_reply_state_report(ha_state_t *ha, myconf_t *myconf,
*
* Implements the decision tree from the ha-repl-hardening design:
*
* if standby_tag < committed_tag → HA_REPL_PUSH_CONFIG
* elif standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT
* elif standby_lsn < committed_lsn → HA_REPL_STREAM_WAL
* if standby_tag < committed_tag → HA_REPL_PUSH_CONFIG
* elif standby_lsn + 1 < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT
* elif standby_lsn < committed_lsn → HA_REPL_STREAM_WAL
* else (tag >= committed AND lsn >= committed) → HA_REPL_CONVERGED
*
* No side effects. All callers must use the returned action to drive the
@ -1271,7 +1271,14 @@ ha_repl_action_t ha_evaluate_repl_action(uint64_t standby_tag,
if (standby_tag < committed_tag)
return HA_REPL_PUSH_CONFIG;
if (standby_lsn < oldest_wal_lsn)
/* Snapshot only when the standby's NEXT-needed entry (standby_lsn + 1) has
* been garbage-collected below the oldest retained WAL entry — i.e. it can
* no longer be served by streaming. An empty standby (standby_lsn=0) on a
* fresh cluster whose oldest retained entry IS lsn 1 (post reserve-lsn-0)
* needs lsn 1, which is retained: 0 + 1 < 1 is false → STREAM, no snapshot.
* Using standby_lsn + 1 (not standby_lsn - ... ) avoids unsigned underflow
* when oldest_wal_lsn == 0 (empty WAL): 0 + 1 < 0 is false → not snapshot. */
if (standby_lsn + 1 < oldest_wal_lsn)
return HA_REPL_SEND_SNAPSHOT;
if (standby_lsn < committed_lsn)