fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)

Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.

Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.

Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.

GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
This commit is contained in:
Rongsong Shen 2026-07-28 09:53:20 +00:00
parent 71223ec712
commit 102e263cce
9 changed files with 850 additions and 20 deletions

View file

@ -3673,6 +3673,52 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
}
}
/*
* ha_compute_repl_target_lsn: the replication/convergence target that
* ha_dispatch_repl_action_for_standby's heartbeat-driven decision compares
* every standby's reported applied_lsn against.
*
* The correct target is simply the real, raft/commit-engine-authoritative
* committed_lsn — the only position a standby can safely (and eventually)
* be told it must catch up to. The ONE documented exception is cluster
* BOOTSTRAP: when committed_lsn is genuinely stuck at 0 (no standby has ever
* acked anything yet, so the commit engine's majority computation has never
* had anything to advance against), last_written (the raw locally-appended
* WAL tip) is used instead, so a first-ever connecting standby is told to
* catch up to real data instead of being handed "nothing to stream".
*
* bug-1478: this fallback must apply ONLY in that committed_lsn==0 bootstrap
* case — NOT unconditionally whenever last_written happens to exceed
* committed_lsn. last_written (persist.current_lsn - 1) counts EVERY
* locally-appended WAL entry, including a local-only, NEVER-replicated
* WAL_CHECKPOINT (persist_write_checkpoint_marker bumps current_lsn
* directly, bypassing raft_log_append/the commit engine entirely — the
* same architectural gap flagged in bug-1466). Once such a checkpoint
* trails an already-advancing (non-zero) committed_lsn, last_written
* permanently sits one LSN past it — and since a checkpoint's own consumed
* LSN slot is, by design, never streamed or applied by any standby
* (ha_stream_wal_to_standby / persist_wal_write_replicated /
* persist_wal_async all explicitly refuse WAL_CHECKPOINT), no standby's
* applied_lsn can EVER legitimately reach that value. Taking the
* max(committed_lsn, last_written) unconditionally turned the bootstrap
* convenience into a permanent, unsatisfiable catch-up target for every
* standby the instant a marker-triggered checkpoint (config-upgrade-
* snapshot task 5) or a periodic WAL-threshold checkpoint interleaves ahead
* of the next real committed advance — ha_evaluate_repl_action's
* standby_lsn < committed_lsn check then never reaches HA_REPL_CONVERGED,
* and ha_stream_wal_to_standby is re-invoked every heartbeat forever,
* finding nothing new to send each time (bug-1478).
*
* Pure, side-effect-free — safe to unit test directly.
*/
uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
uint64_t last_written)
{
if (committed_lsn == 0 && last_written > 0)
return last_written;
return committed_lsn;
}
/* ───────────────────────────────────────────────────────────────────────
* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198).
*
@ -6376,12 +6422,14 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
oldest_wal_lsn = ha_compute_oldest_wal_lsn(c->persist.wal_path);
}
#endif
/* Replication target: use max(committed_lsn, last_written) so
* standbys can catch up even when committed_lsn is stuck at 0
* (no replicator-registered standbys to advance it). */
uint64_t target_lsn = committed_lsn;
if (last_written > target_lsn)
target_lsn = last_written;
/* Replication target (bug-1478): committed_lsn, with the
* last_written bootstrap fallback ONLY when committed_lsn is
* genuinely stuck at 0 — see ha_compute_repl_target_lsn's own
* doc comment for why an unconditional max() permanently wedges
* catch-up once a local-only checkpoint trails a real,
* already-advancing committed_lsn. */
uint64_t target_lsn =
ha_compute_repl_target_lsn(committed_lsn, last_written);
ha_dispatch_repl_action_for_standby(ha, c, ep, peer_idx,
committed_tag, target_lsn,