forked from lei.chen/myxce
fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)
Two independent call sites duplicated target = max(committed_lsn, last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_ dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven catch-up continuation path). Once config-upgrade-snapshot's marker-triggered checkpoint advances last_written one LSN past an already-committed committed_lsn (the checkpoint's own consumed, never-replicated LSN slot), the unconditional max() permanently wedges every standby's catch-up target at an unreachable value -- the primary re-streams an empty range every heartbeat forever. Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn, last_written) (ha.h/ha.c), which only falls back to last_written on genuine committed_lsn==0 bootstrap, and pointed both call sites at it so they can never disagree again. Fixing the second call site exposed three pre-existing tests in test_convergence_verification.c that hardcoded the old buggy formula's output as their expected target; updated their fixture values (committed_lsn 4->9, matching appended_lsn) to decouple them from the now-fixed edge case, which already has dedicated pure-function coverage in test_ha_role.c. GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean across multiple runs. Live-verified via run-ha-soak.sh --config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3 using the preserved floor segment) -- unblocks config-upgrade-snapshot task 8.9.
This commit is contained in:
parent
71223ec712
commit
102e263cce
9 changed files with 850 additions and 20 deletions
|
|
@ -3673,6 +3673,52 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
|
|||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_compute_repl_target_lsn: the replication/convergence target that
|
||||
* ha_dispatch_repl_action_for_standby's heartbeat-driven decision compares
|
||||
* every standby's reported applied_lsn against.
|
||||
*
|
||||
* The correct target is simply the real, raft/commit-engine-authoritative
|
||||
* committed_lsn — the only position a standby can safely (and eventually)
|
||||
* be told it must catch up to. The ONE documented exception is cluster
|
||||
* BOOTSTRAP: when committed_lsn is genuinely stuck at 0 (no standby has ever
|
||||
* acked anything yet, so the commit engine's majority computation has never
|
||||
* had anything to advance against), last_written (the raw locally-appended
|
||||
* WAL tip) is used instead, so a first-ever connecting standby is told to
|
||||
* catch up to real data instead of being handed "nothing to stream".
|
||||
*
|
||||
* bug-1478: this fallback must apply ONLY in that committed_lsn==0 bootstrap
|
||||
* case — NOT unconditionally whenever last_written happens to exceed
|
||||
* committed_lsn. last_written (persist.current_lsn - 1) counts EVERY
|
||||
* locally-appended WAL entry, including a local-only, NEVER-replicated
|
||||
* WAL_CHECKPOINT (persist_write_checkpoint_marker bumps current_lsn
|
||||
* directly, bypassing raft_log_append/the commit engine entirely — the
|
||||
* same architectural gap flagged in bug-1466). Once such a checkpoint
|
||||
* trails an already-advancing (non-zero) committed_lsn, last_written
|
||||
* permanently sits one LSN past it — and since a checkpoint's own consumed
|
||||
* LSN slot is, by design, never streamed or applied by any standby
|
||||
* (ha_stream_wal_to_standby / persist_wal_write_replicated /
|
||||
* persist_wal_async all explicitly refuse WAL_CHECKPOINT), no standby's
|
||||
* applied_lsn can EVER legitimately reach that value. Taking the
|
||||
* max(committed_lsn, last_written) unconditionally turned the bootstrap
|
||||
* convenience into a permanent, unsatisfiable catch-up target for every
|
||||
* standby the instant a marker-triggered checkpoint (config-upgrade-
|
||||
* snapshot task 5) or a periodic WAL-threshold checkpoint interleaves ahead
|
||||
* of the next real committed advance — ha_evaluate_repl_action's
|
||||
* standby_lsn < committed_lsn check then never reaches HA_REPL_CONVERGED,
|
||||
* and ha_stream_wal_to_standby is re-invoked every heartbeat forever,
|
||||
* finding nothing new to send each time (bug-1478).
|
||||
*
|
||||
* Pure, side-effect-free — safe to unit test directly.
|
||||
*/
|
||||
uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
|
||||
uint64_t last_written)
|
||||
{
|
||||
if (committed_lsn == 0 && last_written > 0)
|
||||
return last_written;
|
||||
return committed_lsn;
|
||||
}
|
||||
|
||||
/* ───────────────────────────────────────────────────────────────────────
|
||||
* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198).
|
||||
*
|
||||
|
|
@ -6376,12 +6422,14 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
|
|||
oldest_wal_lsn = ha_compute_oldest_wal_lsn(c->persist.wal_path);
|
||||
}
|
||||
#endif
|
||||
/* Replication target: use max(committed_lsn, last_written) so
|
||||
* standbys can catch up even when committed_lsn is stuck at 0
|
||||
* (no replicator-registered standbys to advance it). */
|
||||
uint64_t target_lsn = committed_lsn;
|
||||
if (last_written > target_lsn)
|
||||
target_lsn = last_written;
|
||||
/* Replication target (bug-1478): committed_lsn, with the
|
||||
* last_written bootstrap fallback ONLY when committed_lsn is
|
||||
* genuinely stuck at 0 — see ha_compute_repl_target_lsn's own
|
||||
* doc comment for why an unconditional max() permanently wedges
|
||||
* catch-up once a local-only checkpoint trails a real,
|
||||
* already-advancing committed_lsn. */
|
||||
uint64_t target_lsn =
|
||||
ha_compute_repl_target_lsn(committed_lsn, last_written);
|
||||
|
||||
ha_dispatch_repl_action_for_standby(ha, c, ep, peer_idx,
|
||||
committed_tag, target_lsn,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue