fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node rotation) can leave a standby log-conflicted after a rejoin. Fixing this needed three compounding layers, each unit-green but still failing the identical live scenario until the next layer landed: 1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft library's nextIndex backoff and immediately repl_start_catchup's the standby at the corrected position, instead of the 16-strike silent eviction with no catch-up driver after it. Correct from the start, but live soak still failed identically: nothing ever produced the NACK this code was waiting for. 2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of silently dropping the reject. ha_send_wal_ack_reply had hardcoded success=1 unconditionally -- the only WalAckReply sender in the controller -- making (1)'s NACK-routing correct but unreachable dead code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe the NACK without a live endpoint. Live-confirmed the NACK now reaches the primary -- but the scenario still failed for a third reason. 3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, which can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Widens the STREAM_WAL ceiling (and the convergence check gating it) to the leader's own log tail, matching raft.tla's real AppendEntries semantics (leaderCommit is advisory, never a send-cap), while reusing bug-1478's exact guard so this doesn't reopen that trap (never widen onto a trailing local-only checkpoint's own unstreamable LSN). Verification: controller_unit_test 1053/1053 PASS, clean twice, including new regression tests for all three layers (test_config_repl.c, test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its pre-fix behavior before the real fix was restored. Three successive live run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the final run passed end-to-end -- all 3 force_elect_node rounds succeeded (ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never previously reached), zero materialized-state divergence, zero job loss. Unblocks config-upgrade-snapshot task 8.6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
aecd8766f4
commit
34ab78d72d
14 changed files with 1256 additions and 377 deletions
|
|
@ -137,6 +137,16 @@ void repl_start_catchup(replicator_t *r, repl_ep_t ep, uint64_t join_applied_lsn
|
|||
/* Mark standby catch-up complete (called on OP_replCatchUpDone receipt) */
|
||||
void repl_complete_catchup(replicator_t *r, repl_ep_t ep);
|
||||
|
||||
/* bug-1479: look up whether the given standby is currently in catch-up and,
|
||||
* if so, its corrected resume point (join_applied_lsn). Returns 1 and sets
|
||||
* *join_applied_lsn_out when in_catch_up is set; returns 0 (leaves
|
||||
* *join_applied_lsn_out untouched) otherwise. Lets a caller (the heartbeat-
|
||||
* driven replication decision) use the corrected position instead of a
|
||||
* standby's own self-reported LSN, which is unreliable for a standby stuck
|
||||
* rejecting every AppendEntries due to a log conflict. */
|
||||
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
|
||||
uint64_t *join_applied_lsn_out);
|
||||
|
||||
/* Returns 1 if standby needs a full snapshot transfer (its LSN predates oldest WAL).
|
||||
* oldest_wal_lsn == 0 means no WAL exists — no snapshot needed for LSN 0 standbys. */
|
||||
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn, uint64_t oldest_wal_lsn);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue