lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

28
src/controller/repl.c
View file

@ -775,6 +775,12 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
if (r->standbys[i].ep == ep) {
atomic_store_explicit(&r->standbys[i].in_catch_up, 0,
memory_order_release);
/* bug-1479: reset the eviction counter on re-entry to live
* broadcast -- otherwise a stale pending_ack_count carried over
* from before catch-up could immediately re-trip
* repl_broadcast's MAX_PENDING_ACKS eviction on the very next
* send, undoing the catch-up this function just completed. */
r->standbys[i].pending_ack_count = 0;
changed = 1;
break;
}
@ -794,6 +800,28 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
pthread_mutex_unlock(&r->ack_lock);
}
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
uint64_t *join_applied_lsn_out)
{
if (r == NULL || join_applied_lsn_out == NULL)
return 0;
int found = 0;
mylock_lock(&r->lock);
for (int i = 0; i < MAX_STANDBYS; i++) {
if (r->standbys[i].ep == ep) {
if (atomic_load_explicit(&r->standbys[i].in_catch_up,
memory_order_acquire)) {
*join_applied_lsn_out = r->standbys[i].join_applied_lsn;
found = 1;
}
break;
}
}
mylock_unlock(&r->lock);
return found;
}
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn,
uint64_t oldest_wal_lsn)
{