lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

174
src/controller/ha.c
View file

@ -3081,19 +3081,36 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
return 0;
}
/* ha_send_wal_ack_reply: send WalAckReply {term, lsn, success=1} back to
* the primary over ep. Skipped when ep or c is NULL (test/offline path).
* Task 3.1 (ha-wal-replication): ACK each successfully applied WAL entry. */
static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
cluster_t *c, uint64_t lsn)
#ifdef XCE_TESTING
void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success) = NULL;
#endif
/* ha_send_wal_ack_reply_ex: send WalAckReply {term, lsn, success} back to
* the primary over ep. Skipped when ep or c is NULL (test/offline path),
* unless ha_wal_ack_send_hook is set (XCE_TESTING), in which case the hook is
* called with the built {term, lsn, success} INSTEAD of sending over ep --
* lets a unit test observe a NACK (success=0) without a live endpoint
* (bug-1479). Task 3.1 (ha-wal-replication): ACK/NACK each applied/rejected
* WAL entry. */
static void ha_send_wal_ack_reply_ex(ha_state_t *ha, struct endpoint_t *ep,
cluster_t *c, uint64_t lsn, int success)
{
uint64_t term = (uint64_t)atomic_load_explicit(&ha->term, memory_order_acquire);
#ifdef XCE_TESTING
if (ha_wal_ack_send_hook) {
ha_wal_ack_send_hook(term, lsn, success);
return;
}
#endif
if (!ep || !c)
return;
wal_ack_reply_t ack;
ack.term = (uint64_t)atomic_load_explicit(&ha->term, memory_order_acquire);
ack.term = term;
ack.lsn = lsn;
ack.success = 1;
ack.success = success ? 1 : 0;
xce_reply_t reply;
memset(&reply, 0, sizeof(reply));
@ -3105,6 +3122,14 @@ static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
xce_reply(ep, c->myhost, &reply);
}
/* ha_send_wal_ack_reply: send a successful WalAckReply {term, lsn,
* success=1} -- the common case, every existing call site. */
static void ha_send_wal_ack_reply(ha_state_t *ha, struct endpoint_t *ep,
cluster_t *c, uint64_t lsn)
{
ha_send_wal_ack_reply_ex(ha, ep, c, lsn, /*success*/1);
}
/* ha_apply_term_entry_via_library: route ONE term-carrying replicated entry
* (a live broadcast or a catch-up sub-entry) through the raft library
* (raft_on_append_entries_request via ha_consume_append_entries) so the LIBRARY
@ -3227,8 +3252,18 @@ static int ha_apply_term_entry_via_library(ha_state_t *ha, persist_t *p,
ha_send_wal_ack_reply(ha, ep, c, e->lsn);
return 1;
}
/* logOk reject (e.g. forward gap before the prefix arrives): no ACK; the
* ACK-driven catch-up continuation re-streams from current_lsn in order. */
/* logOk reject (e.g. forward gap before the prefix arrives, or a genuine
* (term,lsn) conflict): bug-1479 -- send a NACK (success=0) so the
* primary's ha_raft_on_repl_nack can back off nextIndex and immediately
* mark this standby for catch-up, matching raft.tla's AppendEntries
* action (the receiver ALWAYS replies, success = logOk). Before this fix
* NOTHING was sent on reject: the primary's NACK-handling branch
* (repl_ack_decode_and_dispatch) was correct but unreachable, since no
* production path ever produced the success=0 reply it was waiting for
* -- the standby would loop rejecting the identical re-broadcast entry
* forever, discovered live via the config-upgrade soak's force-elect
* rotation. */
ha_send_wal_ack_reply_ex(ha, ep, c, e->lsn, /*success*/0);
return 0;
}
@ -3843,6 +3878,20 @@ uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
return committed_lsn;
}
/* bug-1479 (gap 3): see full doc comment in ha.h. Distinct from
* ha_compute_repl_target_lsn above (the CONVERGENCE target) -- this is the
* STREAM ceiling: how far a leader may actually push WAL to a standby, which
* must extend past committed_lsn or a standby whose own ack is required to
* advance committed_lsn can never receive the entries it needs to ack. */
uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
uint64_t last_written,
uint64_t checkpoint_lsn)
{
if (last_written == checkpoint_lsn && last_written > committed_lsn)
last_written = committed_lsn;
return (last_written > committed_lsn) ? last_written : committed_lsn;
}
/* ───────────────────────────────────────────────────────────────────────
* raft-drive-async Task 3.2 — SnapshotSend slow-lane offload (bug-1198).
*
@ -4233,8 +4282,38 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
uint64_t standby_tag = ha->standby_reported_tag[peer_idx];
uint64_t standby_lsn = ha->standby_reported_lsn[peer_idx];
/* bug-1479: a standby marked in_catch_up (via ha_raft_on_repl_nack's
* immediate trigger on log-conflict rejection, or repl_broadcast's
* MAX_PENDING_ACKS eviction) can never correct its OWN self-reported
* standby_reported_lsn — it's stuck rejecting every AppendEntries, so it
* never sends a stateReport reflecting genuine progress. Use the
* corrected join_applied_lsn instead: this heartbeat-driven cycle (runs
* every heartbeat_interval_ms regardless of ACK/reject outcome) becomes
* the actual catch-up driver for this case — the ACK-driven continuation
* (ha_walack_on_ack, config_repl.c) can only fire on a REAL ack, which a
* log-conflicted standby can never produce until streamed from the
* corrected position at least once. */
if (c) {
uint64_t join_lsn;
if (repl_get_catchup_join_lsn(&c->persist.repl, ep, &join_lsn))
standby_lsn = join_lsn;
}
/* bug-1479 (gap 3): stream (and evaluate convergence against) the
* leader's own log tail, not just committed_lsn -- see
* ha_compute_repl_stream_ceiling_lsn's doc comment (ha.h) for why capping
* at committed_lsn is a permanent deadlock whenever this exact standby's
* own ack is required to advance committed_lsn in the first place. */
uint64_t stream_ceiling_lsn = committed_lsn;
if (c) {
uint64_t last_written = (c->persist.current_lsn > 0)
? c->persist.current_lsn - 1 : 0;
stream_ceiling_lsn = ha_compute_repl_stream_ceiling_lsn(
committed_lsn, last_written, c->persist.checkpoint_lsn);
}
ha_repl_action_t action = ha_evaluate_repl_action(standby_tag, standby_lsn,
committed_tag, committed_lsn,
committed_tag, stream_ceiling_lsn,
oldest_wal_lsn);
switch (action) {
@ -4283,16 +4362,19 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
case HA_REPL_STREAM_WAL:
/* Task 3.4: batch WAL streaming.
* Enqueues all batch OPCode_walCatchupEntry messages for the range
* [standby_lsn+1, committed_lsn] via ha_stream_wal_to_standby().
* [standby_lsn+1, stream_ceiling_lsn] via ha_stream_wal_to_standby()
* (bug-1479 gap 3: the ceiling extends past committed_lsn to the
* leader's own log tail — see ha_compute_repl_stream_ceiling_lsn).
* All xce_request() calls are non-blocking (enqueue-only). */
INFO("HA repl: standby[%d] lsn=%" PRIu64
" < committed_lsn=%" PRIu64 " → stream WAL batches",
peer_idx, standby_lsn, committed_lsn);
" < stream_ceiling_lsn=%" PRIu64 " (committed_lsn=%" PRIu64
") → stream WAL batches",
peer_idx, standby_lsn, stream_ceiling_lsn, committed_lsn);
#ifdef XCE_TESTING
if (ha_repl_stream_wal_hook != NULL) {
const char *wal_path = (c && c->persist.wal_path[0])
? c->persist.wal_path : "";
ha_repl_stream_wal_hook(ep, wal_path, standby_lsn, committed_lsn);
ha_repl_stream_wal_hook(ep, wal_path, standby_lsn, stream_ceiling_lsn);
break;
}
#endif
@ -4311,7 +4393,7 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
uint64_t leader_term =
atomic_load_explicit(&c->ha.term, memory_order_acquire);
ha_stream_wal_to_standby(ep, c->persist.wal_path, &c->persist,
leader_term, standby_lsn, committed_lsn);
leader_term, standby_lsn, stream_ceiling_lsn);
}
break;
@ -4820,6 +4902,38 @@ int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
return 0;
}
int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
uint64_t *corrected_lsn_out)
{
if (!c)
return -EINVAL;
/* Same defense-in-depth bounds/self check as ha_raft_on_repl_ack. */
int self_idx = c->ha.my_priority;
int n_peers = c->ha.raft.n_peers;
if (abs_peer_idx < 0 || abs_peer_idx >= n_peers || abs_peer_idx == self_idx) {
ERROR("ha_raft_on_repl_nack: rejecting out-of-range peer index %d "
"(n_peers=%d self_idx=%d) — skipping library NACK accounting",
abs_peer_idx, n_peers, self_idx);
return -EINVAL;
}
/* A rejected AppendEntries reports a log conflict at the sent
* prevLogIndex/prevLogTerm. match_index is unused on the failure branch
* (raft.c: "failure: next_index[peer] = Max(next_index[peer]-1, 1). Does
* NOT advance commit.") -- pass 0. The library applies the backoff only
* when reply_term == s->term (stale-term drop), same as the ack path. */
(void)raft_on_append_entries_reply(&c->ha.raft, abs_peer_idx, /*success*/0,
/*match_index unused*/0, reply_term);
if (corrected_lsn_out) {
uint64_t next = c->ha.raft.next_index[abs_peer_idx];
*corrected_lsn_out = (next > 0) ? next - 1 : 0;
}
return 0;
}
/*
* ha_arm_own_term_gate: append the own-term WAL_NOOP and arm the commit
* engine's writable_lsn gate (ha-single-leader-role task 2.2, Req 1.4/2.1/2.2).
@ -6638,8 +6752,36 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx,
pthread_mutex_lock(&ha->raft_lock);
(void)raft_on_higher_term_observed(&ha->raft, reply_term);
pthread_mutex_unlock(&ha->raft_lock);
/* bug-1479: a same-or-lower-term NACK means an ordinary Raft
* log conflict (ae_reject_logmismatch), NOT a stale-term
* primary — the case above doesn't cover it. Before this fix
* the reject was silently dropped: the library's own,
* already-correct nextIndex backoff (raft_on_append_entries_
* reply) was never invoked, so the primary kept re-
* broadcasting NEW entries assuming agreement at an index
* the standby had already rejected, forever — the standby
* would eventually get silently evicted via
* MAX_PENDING_ACKS, but nothing would ever stream it back
* into sync (repl_start_catchup's own driver, the ACK-driven
* continuation, needs a real ACK to fire, and a log-
* conflicted standby can never send one while stuck).
* Immediately (not after 16 silent failures) back off via
* the library and mark catch-up at the corrected position —
* ha_dispatch_repl_action_for_standby's heartbeat-driven
* decision (below) is what actually streams to it. */
if (c) {
int ack_abs_idx = ha_repl_ack_abs_peer_idx(
&c->ha, /*ep_is_vote*/ vctx->ctx != NULL, vctx->peer_idx);
uint64_t corrected_lsn = 0;
if (ha_raft_on_repl_nack(c, ack_abs_idx, reply_term,
&corrected_lsn) == 0) {
repl_start_catchup(&c->persist.repl, ep, corrected_lsn);
}
}
WARN("repl_ack_decode_and_dispatch: WAL NACK from standby"
" lsn=%lu success=0 — ignoring", (unsigned long)acked_lsn);
" lsn=%lu success=0 — marking catch-up",
(unsigned long)acked_lsn);
}
} else if (rep_s.data_which == RequestReply_data_replconfigpushreplymsg) {
/* Config-push ACK from standby (ha-commit-barrier-rollout task