lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

180
src/controller/testsuite/test_term_safety_adapter.c
View file

@ -1983,6 +1983,181 @@ static void test_repl_ack_out_of_range_peer_rejected(void)
printf("test_repl_ack_out_of_range_peer_rejected: PASS\n");
}
/* ================================================================== */
/* bug-1479 — drive the library on a REJECTED AppendEntries (WAL NACK) */
/* ================================================================== */
/* Before this fix, a WAL NACK (WalAckReply.success == 0) for an ordinary
* Raft log conflict (ae_reject_logmismatch) was silently dropped after
* stale-term discovery -- the library's own, already-correct nextIndex
* backoff (raft_on_append_entries_reply's failure branch, raft.tla 401) was
* never invoked, so a standby stuck in log conflict never had its resend
* position corrected: the primary kept broadcasting new entries assuming
* agreement at an index the standby had already rejected, forever, until
* MAX_PENDING_ACKS silently evicted it into catch-up -- at which point
* NOTHING streamed it back into sync either (the ACK-driven continuation
* needs a real ACK to fire, which a log-conflicted standby can never send).
*
* ha_raft_on_repl_nack mirrors ha_raft_on_repl_ack's bounds-checked
* peer-index resolution, but routes success=0 into the library (backing
* off next_index[peer] by one, per raft.c's documented failure branch) and
* returns the corrected resume LSN instead of trying to advance commit. */
/* (a) A rejection backs off next_index[peer] by exactly one and returns the
* corrected LSN (next_index - 1) -- the position the standby is now
* believed caught up through. */
static void test_repl_nack_backs_off_next_index_and_returns_corrected_lsn(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_backoff_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
/* make_leader pokes role=RAFT_LEADER directly without going through the
* library's real BecomeLeader transition (raft_init_leader_progress,
* raft.c:97-103), so next_index[] stays 0 for every peer regardless of
* log_len. Seed it here to what a genuine BecomeLeader would have set
* (log_len + 1), so this test exercises a realistic starting position. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
TEST_ASSERT_EQUAL_UINT64(6, c->ha.raft.next_index[1]);
uint64_t corrected_lsn = 999; /* sentinel -- must be overwritten */
int rc = ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn);
TEST_ASSERT_EQUAL_INT(0, rc);
/* next_index backed off by exactly one (raft.c: Max(next_index-1, 1)). */
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.next_index[1]);
/* Corrected resume point is next_index - 1. */
TEST_ASSERT_EQUAL_UINT64(4, corrected_lsn);
/* A rejection never advances commit. */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.commit_index);
teardown_cluster(c);
printf("test_repl_nack_backs_off_next_index_and_returns_corrected_lsn: PASS\n");
}
/* (b) Repeated rejections keep backing off, floored at 1 -- never underflows
* (raft.c's Max(next_index-1, 1), matching raft.tla exactly). */
static void test_repl_nack_repeated_rejections_floor_at_one(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_floor_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] to what a real BecomeLeader would set (log_len+1 =
* 3) -- make_leader itself leaves it at 0 (see the backoff test above).
* Reject enough times to walk it down to 1 and confirm it never goes
* below. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
for (int i = 0; i < 5; i++) {
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn));
}
TEST_ASSERT_EQUAL_UINT64(1, c->ha.raft.next_index[1]);
TEST_ASSERT_EQUAL_UINT64(0, corrected_lsn);
teardown_cluster(c);
printf("test_repl_nack_repeated_rejections_floor_at_one: PASS\n");
}
/* (c) Defense-in-depth: same out-of-range/self rejection as the ACK seam --
* no library accounting on a bogus peer index. */
static void test_repl_nack_out_of_range_peer_rejected(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_oob_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] as a real BecomeLeader would (log_len+1 = 3); see
* the backoff test above for why make_leader itself leaves it at 0. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/3, /*reply_term*/T, &corrected_lsn));
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/0, /*reply_term*/T, &corrected_lsn));
/* next_index untouched by either rejected call. */
TEST_ASSERT_EQUAL_UINT64(3, c->ha.raft.next_index[1]);
teardown_cluster(c);
printf("test_repl_nack_out_of_range_peer_rejected: PASS\n");
}
/* (d) THE CORE REGRESSION: composing ha_raft_on_repl_nack + repl_start_catchup
* -- exactly what repl_ack_decode_and_dispatch's NACK branch now does --
* marks the standby in_catch_up at the CORRECTED position on the very FIRST
* rejection. Before this fix, nothing marked it at all until repl_broadcast's
* own MAX_PENDING_ACKS=16 eviction fired (silently, using the standby's last
* ACKED lsn rather than a library-backed-off position) -- and even then,
* nothing would ever stream to it (the ACK-driven continuation requires a
* real ACK a log-conflicted standby can never send). This test proves the
* immediate half of the fix: no 16-strike wait, and the resume point comes
* from the library's own nextIndex backoff, not a guess. */
static void test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_catchup_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
TEST_ASSERT_EQUAL_INT(0,
repl_add_standby(&c->persist.repl, &g_ae_dummy_ep, "sb0", 7000));
uint64_t join_lsn_before;
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_before),
"a freshly-added standby must not already be in catch-up");
/* THE FIRST rejection -- not the 16th. */
uint64_t corrected_lsn = 999;
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T, &corrected_lsn));
repl_start_catchup(&c->persist.repl, &g_ae_dummy_ep, corrected_lsn);
uint64_t join_lsn_after;
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_after),
"a single rejection must immediately mark the standby in catch-up "
"(bug-1479 -- no 16-strike wait)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(corrected_lsn, join_lsn_after,
"the catch-up resume point must be the library's backed-off "
"next_index - 1, not a guess");
teardown_cluster(c);
printf("test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes: PASS\n");
}
/* ================================================================== */
/* Task 5.1 — every-callback + marker-path end-to-end integration */
/* ================================================================== */
@ -2209,6 +2384,11 @@ void test_term_safety_adapter(void)
RUN_TEST(test_repl_ack_abs_peer_idx_endpoint_type_roundtrip);
RUN_TEST(test_repl_ack_heartbeat_standby_absolute_slot);
RUN_TEST(test_repl_ack_out_of_range_peer_rejected);
/* bug-1479: NACK seam drives the library's nextIndex backoff. */
RUN_TEST(test_repl_nack_backs_off_next_index_and_returns_corrected_lsn);
RUN_TEST(test_repl_nack_repeated_rejections_floor_at_one);
RUN_TEST(test_repl_nack_out_of_range_peer_rejected);
RUN_TEST(test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes);
/* Task 5.1: every-callback + marker-path end-to-end integration. */
RUN_TEST(test_every_callback_and_marker_path_end_to_end);
}