lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

88
src/controller/testsuite/test_config_repl.c
View file

@ -3922,6 +3922,93 @@ static void test_wal_catchup_entry_error_skipped_no_crash(void)
printf("test_wal_catchup_entry_error_skipped_no_crash: PASS\n");
}
/* bug-1479: a term-carrying entry that fails logOk (forward gap / genuine
* (term,lsn) conflict) must now send a NACK {success=0} back to the primary,
* not silence the reject entirely -- otherwise the primary's NACK-handling
* (ha_raft_on_repl_nack, ha.c) can never engage, no matter how correct it is,
* because nothing ever produces the reply it is waiting for. This is exactly
* the live gap the config-upgrade soak's force-elect rotation exposed: a
* standby that fell behind (a genuine forward gap against its empty log)
* looped rejecting the identical re-broadcast entry forever, with no NACK
* ever reaching the primary to trigger backoff + catch-up.
*
* ha_wal_ack_send_hook (ha.h, XCE_TESTING) captures the {term, lsn, success}
* that ha_send_wal_ack_reply_ex would have sent, without a live endpoint. */
static uint64_t g_b1479_hook_term;
static uint64_t g_b1479_hook_lsn;
static int g_b1479_hook_success;
static int g_b1479_hook_called;
static void b1479_wal_ack_capture(uint64_t term, uint64_t lsn, int success)
{
g_b1479_hook_called = 1;
g_b1479_hook_term = term;
g_b1479_hook_lsn = lsn;
g_b1479_hook_success = success;
}
static void test_bug1479_ae_reject_sends_nack_not_silence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr_b1479_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
/* Bring the node up as a raft FOLLOWER at term 3 with an EMPTY log
* (log_len=0) -- mirrors t_bug1475_make_follower further below in this
* file (that fixture is defined later; duplicated inline here to avoid
* an ordering dependency). */
atomic_store(&c->ha.role, CTRL_STANDBY);
atomic_store(&c->ha.term, 3);
ha_init_raft(&c->ha, c, /*n_peers*/3);
c->ha.raft.term = 3;
c->ha.raft.role = RAFT_FOLLOWER;
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len);
g_b1479_hook_called = 0;
ha_wal_ack_send_hook = b1479_wal_ack_capture;
/* A term-carrying entry at lsn=5 whose prev_log_index=4 is a forward gap
* against this node's empty log (0 < 4) -- logOk MUST reject it. */
wal_catchup_entry_t inner;
memset(&inner, 0, sizeof(inner));
inner.lsn = 5;
inner.term = 3;
inner.prev_log_index = 4;
inner.prev_log_term = 3;
inner.entry_type = (uint8_t)WAL_JOB_UPDATE;
wal_catchup_entry_t *sub_arr[1] = { &inner };
wal_catchup_entry_t outer;
memset(&outer, 0, sizeof(outer));
outer.n_entries = 1;
outer.entries = sub_arr;
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &outer,
/*ep*/NULL, c);
ha_wal_ack_send_hook = NULL; /* uninstall before any assertion can fail */
TEST_ASSERT_EQUAL_INT(0, rc); /* reject, not a hard error */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len); /* nothing was accepted */
TEST_ASSERT_TRUE_MESSAGE(g_b1479_hook_called,
"bug-1479: an AppendEntries reject must send a WalAckReply "
"(a NACK), not silence it entirely");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_b1479_hook_success,
"the reply for a rejected entry must carry success=0");
TEST_ASSERT_EQUAL_UINT64(3, g_b1479_hook_term);
TEST_ASSERT_EQUAL_UINT64(5, g_b1479_hook_lsn);
t42_teardown(c);
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
system(cmd);
printf("test_bug1479_ae_reject_sends_nack_not_silence: PASS\n");
}
/* Test 2.1-C: NULL entry is handled gracefully (null guard) */
static void test_wal_catchup_entry_null_entry_no_crash(void)
{
@ -6764,6 +6851,7 @@ void test_config_repl(void)
/* Task 2.1: OPCode_walCatchupEntry handler */
RUN_TEST(test_wal_catchup_entry_advances_applied_lsn);
RUN_TEST(test_wal_catchup_entry_error_skipped_no_crash);
RUN_TEST(test_bug1479_ae_reject_sends_nack_not_silence);
RUN_TEST(test_wal_catchup_entry_null_entry_no_crash);
/* Task 2.2: batch WAL catch-up entries */
RUN_TEST(test_wal_catchup_batch_advances_applied_lsn);

60
src/controller/testsuite/test_ha_role.c
View file

@ -718,6 +718,61 @@ static void test_compute_repl_target_lsn_committed_already_ahead(void)
printf("test_compute_repl_target_lsn_committed_already_ahead: PASS\n");
}
/* ── bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn ─────────────────
*
* Distinct from ha_compute_repl_target_lsn above (the CONVERGENCE target,
* bootstrap-only fallback). This is the STREAM ceiling: a live 3-node HA
* soak (config-upgrade-snapshot task 8.6, force-elect round) caught
* committed_lsn-bounded streaming in a permanent deadlock -- with one member
* down and the other standby log-conflicted after a chaotic election, that
* standby's own ack is the ONLY path to a new majority, but it can never
* receive any entry past committed_lsn to ack in the first place, since
* ha_stream_wal_to_standby refuses to send anything beyond it. The ceiling
* must extend to the leader's real log tail so catch-up can make progress
* past a stalled commit point, while still excluding a trailing local-only
* checkpoint LSN (the exact bug-1478 trap) since no standby can ever
* legitimately ack that exact slot either. */
static void test_compute_repl_stream_ceiling_lsn_widens_past_committed(void)
{
/* The core bug-1479 gap-3 fix: committed_lsn is stalled at 14, but the
* leader has real, streamable entries up to 33 -- the ceiling must
* widen to 33, not stay capped at 14. */
TEST_ASSERT_EQUAL_UINT64(33,
ha_compute_repl_stream_ceiling_lsn(14, 33, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_widens_past_committed: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint(void)
{
/* bug-1478's exact trap, reintroduced here if unguarded: last_written
* (15) IS the trailing local-only checkpoint's own consumed LSN slot
* (checkpoint_lsn=15) -- no standby can ever legitimately ack that exact
* value, so the ceiling must fall back to committed_lsn (14), not widen
* to the unreachable checkpoint slot. */
TEST_ASSERT_EQUAL_UINT64(14,
ha_compute_repl_stream_ceiling_lsn(14, 15, /*checkpoint_lsn*/15));
printf("test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_committed_already_ahead(void)
{
/* Normal steady state: committed_lsn is already the higher value (a
* standby that hasn't caught up to a recently-advanced commit). The
* ceiling must simply be committed_lsn, never go backward. */
TEST_ASSERT_EQUAL_UINT64(100,
ha_compute_repl_stream_ceiling_lsn(100, 50, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_committed_already_ahead: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_zero_zero(void)
{
/* Genuinely fresh cluster, nothing appended yet. */
TEST_ASSERT_EQUAL_UINT64(0,
ha_compute_repl_stream_ceiling_lsn(0, 0, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_zero_zero: PASS\n");
}
/* The transitional-leader timeout default test was removed: the config knob
* and its default macro are deleted by ha-single-leader-role (the wall-clock
* force-promotion watchdog it configured is gone — Req 1.5). */
@ -955,6 +1010,11 @@ void test_ha_role(void)
RUN_TEST(test_compute_repl_target_lsn_zero_zero);
RUN_TEST(test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn);
RUN_TEST(test_compute_repl_target_lsn_committed_already_ahead);
/* bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn */
RUN_TEST(test_compute_repl_stream_ceiling_lsn_widens_past_committed);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_committed_already_ahead);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_zero_zero);
/* Task 3.3: ha_dispatch_repl_action_for_standby PUSH_CONFIG wiring */
RUN_TEST(test_dispatch_repl_action_push_config_when_tag_behind);
RUN_TEST(test_dispatch_repl_action_no_push_when_converged);

180
src/controller/testsuite/test_term_safety_adapter.c
View file

@ -1983,6 +1983,181 @@ static void test_repl_ack_out_of_range_peer_rejected(void)
printf("test_repl_ack_out_of_range_peer_rejected: PASS\n");
}
/* ================================================================== */
/* bug-1479 — drive the library on a REJECTED AppendEntries (WAL NACK) */
/* ================================================================== */
/* Before this fix, a WAL NACK (WalAckReply.success == 0) for an ordinary
* Raft log conflict (ae_reject_logmismatch) was silently dropped after
* stale-term discovery -- the library's own, already-correct nextIndex
* backoff (raft_on_append_entries_reply's failure branch, raft.tla 401) was
* never invoked, so a standby stuck in log conflict never had its resend
* position corrected: the primary kept broadcasting new entries assuming
* agreement at an index the standby had already rejected, forever, until
* MAX_PENDING_ACKS silently evicted it into catch-up -- at which point
* NOTHING streamed it back into sync either (the ACK-driven continuation
* needs a real ACK to fire, which a log-conflicted standby can never send).
*
* ha_raft_on_repl_nack mirrors ha_raft_on_repl_ack's bounds-checked
* peer-index resolution, but routes success=0 into the library (backing
* off next_index[peer] by one, per raft.c's documented failure branch) and
* returns the corrected resume LSN instead of trying to advance commit. */
/* (a) A rejection backs off next_index[peer] by exactly one and returns the
* corrected LSN (next_index - 1) -- the position the standby is now
* believed caught up through. */
static void test_repl_nack_backs_off_next_index_and_returns_corrected_lsn(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_backoff_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
/* make_leader pokes role=RAFT_LEADER directly without going through the
* library's real BecomeLeader transition (raft_init_leader_progress,
* raft.c:97-103), so next_index[] stays 0 for every peer regardless of
* log_len. Seed it here to what a genuine BecomeLeader would have set
* (log_len + 1), so this test exercises a realistic starting position. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
TEST_ASSERT_EQUAL_UINT64(6, c->ha.raft.next_index[1]);
uint64_t corrected_lsn = 999; /* sentinel -- must be overwritten */
int rc = ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn);
TEST_ASSERT_EQUAL_INT(0, rc);
/* next_index backed off by exactly one (raft.c: Max(next_index-1, 1)). */
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.next_index[1]);
/* Corrected resume point is next_index - 1. */
TEST_ASSERT_EQUAL_UINT64(4, corrected_lsn);
/* A rejection never advances commit. */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.commit_index);
teardown_cluster(c);
printf("test_repl_nack_backs_off_next_index_and_returns_corrected_lsn: PASS\n");
}
/* (b) Repeated rejections keep backing off, floored at 1 -- never underflows
* (raft.c's Max(next_index-1, 1), matching raft.tla exactly). */
static void test_repl_nack_repeated_rejections_floor_at_one(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_floor_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] to what a real BecomeLeader would set (log_len+1 =
* 3) -- make_leader itself leaves it at 0 (see the backoff test above).
* Reject enough times to walk it down to 1 and confirm it never goes
* below. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
for (int i = 0; i < 5; i++) {
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn));
}
TEST_ASSERT_EQUAL_UINT64(1, c->ha.raft.next_index[1]);
TEST_ASSERT_EQUAL_UINT64(0, corrected_lsn);
teardown_cluster(c);
printf("test_repl_nack_repeated_rejections_floor_at_one: PASS\n");
}
/* (c) Defense-in-depth: same out-of-range/self rejection as the ACK seam --
* no library accounting on a bogus peer index. */
static void test_repl_nack_out_of_range_peer_rejected(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_oob_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] as a real BecomeLeader would (log_len+1 = 3); see
* the backoff test above for why make_leader itself leaves it at 0. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/3, /*reply_term*/T, &corrected_lsn));
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/0, /*reply_term*/T, &corrected_lsn));
/* next_index untouched by either rejected call. */
TEST_ASSERT_EQUAL_UINT64(3, c->ha.raft.next_index[1]);
teardown_cluster(c);
printf("test_repl_nack_out_of_range_peer_rejected: PASS\n");
}
/* (d) THE CORE REGRESSION: composing ha_raft_on_repl_nack + repl_start_catchup
* -- exactly what repl_ack_decode_and_dispatch's NACK branch now does --
* marks the standby in_catch_up at the CORRECTED position on the very FIRST
* rejection. Before this fix, nothing marked it at all until repl_broadcast's
* own MAX_PENDING_ACKS=16 eviction fired (silently, using the standby's last
* ACKED lsn rather than a library-backed-off position) -- and even then,
* nothing would ever stream to it (the ACK-driven continuation requires a
* real ACK a log-conflicted standby can never send). This test proves the
* immediate half of the fix: no 16-strike wait, and the resume point comes
* from the library's own nextIndex backoff, not a guess. */
static void test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_catchup_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
TEST_ASSERT_EQUAL_INT(0,
repl_add_standby(&c->persist.repl, &g_ae_dummy_ep, "sb0", 7000));
uint64_t join_lsn_before;
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_before),
"a freshly-added standby must not already be in catch-up");
/* THE FIRST rejection -- not the 16th. */
uint64_t corrected_lsn = 999;
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T, &corrected_lsn));
repl_start_catchup(&c->persist.repl, &g_ae_dummy_ep, corrected_lsn);
uint64_t join_lsn_after;
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_after),
"a single rejection must immediately mark the standby in catch-up "
"(bug-1479 -- no 16-strike wait)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(corrected_lsn, join_lsn_after,
"the catch-up resume point must be the library's backed-off "
"next_index - 1, not a guess");
teardown_cluster(c);
printf("test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes: PASS\n");
}
/* ================================================================== */
/* Task 5.1 — every-callback + marker-path end-to-end integration */
/* ================================================================== */
@ -2209,6 +2384,11 @@ void test_term_safety_adapter(void)
RUN_TEST(test_repl_ack_abs_peer_idx_endpoint_type_roundtrip);
RUN_TEST(test_repl_ack_heartbeat_standby_absolute_slot);
RUN_TEST(test_repl_ack_out_of_range_peer_rejected);
/* bug-1479: NACK seam drives the library's nextIndex backoff. */
RUN_TEST(test_repl_nack_backs_off_next_index_and_returns_corrected_lsn);
RUN_TEST(test_repl_nack_repeated_rejections_floor_at_one);
RUN_TEST(test_repl_nack_out_of_range_peer_rejected);
RUN_TEST(test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes);
/* Task 5.1: every-callback + marker-path end-to-end integration. */
RUN_TEST(test_every_callback_and_marker_path_end_to_end);
}