lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Browse source 
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-07-29 04:11:32 +00:00
parent aecd8766f4
commit 34ab78d72d
14 changed files with 1256 additions and 377 deletions
Download patch file Download diff file Expand all files Collapse all files

88
src/controller/testsuite/test_config_repl.c
View file

@ -3922,6 +3922,93 @@ static void test_wal_catchup_entry_error_skipped_no_crash(void)
printf("test_wal_catchup_entry_error_skipped_no_crash: PASS\n");
}
/* bug-1479: a term-carrying entry that fails logOk (forward gap / genuine
* (term,lsn) conflict) must now send a NACK {success=0} back to the primary,
* not silence the reject entirely -- otherwise the primary's NACK-handling
* (ha_raft_on_repl_nack, ha.c) can never engage, no matter how correct it is,
* because nothing ever produces the reply it is waiting for. This is exactly
* the live gap the config-upgrade soak's force-elect rotation exposed: a
* standby that fell behind (a genuine forward gap against its empty log)
* looped rejecting the identical re-broadcast entry forever, with no NACK
* ever reaching the primary to trigger backoff + catch-up.
*
* ha_wal_ack_send_hook (ha.h, XCE_TESTING) captures the {term, lsn, success}
* that ha_send_wal_ack_reply_ex would have sent, without a live endpoint. */
static uint64_t g_b1479_hook_term;
static uint64_t g_b1479_hook_lsn;
static int g_b1479_hook_success;
static int g_b1479_hook_called;
static void b1479_wal_ack_capture(uint64_t term, uint64_t lsn, int success)
{
g_b1479_hook_called = 1;
g_b1479_hook_term = term;
g_b1479_hook_lsn = lsn;
g_b1479_hook_success = success;
}
static void test_bug1479_ae_reject_sends_nack_not_silence(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr_b1479_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
/* Bring the node up as a raft FOLLOWER at term 3 with an EMPTY log
* (log_len=0) -- mirrors t_bug1475_make_follower further below in this
* file (that fixture is defined later; duplicated inline here to avoid
* an ordering dependency). */
atomic_store(&c->ha.role, CTRL_STANDBY);
atomic_store(&c->ha.term, 3);
ha_init_raft(&c->ha, c, /*n_peers*/3);
c->ha.raft.term = 3;
c->ha.raft.role = RAFT_FOLLOWER;
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len);
g_b1479_hook_called = 0;
ha_wal_ack_send_hook = b1479_wal_ack_capture;
/* A term-carrying entry at lsn=5 whose prev_log_index=4 is a forward gap
* against this node's empty log (0 < 4) -- logOk MUST reject it. */
wal_catchup_entry_t inner;
memset(&inner, 0, sizeof(inner));
inner.lsn = 5;
inner.term = 3;
inner.prev_log_index = 4;
inner.prev_log_term = 3;
inner.entry_type = (uint8_t)WAL_JOB_UPDATE;
wal_catchup_entry_t *sub_arr[1] = { &inner };
wal_catchup_entry_t outer;
memset(&outer, 0, sizeof(outer));
outer.n_entries = 1;
outer.entries = sub_arr;
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &outer,
/*ep*/NULL, c);
ha_wal_ack_send_hook = NULL; /* uninstall before any assertion can fail */
TEST_ASSERT_EQUAL_INT(0, rc); /* reject, not a hard error */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.log_len); /* nothing was accepted */
TEST_ASSERT_TRUE_MESSAGE(g_b1479_hook_called,
"bug-1479: an AppendEntries reject must send a WalAckReply "
"(a NACK), not silence it entirely");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_b1479_hook_success,
"the reply for a rejected entry must carry success=0");
TEST_ASSERT_EQUAL_UINT64(3, g_b1479_hook_term);
TEST_ASSERT_EQUAL_UINT64(5, g_b1479_hook_lsn);
t42_teardown(c);
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
system(cmd);
printf("test_bug1479_ae_reject_sends_nack_not_silence: PASS\n");
}
/* Test 2.1-C: NULL entry is handled gracefully (null guard) */
static void test_wal_catchup_entry_null_entry_no_crash(void)
{
@ -6764,6 +6851,7 @@ void test_config_repl(void)
/* Task 2.1: OPCode_walCatchupEntry handler */
RUN_TEST(test_wal_catchup_entry_advances_applied_lsn);
RUN_TEST(test_wal_catchup_entry_error_skipped_no_crash);
RUN_TEST(test_bug1479_ae_reject_sends_nack_not_silence);
RUN_TEST(test_wal_catchup_entry_null_entry_no_crash);
/* Task 2.2: batch WAL catch-up entries */
RUN_TEST(test_wal_catchup_batch_advances_applied_lsn);