wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(ha): retry cluster_init_phase2 on failure instead of latching phase2_done unconditionally (bug-1477)

Browse source 
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.

Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).

TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
This commit is contained in:
Rongsong Shen 2026-07-28 13:25:48 +00:00
parent 102e263cce
commit f1e54f3ba0
6 changed files with 384 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

139
src/controller/testsuite/test_config_repl.c
View file

@ -2228,6 +2228,141 @@ static void test_commit_advance_wiring_calls_upgrade_materialization_schedule_re
printf("test_commit_advance_wiring_calls_upgrade_materialization_schedule_retry: PASS\n");
}
/* ── bug-1477: ha_phase2_retry_fn / ha_schedule_phase2_retry ─────────────────
*
* A node that takes myapp_run's skip_election passive path (bug-849 --
* "leader already known after the startup stagger") makes exactly ONE
* cluster_init_phase2 attempt. Before this fix, EVERY call site (myapp.c's
* skip_election block, raft_cb_become_leader, raft_cb_become_follower,
* ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
* UNCONDITIONALLY, regardless of cluster_init_phase2's return value. If that
* one attempt raced config availability and failed, the node was left
* permanently stuck: commit_engine_init never ran, so apply_committed_entry
* never applied ANY committed WAL entry for the rest of that process's
* lifetime -- and since a skip_election node never again contests an
* election, none of the other four call sites, which only fire via THIS
* node's own raft-library state transitions, ever ran again either.
*
* ha_phase2_retry_fn is the idempotent retry entry (mirrors
* upgrade_materialization_retry_fn's shape); ha_schedule_phase2_retry is its
* delaymgr-scheduled, self-limiting trigger (mirrors
* schedule_upgrade_materialization_retry's no-op guards). */
/* (bug-1477-a) ha_phase2_retry_fn is a no-op once phase2_done is already
* set -- does not re-invoke the hook, returns 0. */
static void test_ha_phase2_retry_fn_noop_when_already_done(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr1477_retry_noop_done_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
c->ha.phase2_done = 1;
TEST_ASSERT_EQUAL_INT(0, ha_phase2_retry_fn(c));
TEST_ASSERT_EQUAL_INT(1, c->ha.phase2_done);
t42_teardown(c);
printf("test_ha_phase2_retry_fn_noop_when_already_done: PASS\n");
}
static _Atomic int g_cr1477_hook_calls;
static _Atomic int g_cr1477_hook_fail_first_n;
/* Fails the first g_cr1477_hook_fail_first_n calls (rc=-EAGAIN), succeeds
* (rc=0) thereafter -- models cluster_init_phase2 racing config
* availability and then succeeding once it arrives. */
static int cr1477_flaky_phase2_hook(void *c)
{
(void)c;
int n = atomic_fetch_add(&g_cr1477_hook_calls, 1);
return (n < atomic_load(&g_cr1477_hook_fail_first_n)) ? -EAGAIN : 0;
}
/* (bug-1477-b) THE CORE REGRESSION: a failed cluster_init_phase2 attempt
* must NOT latch phase2_done -- RED under the pre-fix code (which set
* phase2_done = 1 unconditionally right after calling the hook, regardless
* of its return value; this assertion would fail immediately). A second
* call, once the underlying condition clears, then succeeds and latches. */
static void test_ha_phase2_retry_fn_does_not_latch_on_failure_and_succeeds_on_retry(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr1477_retry_fn_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
TEST_ASSERT_EQUAL_INT(0, c->ha.phase2_done); /* fixture default: not done */
atomic_store(&g_cr1477_hook_calls, 0);
atomic_store(&g_cr1477_hook_fail_first_n, 1);
ha_phase2_hook = cr1477_flaky_phase2_hook;
/* First attempt: hook fails -- phase2_done must stay 0 (bug-1477's own
* regression assertion). */
TEST_ASSERT_EQUAL_INT(-EAGAIN, ha_phase2_retry_fn(c));
TEST_ASSERT_EQUAL_INT_MESSAGE(0, c->ha.phase2_done,
"a failed cluster_init_phase2 attempt must not latch phase2_done "
"(bug-1477 -- the pre-fix code did this unconditionally, "
"permanently stranding the member)");
/* Second attempt: hook now succeeds -- phase2_done latches. */
TEST_ASSERT_EQUAL_INT(0, ha_phase2_retry_fn(c));
TEST_ASSERT_EQUAL_INT(1, c->ha.phase2_done);
/* Idempotent: a third call is a pure no-op (hook not re-invoked). */
int calls_before = atomic_load(&g_cr1477_hook_calls);
TEST_ASSERT_EQUAL_INT(0, ha_phase2_retry_fn(c));
TEST_ASSERT_EQUAL_INT(calls_before, atomic_load(&g_cr1477_hook_calls));
ha_phase2_hook = NULL; /* restore -- shared process-global test seam */
t42_teardown(c);
printf("test_ha_phase2_retry_fn_does_not_latch_on_failure_and_succeeds_on_retry: PASS\n");
}
/* (bug-1477-c) ha_schedule_phase2_retry is a no-op when phase2_done is
* already set -- steady state schedules nothing (no busy-spin), mirroring
* schedule_upgrade_materialization_retry's own documented no-op behavior. */
static void test_schedule_phase2_retry_noop_when_already_done(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr1477_sched_noop_done_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
c->ha.phase2_done = 1;
TEST_ASSERT_FALSE(c->delaymgr.running); /* fixture never starts delaymgr */
TEST_ASSERT_EQUAL_INT(0, ha_schedule_phase2_retry(c));
t42_teardown(c);
printf("test_schedule_phase2_retry_noop_when_already_done: PASS\n");
}
/* (bug-1477-d) ha_schedule_phase2_retry is a no-op when the delaymgr is not
* running, even while phase2_done is still 0 -- the unit-test fixture path
* (mirrors config_schedule_activate_retry / schedule_upgrade_materialization_
* retry's own documented delaymgr-not-running no-op). */
static void test_schedule_phase2_retry_noop_when_delaymgr_not_running(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_cr1477_sched_noop_nodmgr_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
cluster_t *c = t42_make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
TEST_ASSERT_EQUAL_INT(0, c->ha.phase2_done);
TEST_ASSERT_FALSE(c->delaymgr.running);
TEST_ASSERT_EQUAL_INT(0, ha_schedule_phase2_retry(c));
TEST_ASSERT_EQUAL_INT(0, c->ha.phase2_done); /* untouched -- no-op */
t42_teardown(c);
printf("test_schedule_phase2_retry_noop_when_delaymgr_not_running: PASS\n");
}
/* ── config-upgrade-snapshot task 6.2 remediation: Activation Fence set by a
* REAL production trigger, not a direct test call ──────────────────────────
*
@ -6601,6 +6736,10 @@ void test_config_repl(void)
RUN_TEST(test_schedule_upgrade_materialization_retry_noop_when_delaymgr_not_running);
RUN_TEST(test_upgrade_materialization_retry_fn_recovers_from_stale_bytes_fence);
RUN_TEST(test_commit_advance_wiring_calls_upgrade_materialization_schedule_retry);
RUN_TEST(test_ha_phase2_retry_fn_noop_when_already_done);
RUN_TEST(test_ha_phase2_retry_fn_does_not_latch_on_failure_and_succeeds_on_retry);
RUN_TEST(test_schedule_phase2_retry_noop_when_already_done);
RUN_TEST(test_schedule_phase2_retry_noop_when_delaymgr_not_running);
/* config-upgrade-snapshot task 6.2 remediation: fence set by a REAL
* production trigger (apply_committed_entry's WAL_CONFIG_MARKER case),
* not a direct test call -- proves the fence is reachable at all. */