lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): restored-but-stale raft term permanently blocked re-election after restart (bug-1695)

Browse source 
Found via the mandatory live HA chaotic soak for this session's controller
audit campaign -- unrelated to any of the 9 audit fixes (bug-1684/1687/
1689/1690/1691/1692/1693/1694), all in ha.c/persist.c/job.c/cluster.c/
sched.c/worker.c; this is in myapp.c, untouched by any of them.

myapp_run()'s bootstrap "is a leader already known" check had a fallback
`c.ha.raft.term > 0` meant to detect genuine new raft activity processed
during the startup-stagger nap. But ha_restore_term() loads ha->term from
disk BEFORE this check runs, on every boot -- and Raft terms only ever
increase, never reset. So `raft.term > 0` is trivially true on every
restart for the rest of the cluster's on-disk lifetime, once it has ever
held one election, regardless of whether any leader is currently
reachable. Live-reproduced: every node, on every restart after the
cluster's first election, unconditionally skipped ha_start_election_async
forever -- no node could ever re-assert leadership through a real
election again. bug-1477's existing cluster_init_phase2 retry loop kept
firing correctly (confirmed in the reproduction logs) but could never
succeed, since the deeper cause -- no election ever runs to produce a
live primary to push config from -- was never addressed by that fix.

Fixed by snapshotting `term_at_boot` right after ha_restore_term()/
ha_init_raft() seed the term, strictly before the stagger nap, and
comparing `raft.term > term_at_boot` instead of `raft.term > 0` -- only a
term that has genuinely ADVANCED during this boot cycle (real vote/
heartbeat traffic processed) counts as evidence a leader is known. The
decision was extracted into a new, unit-testable myapp_leader_already_known()
(startup_validation.c, since controller_unit_test deliberately excludes
myapp.c/main.c). voted_for_name/vft==cur and primary_ctrl_fd>=0 checks
are unchanged.

7 new tests in test_ha_startup.c; RED/GREEN independently verified.
Rigorous A/B live-soak proof (identical 6-round --chaotic run, fixed vs
git-stashed-back-to-original): baseline showed term frozen for the whole
run and zero election activity on any non-primary node; fixed run showed
term genuinely progressing and ctrl-2 achieving a real, log-verified
CTRL_PRIMARY transition via a won PreVote round -- proof a node other
than the original primary can now take over, which was structurally
impossible before. Full controller suite: 1191 tests, 2 failures -- both
the pre-existing, unrelated bug-1561 overlayfs sandbox flake.

Residual, NOT independently verified yet (flagged honestly, not fixed
here): the fixed run's chaotic verdict showed committed_job_loss=6
(baseline 0) -- traced via waldump to one jobid missing from every node's
WAL, at the cold-boot window before round 1. Resembles the already-fixed
bug-1487 false-positive class (apply-pump-lag racing verify_classify's
expect-job check) but the reported mechanism doesn't perfectly match
(bug-1487's job was present-in-WAL/not-yet-visible; this one is described
as absent from every WAL entirely) -- needs independent re-verification
before being dismissed, tracked as a follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 16:11:37 +00:00
parent 23c5c26a23
commit ac46bfa649
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 792 additions and 444 deletions
Download patch file Download diff file Expand all files Collapse all files

203
src/controller/testsuite/test_ha_startup.c
View file

@ -47,6 +47,12 @@ void ha_start_election_async(ha_state_t *ha, cluster_t *c,
uint64_t my_config_tag, uint64_t my_lsn,
const char *my_name);
/* bug-1695 — myapp_run's post-stagger "leader already known" bootstrap check,
* extracted from myapp.c for unit testability. See myapp.c's doc comment on
* myapp_leader_already_known for the full contract. */
int myapp_leader_already_known(ha_state_t *ha, const char *my_name,
uint64_t term_at_boot);
/* ── Dispatch gate (task 3.3) — does NOT exist yet; linker failure expected ──
*
* ha_dispatch_gate: role-based operation filter for the vtaskmgr dispatch loop.
@ -715,6 +721,192 @@ static void test_nonblocking_election_returns_in_under_10ms(void)
elapsed_ms_val);
}
/* ── bug-1695: myapp_run's "leader already known" bootstrap check ──────────
*
* myapp_leader_already_known(ha, my_name, term_at_boot) is myapp.c's
* extracted, unit-testable decision function (see myapp.c's doc comment on
* myapp_leader_already_known for the full contract). These tests reproduce
* the exact live-soak-confirmed regression: a RESTORED (from ha-term.bin),
* never-since-advanced raft term must NOT be treated as "a leader is known"
* -- only a term that genuinely advances during the startup-stagger nap
* (real raft traffic processed by iomgr) may.
*/
static void test_bug1695_stale_restored_term_does_not_skip_election(void)
{
/* Core regression: a node reboots after the cluster's first-ever
* election. ha_restore_term() + ha_init_raft() would have seeded
* ha->raft.term = 5 (the persisted, historical term) -- and the caller
* snapshots term_at_boot = 5 at that same moment, BEFORE the nap. No new
* raft traffic is processed during the nap (no vote grant, no AE/
* heartbeat observed) -- raft.term is still 5 when this check runs. No
* live primary_ctrl_fd connection exists either (ha_init defaults it to
* -1, left untouched here). This must NOT report "leader known": the
* cluster has no live, reachable primary this node has actually
* observed, so it must proceed to ha_start_election_async instead of
* permanently stalling (bug-1695's exact live-soak symptom: "raft_
* term=1, primary_fd=-1) -- skipping own election", repeating forever
* on every restart). */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
ha.raft.term = 5;
memset(ha.voted_for_name, 0, sizeof(ha.voted_for_name));
ha.voted_for_term = 0;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
5 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, leader_known,
"bug-1695: a merely-restored, never-since-advanced raft term must "
"NOT be treated as evidence a leader is known -- the node must "
"proceed to its own election, not permanently skip_election");
ha_fini(&ha);
printf("test_bug1695_stale_restored_term_does_not_skip_election: PASS\n");
}
static void test_bug1695_term_advanced_during_nap_skips_election(void)
{
/* Correct fast path #1 (must still work): the term genuinely ADVANCED
* during this boot's nap (e.g. iomgr processed a real incoming
* RequestVote/AppendEntries at a higher term while this thread napped).
* This IS real evidence a leader/candidate is active -- skip own
* election. */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
ha.raft.term = 6; /* advanced past term_at_boot=5 during the nap */
memset(ha.voted_for_name, 0, sizeof(ha.voted_for_name));
ha.voted_for_term = 0;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
5 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, leader_known,
"a raft term that genuinely advanced past term_at_boot during the "
"nap must still be treated as evidence a leader/candidate is known");
ha_fini(&ha);
printf("test_bug1695_term_advanced_during_nap_skips_election: PASS\n");
}
static void test_bug1695_live_primary_fd_skips_election_without_term_advance(void)
{
/* Correct fast path #2 (must still work): primary_ctrl_fd is already
* connected (winner's ctrlhello arrived) even though raft.term has not
* advanced past term_at_boot. Must still skip. */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
ha.raft.term = 5; /* no advance */
ha.primary_ctrl_fd = 42; /* simulated live connection; never opened/
closed by this test or ha_fini */
memset(ha.voted_for_name, 0, sizeof(ha.voted_for_name));
ha.voted_for_term = 0;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
5 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, leader_known,
"an already-connected primary_ctrl_fd must still count as leader "
"known, independent of the raft.term fallback");
ha.primary_ctrl_fd = -1;
ha_fini(&ha);
printf("test_bug1695_live_primary_fd_skips_election_without_term_advance: PASS\n");
}
static void test_bug1695_first_boot_zero_term_no_advance_does_not_skip(void)
{
/* First-ever-boot case: no ha-term.bin exists yet, so term_at_boot == 0
* and raft.term == 0 (unadvanced). Must proceed to election exactly as
* before this fix (the old `raft.term > 0` check was already false here
* too) -- confirms the fix does not regress first-boot behavior. */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
ha.raft.term = 0;
memset(ha.voted_for_name, 0, sizeof(ha.voted_for_name));
ha.voted_for_term = 0;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
0 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, leader_known,
"first-ever-boot (term_at_boot=0, raft.term=0) must proceed to its "
"own election, same as before this fix");
ha_fini(&ha);
printf("test_bug1695_first_boot_zero_term_no_advance_does_not_skip: PASS\n");
}
static void test_bug1695_first_boot_term_advances_from_zero_skips(void)
{
/* First-ever-boot, but real vote-request traffic advances the term from
* 0 to 1 during the nap -- must skip, same as before this fix. */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
ha.raft.term = 1;
memset(ha.voted_for_name, 0, sizeof(ha.voted_for_name));
ha.voted_for_term = 0;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
0 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, leader_known,
"first-ever-boot with a genuine term advance (0 -> 1) during the "
"nap must still skip its own election");
ha_fini(&ha);
printf("test_bug1695_first_boot_term_advances_from_zero_skips: PASS\n");
}
static void test_bug1695_voted_for_other_in_current_term_skips_unaffected(void)
{
/* Sanity/no-regression: the voted_for_name/vft==cur fast path is
* untouched by this fix and must keep working -- this node granted a
* vote to a DIFFERENT candidate in the CURRENT term, so it is a
* follower and must skip its own election, even with no raft.term
* advance and no live primary_ctrl_fd. */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
atomic_store(&ha.term, 3ULL);
ha.raft.term = 3;
strncpy(ha.voted_for_name, "ctrl-1", sizeof(ha.voted_for_name) - 1);
ha.voted_for_term = 3;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
3 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, leader_known,
"a vote granted to a different candidate in the current term must "
"still skip own election (untouched by this fix)");
ha_fini(&ha);
printf("test_bug1695_voted_for_other_in_current_term_skips_unaffected: PASS\n");
}
static void test_bug1695_self_vote_does_not_skip_election(void)
{
/* Sanity/no-regression: a self-vote (voted_for_name == own name) must
* NOT count as "leader known" -- it means THIS node was the candidate,
* not that an external leader was observed. With no term advance and no
* live primary_ctrl_fd, this must proceed to its own election. */
ha_state_t ha;
ha_init(&ha, &g_startup_cfg, 0);
atomic_store(&ha.term, 3ULL);
ha.raft.term = 3;
strncpy(ha.voted_for_name, "ctrl-2", sizeof(ha.voted_for_name) - 1);
ha.voted_for_term = 3;
int leader_known = myapp_leader_already_known(&ha, "ctrl-2",
3 /* term_at_boot */);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, leader_known,
"a self-vote must not be treated as evidence an external leader is "
"known");
ha_fini(&ha);
printf("test_bug1695_self_vote_does_not_skip_election: PASS\n");
}
/* ── Req 2.4: heartbeat timing validation at startup ─────────────────────── */
/*
@ -829,6 +1021,17 @@ void test_ha_startup(void)
RUN_TEST(test_nonblocking_single_node_election_enters_primary_directly);
RUN_TEST(test_nonblocking_election_returns_in_under_10ms);
/* bug-1695 — myapp_run's "leader already known" bootstrap check must
* require a genuine term ADVANCE during the startup nap, not merely a
* nonzero restored term */
RUN_TEST(test_bug1695_stale_restored_term_does_not_skip_election);
RUN_TEST(test_bug1695_term_advanced_during_nap_skips_election);
RUN_TEST(test_bug1695_live_primary_fd_skips_election_without_term_advance);
RUN_TEST(test_bug1695_first_boot_zero_term_no_advance_does_not_skip);
RUN_TEST(test_bug1695_first_boot_term_advances_from_zero_skips);
RUN_TEST(test_bug1695_voted_for_other_in_current_term_skips_unaffected);
RUN_TEST(test_bug1695_self_vote_does_not_skip_election);
/* Task 1.1 (ha-heartbeat-liveness) — ha_init zero-initializes last_activity */
RUN_TEST(test_ha_init_last_activity_is_zero);