lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): restored-but-stale raft term permanently blocked re-election after restart (bug-1695)

Browse source 
Found via the mandatory live HA chaotic soak for this session's controller
audit campaign -- unrelated to any of the 9 audit fixes (bug-1684/1687/
1689/1690/1691/1692/1693/1694), all in ha.c/persist.c/job.c/cluster.c/
sched.c/worker.c; this is in myapp.c, untouched by any of them.

myapp_run()'s bootstrap "is a leader already known" check had a fallback
`c.ha.raft.term > 0` meant to detect genuine new raft activity processed
during the startup-stagger nap. But ha_restore_term() loads ha->term from
disk BEFORE this check runs, on every boot -- and Raft terms only ever
increase, never reset. So `raft.term > 0` is trivially true on every
restart for the rest of the cluster's on-disk lifetime, once it has ever
held one election, regardless of whether any leader is currently
reachable. Live-reproduced: every node, on every restart after the
cluster's first election, unconditionally skipped ha_start_election_async
forever -- no node could ever re-assert leadership through a real
election again. bug-1477's existing cluster_init_phase2 retry loop kept
firing correctly (confirmed in the reproduction logs) but could never
succeed, since the deeper cause -- no election ever runs to produce a
live primary to push config from -- was never addressed by that fix.

Fixed by snapshotting `term_at_boot` right after ha_restore_term()/
ha_init_raft() seed the term, strictly before the stagger nap, and
comparing `raft.term > term_at_boot` instead of `raft.term > 0` -- only a
term that has genuinely ADVANCED during this boot cycle (real vote/
heartbeat traffic processed) counts as evidence a leader is known. The
decision was extracted into a new, unit-testable myapp_leader_already_known()
(startup_validation.c, since controller_unit_test deliberately excludes
myapp.c/main.c). voted_for_name/vft==cur and primary_ctrl_fd>=0 checks
are unchanged.

7 new tests in test_ha_startup.c; RED/GREEN independently verified.
Rigorous A/B live-soak proof (identical 6-round --chaotic run, fixed vs
git-stashed-back-to-original): baseline showed term frozen for the whole
run and zero election activity on any non-primary node; fixed run showed
term genuinely progressing and ctrl-2 achieving a real, log-verified
CTRL_PRIMARY transition via a won PreVote round -- proof a node other
than the original primary can now take over, which was structurally
impossible before. Full controller suite: 1191 tests, 2 failures -- both
the pre-existing, unrelated bug-1561 overlayfs sandbox flake.

Residual, NOT independently verified yet (flagged honestly, not fixed
here): the fixed run's chaotic verdict showed committed_job_loss=6
(baseline 0) -- traced via waldump to one jobid missing from every node's
WAL, at the cold-boot window before round 1. Resembles the already-fixed
bug-1487 false-positive class (apply-pump-lag racing verify_classify's
expect-job check) but the reported mechanism doesn't perfectly match
(bug-1487's job was present-in-WAL/not-yet-visible; this one is described
as absent from every WAL entirely) -- needs independent re-verification
before being dismissed, tracked as a follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
This commit is contained in:
Rongsong Shen 2026-08-25 16:11:37 +00:00
parent 23c5c26a23
commit ac46bfa649
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 792 additions and 444 deletions
Download patch file Download diff file Expand all files Collapse all files

54
src/controller/myapp.c
View file

@ -28,6 +28,13 @@ int load_primary_config(cluster_t *c);
/* ha-term-safety task 4.3: hash-verified recovery-activate from the latest
* committed WAL_CONFIG_MARKER (-ENOENT falls back to load_primary_config). */
int config_activate_committed(cluster_t *c);
/* bug-1695: bootstrap "leader already known" decision — implemented in
* startup_validation.c (not myapp.c) so it links into controller_unit_test,
* which does not build myapp.c (myapp_run is an untestable integration entry
* point requiring a live network socket). See that function's doc comment
* in startup_validation.c for the full contract. */
int myapp_leader_already_known(ha_state_t *ha, const char *my_name,
uint64_t term_at_boot);
time_t last_print_time = 0;
unsigned long npkts = 0;
@ -326,6 +333,19 @@ int myapp_run(myconf_t *myconf)
* raft_on_leader_lost() / raft_start_election(). */
ha_init_raft(&c.ha, &c, n_peers);
/* bug-1695: snapshot ha->raft.term immediately after ha_init_raft() seeds
* it from the just-restored (ha_restore_term(), inside
* cluster_init_phase1 above) on-disk term — strictly BEFORE the
* startup-stagger nap below and before any raft callback (vote grant /
* AppendEntries UpdateTerm) can fire from iomgr traffic processed during
* the nap. myapp_leader_already_known() requires raft.term to have
* genuinely advanced past this baseline — not merely be nonzero — to
* treat "a leader is known" as true; see that function's doc comment for
* the full rationale (a merely-restored, never-since-advanced term was
* trivially nonzero on every restart after the cluster's first election,
* permanently skipping every later election attempt). */
uint64_t term_at_boot = c.ha.raft.term;
/* Startup stagger: delay election start by priority * election_priority_step_ms.
* Node 0 starts first; lower-priority nodes wait, see the winner's vote request
* arrive, and grant the vote instead of starting their own election.
@ -357,28 +377,28 @@ int myapp_run(myconf_t *myconf)
* name (myconf->name); the "self" placeholder only appears in test mode,
* so both spellings must be checked. A stale vote (voted_for_term <
* ha->term) is also discarded. All these cases must fall through to
* ha_start_election_async. */
* ha_start_election_async.
*
* bug-1695: the raft-term fallback specifically requires raft.term to
* have ADVANCED past term_at_boot (captured above, right after
* ha_init_raft()) — not merely be nonzero. See
* myapp_leader_already_known()'s doc comment (startup_validation.c) for
* the full rationale: a bare `> 0` was trivially true on every restart
* after the cluster's first election, since ha_restore_term() restores a
* persisted, never-decreasing term. */
{
int leader_known = 0;
pthread_mutex_lock(&c.ha.lock);
uint64_t vft = c.ha.voted_for_term;
uint64_t cur = atomic_load_explicit(&c.ha.term, memory_order_relaxed);
if (c.ha.voted_for_name[0] != '\0' &&
strcmp(c.ha.voted_for_name, "self") != 0 &&
strcmp(c.ha.voted_for_name, myconf->name) != 0 &&
vft == cur)
leader_known = 1;
pthread_mutex_unlock(&c.ha.lock);
if (!leader_known && c.ha.raft.term > 0)
leader_known = 1;
if (!leader_known && c.ha.primary_ctrl_fd >= 0)
leader_known = 1;
int leader_known = myapp_leader_already_known(&c.ha, myconf->name,
term_at_boot);
if (leader_known) {
pthread_mutex_lock(&c.ha.lock);
uint64_t vft_log = c.ha.voted_for_term;
pthread_mutex_unlock(&c.ha.lock);
INFO("myapp_run: leader already known after startup delay "
"(voted_for=%s, voted_term=%" PRIu64 ", raft_term=%" PRIu64
", primary_fd=%d) — skipping own election",
", term_at_boot=%" PRIu64 ", primary_fd=%d) — skipping own "
"election",
c.ha.voted_for_name[0] ? c.ha.voted_for_name : "(none)",
vft, c.ha.raft.term, c.ha.primary_ctrl_fd);
vft_log, c.ha.raft.term, term_at_boot, c.ha.primary_ctrl_fd);
goto skip_election;
}
}