lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): retry cluster_init_phase2 on failure instead of latching phase2_done unconditionally (bug-1477)

Browse source 
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.

Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).

TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
This commit is contained in:
Rongsong Shen 2026-07-28 13:25:48 +00:00
parent 102e263cce
commit f1e54f3ba0
6 changed files with 384 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

50
src/controller/ha.h
View file

@ -859,6 +859,56 @@ int upgrade_materialization_retry_fn(void *arg);
*/
int schedule_upgrade_materialization_retry(cluster_t *c);
/*
* ha_phase2_retry_fn (bug-1477): re-attempts cluster_init_phase2 for a node
* whose earlier attempt(s) failed and never set ha->phase2_done. Idempotent
* -- a no-op once phase2_done is already set by ANY call site (the shared
* once-per-process contract every cluster_init_phase2 caller relies on).
* Does the actual blocking work (cluster_init_phase2 runs persist_recover,
* wires the commit engine, etc.); production callers invoke it off the
* delaymgr thread via ha_schedule_phase2_retry's trampoline (below) --
* mirrors upgrade_materialization_retry_fn/config_activate_retry_fn exactly.
* Public so tests can drive it directly without a vtask pool. `arg` is a
* `cluster_t *`. Returns 0 on success/already-done, the underlying negative
* errno on a genuine cluster_init_phase2 failure (and self-reschedules
* before returning, so the caller need not).
*
* Root cause this closes: a node that takes myapp_run's skip_election
* passive path (bug-849) -- because a leader was already known after the
* startup stagger -- makes exactly ONE cluster_init_phase2 attempt. If that
* attempt races config availability and fails, NOTHING else ever retries
* it: raft_cb_become_follower / raft_cb_become_leader /
* ha_elect_evaluate_and_signal (the other three call sites) only fire via
* THIS node's own raft-library state transitions, which never happen for a
* node that stays passively CTRL_STANDBY while its primary keeps
* heartbeating normally. The node is then permanently stuck without a
* wired commit engine -- silently unable to apply ANY committed entry for
* the rest of its process lifetime (never self-terminates, per
* controller/CLAUDE.md's Process Lifecycle invariant, but never recovers
* either without this retry).
*/
int ha_phase2_retry_fn(void *arg);
/*
* ha_schedule_phase2_retry (bug-1477): non-blocking, self-rescheduling retry
* for a node whose cluster_init_phase2 has not yet completed -- mirrors
* config_schedule_activate_retry / schedule_upgrade_materialization_retry's
* delaymgr-trampoline-then-vtask shape, except it re-arms ITSELF on failure
* (via ha_phase2_retry_fn) since -- unlike those two, which are re-triggered
* externally by every commit-advance/config-push event -- there is no
* other recurring event that would naturally re-trigger this for a node
* stuck on the passive skip_election path. Retries at ha->config.
* heartbeat_interval_ms cadence (the natural rhythm config/connectivity
* changes arrive on), indefinitely (never gives up -- never-self-terminate
* invariant). No-op when phase2_done is already set (steady state
* schedules nothing -- no busy-spin) or when the delaymgr is not running
* (unit-test fixture path; caller drives the retry directly instead).
* Falls back to running the retry inline (stay alive, never silently drop
* a scheduled retry) if the vtask pool is unavailable. Returns 0 on
* enqueue/no-op, negative on enqueue failure.
*/
int ha_schedule_phase2_retry(cluster_t *c);
/*
* ha_apply_wal_catchup_entry: apply a single WAL catch-up entry to local
* persistent storage and advance ha->applied_lsn on success.