lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(ha): retry cluster_init_phase2 on failure instead of latching phase2_done unconditionally (bug-1477)

Browse source 
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.

Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).

TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
This commit is contained in:
Rongsong Shen 2026-07-28 13:25:48 +00:00
parent 102e263cce
commit f1e54f3ba0
6 changed files with 384 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

7
src/controller/myapp.c
View file

@ -420,8 +420,11 @@ skip_election:;
int prc = cluster_init_phase2(&c);
if (prc != 0) {
WARN("myapp_run: cluster_init_phase2 on standby rejoin "
"failed (rc=%d) — WAL not recovered; falling back to "
"election-path recovery", prc);
"failed (rc=%d) — WAL not recovered; member stays "
"alive and will retry (bug-1477: this passive "
"skip_election path never sees the election-result "
"callers that would otherwise retry it)", prc);
ha_schedule_phase2_retry(&c);
} else {
c.ha.phase2_done = 1;
}