fix: bug-1445 -- first-boot promotion check races its own first replay

raft_cb_become_leader()'s eligibility gate read
persist_apply_resource_fenced() before this process's own first
cluster_init_phase2()/persist_recover()/replay_wal() call ever ran --
so a node whose first-ever replay hit a resource-exhaustion apply
failure could still become CTRL_PRIMARY, because the fence that
replay would engage didn't exist yet when the gate checked it. Every
LATER promotion evaluation was already correctly protected; this was
a narrow gap specific to a process's single very-first election.

Fix: re-check persist_apply_resource_fenced() immediately after
cluster_init_phase2() completes on first boot, refusing promotion
(same WARN + ha_arm_promotion_fence_timer() + return pattern as the
existing refusal branch) if that first replay just engaged the fence.

Confirmed ha_elect_evaluate_and_signal (the other election path) is
dead in production for the promotion outcome -- every live call site
of handle_request_vote_reply passes vote_granted=0 since bug-828
redirected real grant recording to the record-only, non-signaling
ha_record_vote_reply -- so raft_cb_become_leader is the sole live
promotion path and this fix's scope is complete.

TDD: new test test_first_boot_promotion_evaluates_own_first_replay_fence
wires ha_phase2_hook to actually call persist_recover(), reproducing
production ordering exactly. RED-confirmed against the unfixed code
at the precise predicted assertion; GREEN-confirmed after the fix.
Full controller_unit_test regression (bundles every HA-adjacent
testsuite/test_*.c) green, 0 new failures.

Originally documented (not fixed) in job-array-single-wal-record
task 7.2's Implementation Notes; logged as bug-1445 and fixed here,
outside that spec's own task list per its persist.c-only boundary.
This commit is contained in:
Rongsong Shen 2026-07-17 10:42:13 +00:00
parent 5ddc20f359
commit ff01db3665
4 changed files with 227 additions and 6 deletions

View file

@ -1436,6 +1436,41 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
cluster_init_phase2(c);
#endif
ha->phase2_done = 1;
/* bug-1445 fix: cluster_init_phase2() above just ran this process's
* very FIRST persist_recover()/replay_wal() -- the ONLY thing that
* can engage the Live Apply Resource Fence (job-array-single-wal-
* record tasks 4.1/4.2, extended to boot replay by task 7.2) for
* THIS node. The eligibility gate above (~line 1337) necessarily
* read persist_apply_resource_fenced() as false: a fresh persist_t
* defaults to unfenced (persist_init) and this is the first replay
* that could ever have set it -- so that gate's ELIGIBLE decision
* was made blind to whatever this first replay just did. Re-check
* here, once, right after the only call that could have changed the
* answer: if this node's own first-ever replay just fenced it,
* refuse promotion now instead of falling through to CTRL_PRIMARY
* below on a decision that is now stale. Every later election in
* this process's lifetime is unaffected -- is_first_boot is false
* by then, so the gate above already covers it correctly (this is
* exactly what the sibling test,
* test_replay_wal_engages_fence_on_resource_exhaustion_failure,
* proves). Mirrors the existing refusal branch above verbatim (same
* WARN shape, same ha_arm_promotion_fence_timer() call, same
* unconditional-refuse treatment as apply_divergent/
* self_restart_pending) rather than inventing a second mechanism. */
if (c != NULL && persist_apply_resource_fenced(&c->persist)) {
WARN("HA: promotion REFUSED — this node's own first-ever boot "
"replay (cluster_init_phase2, just completed) engaged the "
"Live Apply Resource Fence; the earlier eligibility check "
"in this same election evaluation necessarily missed it "
"(bug-1445) — member stays ALIVE, non-promotable, "
"retrying apply (job-array-single-wal-record Req 9.4/9.5)");
ha_arm_promotion_fence_timer(ha, c,
persist_recovery_truncated(&c->persist),
persist_durable_floor(&c->persist),
promotion_safe_lsn);
return;
}
}
/* ha-single-leader-role (Req 1.1): enter CTRL_PRIMARY directly — there is