wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3)

Browse source 
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.

A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.

Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.

New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.

Closes out task-group 5 (restart apply-correctness verification).
This commit is contained in:
Rongsong Shen 2026-07-04 13:33:03 +00:00
parent 099ee1a216
commit d3da3ca8bd
5 changed files with 616 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

65
src/controller/ha.c
View file

@ -859,10 +859,21 @@ static int promotion_fence_timeout_fn(void *arg)
? (repl_active_standby_count(ha->repl) > 0)
: false;
/* Caught up: not truncated AND durable position reached the boundary →
* release the fence + clear the sticky flag; the member is eligible on its
* next election (Req 2.6). */
if (!truncated && !below_hwm) {
/* raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4): a fence
* set for an APPLY_DIVERGENT reason must NEVER be live-cleared by this
* purely-structural re-check. apply_verdict is only ever recomputed by
* persist_recover() (a boot-time-once replay, cluster.c) — more WAL
* bytes arriving or a healthy peer appearing cannot prove a local
* apply-dispatch defect is fixed, so this periodic timer must treat an
* apply-divergent fence as untouchable until the next full restart
* replays cleanly (persist_recover_decision fences apply_divergent
* unconditionally for the same reason — see persist.c). */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* Caught up: not truncated AND durable position reached the boundary AND
* the current apply verdict is not DIVERGENT → release the fence + clear
* the sticky flag; the member is eligible on its next election (Req 2.6). */
if (!truncated && !below_hwm && !apply_divergent) {
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
INFO("HA: promotion fence RELEASED after catch-up — durable_floor=%"
@ -1077,6 +1088,16 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
uint64_t durable_floor = persist_durable_floor(&c->persist);
bool unrecoverable = persist_recovery_unrecoverable(&c->persist);
/* raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4):
* same guard as promotion_fence_timeout_fn above — a fence set for
* an APPLY_DIVERGENT reason must never be live-cleared, and this
* gate must never promote while the current apply verdict is
* divergent, REGARDLESS of what `unrecoverable` currently reads
* (defense-in-depth: it must not rely on the other call site alone
* having kept `unrecoverable` accurate). apply_verdict is only ever
* recomputed by a fresh boot-time persist_recover() replay. */
bool apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT;
/* A healthy peer is reachable for catch-up iff some voter granted with a
* live endpoint survives this election (these become the standby
* channels promoted below). */
@ -1106,26 +1127,30 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
truncated, durable_floor, promotion_safe_lsn,
peer_reachable, /*catch_up_succeeded=*/false);
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable) {
/* ELIGIBLE and not sticky-unrecoverable: release the fence (a
* caught-up member regains eligibility, Req 2.6), clear the sticky
* flag if set, and cancel any pending re-alert timer. Proceed to
* authoritative promotion below. */
if (gate == RECOVER_DECISION_ELIGIBLE && !unrecoverable && !apply_divergent) {
/* ELIGIBLE, not sticky-unrecoverable, AND not apply-divergent:
* release the fence (a caught-up member regains eligibility,
* Req 2.6), clear the sticky flag if set, and cancel any pending
* re-alert timer. Proceed to authoritative promotion below. */
atomic_store_explicit(&ha->in_catch_up, 0, memory_order_release);
persist_clear_recovery_unrecoverable(&c->persist);
ha_cancel_promotion_fence_timer(ha);
} else {
/* FENCE, UNRECOVERABLE, or sticky-unrecoverable: refuse promotion —
* do NOT become CTRL_PRIMARY. Fence the member and arm the catch-up
* re-alert timer. The member STAYS ALIVE, non-promotable, retrying
* catch-up (Req 2.2/2.4). NEVER self-exit, never fence silently. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d) — "
"recovery_truncated=%d durable_floor=%" PRIu64
" cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + arming "
"catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4)",
(int)gate, unrecoverable, truncated, durable_floor,
promotion_safe_lsn, peer_reachable);
/* FENCE, UNRECOVERABLE, sticky-unrecoverable, or apply-divergent:
* refuse promotion — do NOT become CTRL_PRIMARY. Fence the member
* and arm the catch-up re-alert timer. The member STAYS ALIVE,
* non-promotable, retrying catch-up (Req 2.2/2.4). NEVER
* self-exit, never fence silently. An apply-divergent member is
* refused unconditionally (Req 8.4) — no peer/HWM condition can
* clear that trigger; only a fresh, clean restart replay can. */
WARN("HA: promotion REFUSED (decision=%d unrecoverable=%d "
"apply_divergent=%d) — recovery_truncated=%d durable_floor=%"
PRIu64 " cluster_hwm=%" PRIu64 " peer_reachable=%d; fencing + "
"arming catch-up re-alert window, staying ALIVE "
"(ha-commit-durability Req 2.2/2.4; raft-apply-decoupling "
"Req 8.4)",
(int)gate, unrecoverable, apply_divergent, truncated,
durable_floor, promotion_safe_lsn, peer_reachable);
ha_arm_promotion_fence_timer(ha, c, truncated, durable_floor,
promotion_safe_lsn);
return;