lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(controller): release the promotion fence once a truncated-boot node has caught up (bug-1919)

Browse source 
The fence re-check required !persist_recovery_truncated(), a boot-time
verdict nothing cleared, so a node that booted RECOVER_TRUNCATED stayed
fenced (fenced: true, non-promotable) until restart however far it
caught up. And the "progress is possible" branch did not re-arm, so the
fence was never re-evaluated.

persist_t gains a runtime recovery_truncation_healed latch, which
persist_recovery_truncated() honours (the raw boot verdict stays
available). The re-check marks the truncation healed once the durable
floor has advanced past the fence-time baseline (entries replicated from
the live primary) and reached the HWM, and the progress branch re-arms
with the same baseline and HWM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-04 09:14:48 +00:00
parent 88a8bf8a47
commit 0ab12074f5
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 197 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

35
src/controller/ha.c
View file

@ -1534,10 +1534,12 @@ static void ha_arm_promotion_fence_timer(ha_state_t *ha, cluster_t *c,
* - caught up (NOT truncated AND durable_floor >= target_hwm) → release the
* in_catch_up fence + clear the sticky unrecoverable flag; eligible again
* (Req 2.6).
* - booted truncated, but catch-up advanced the durable floor past the
* fence-time baseline up to the HWM → mark the truncation healed
* (bug-1919) and fall through to the release check.
* - a healthy peer is reachable now, OR catch-up is making progress → leave
* the member fenced under the existing catch-up machinery (it will release
* via ha_evaluate_promotion_fence → ELIGIBLE when it completes); do not
* re-arm the re-alert while progress is happening.
* the member fenced, no loud alert, and RE-ARM with the same baseline/HWM
* so the next window can release it (bug-1919).
* - STILL unrecoverable (truncated/behind AND no peer AND no progress) →
* RE-EMIT the loud operator alert via persist_recover_loud_fence_if_-
* unrecoverable (keeps the sticky flag + in_catch_up set) and RE-ARM the
@ -1605,6 +1607,23 @@ static int promotion_fence_timeout_fn(void *arg)
* inside persist_apply_pump_run itself). */
bool apply_resource_fenced = persist_apply_resource_fenced(&c->persist);
/* bug-1919: recovery_truncated is the BOOT-time verdict and nothing else
* clears it, so without this a member that booted truncated could never
* pass the release check below however far it caught up. A fenced member
* is not primary, so a durable floor that advanced past the fence-time
* baseline was advanced by contiguous entries replicated from the live
* primary on top of the truncated log; once that reaches the HWM the lost
* tail is refilled and the truncation is healed. No progress means
* nothing proves the log matches the primary's, so the fence stays. */
if (truncated && !below_hwm && made_progress) {
persist_mark_recovery_truncation_healed(&c->persist);
truncated = false;
INFO("HA: recovery-truncated log healed by catch-up — durable_floor=%"
PRIu64 " (baseline=%" PRIu64 ") >= cluster HWM=%" PRIu64
" (bug-1919)",
durable_floor, t->baseline_durable_floor, t->target_hwm);
}
/* Caught up: not truncated AND durable position reached the boundary AND
* the current apply verdict is not DIVERGENT AND no self-restart is
* pending AND no live apply resource fence is engaged → release the
@ -1623,8 +1642,10 @@ static int promotion_fence_timeout_fn(void *arg)
/* Still hollow, but a peer is reachable or catch-up advanced the durable
* floor: progress is possible. Leave the member fenced (in_catch_up stays
* set) under the existing catch-up machinery; do not re-arm the re-alert
* while progress is happening. */
* set) without the loud alert, and re-arm with the SAME baseline and HWM so
* the next window re-evaluates and releases the fence once catch-up
* completes. Nothing else releases this fence: before bug-1919 this branch
* did not re-arm and the member stayed fenced until restart. */
if (peer_reachable || made_progress) {
INFO("HA: promotion fence catch-up window elapsed but progress is "
"possible (peer_reachable=%d durable_floor=%" PRIu64 " baseline=%"
@ -1632,7 +1653,11 @@ static int promotion_fence_timeout_fn(void *arg)
"(ha-commit-durability Req 2.2)",
peer_reachable, durable_floor, t->baseline_durable_floor,
t->target_hwm);
bool was_truncated = t->was_truncated;
uint64_t baseline = t->baseline_durable_floor;
uint64_t hwm = t->target_hwm;
free(t);
ha_arm_promotion_fence_timer(ha, c, was_truncated, baseline, hwm);
return 0;
}