lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(job-array-single-wal-record): add live, self-healing apply resource fence

Browse source 
Adds a new atomic LSN field on persist_t (distinct from the existing
boot-only unrecoverable-fence flag) plus three primitives:
persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(),
persist_clear_apply_resource_fence(). Wired into
persist_apply_pump_run(): engages on task 4.1's halt condition, clears
unconditionally on the walk's next successful advance — provably
equivalent to "clears only when the same stuck LSN's retry succeeds"
since the walk is strictly sequential and never advances past a
failed position.

Known residual gap, out of this task's boundary (documented in
tasks.md Implementation Notes): replay_wal() during boot/restart
replay does not yet share this protection, so a node crashing while
fenced can silently re-pass the same failure on restart. Flagged for
a follow-up task, not fixed here.

Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
This commit is contained in:
Rongsong Shen 2026-07-15 03:54:20 +00:00
parent cc7e99c2f7
commit b47cc660ba
4 changed files with 296 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

71
src/controller/persist.c
View file

@ -623,6 +623,15 @@ int persist_init(persist_t* p, persist_config_t* cfg, cluster_t* c)
p->apply_verdict = APPLY_CLEAN;
atomic_store(&p->last_applied_lsn, 0);
/* Live Apply Resource Fence starts clear (job-array-single-wal-record
* task 4.2, Req 9.4): a fresh store has nothing stuck failing to apply.
* Set by persist_mark_apply_resource_fenced on a resource-exhaustion
* apply failure; cleared by persist_clear_apply_resource_fence the
* instant a retry of the same LSN succeeds -- see persist_t's field doc
* comment for why this is deliberately separate from
* recovery_unrecoverable just above. */
atomic_store(&p->apply_resource_fenced_lsn, 0);
/* bug-1416: defaults to true (nothing to gate) so every fixture that
* calls persist_init() but never persist_recover() -- most of this
* spec's own test suite -- sees identical, ungated apply-pump/live-
@ -3374,8 +3383,16 @@ int persist_apply_pump_run(persist_t* p)
"committed entry lsn=%" PRIu64 " wal_type=%" PRIu32
" (rc=%d) -- stopping this pass at this position; the "
"entry stays committed and will be retried, from the same "
"position, on the next kick",
"position, on the next kick; Live Apply Resource Fence "
"ENGAGED (job-array-single-wal-record Req 9.4)",
next, wal_type, apply_rc);
/* job-array-single-wal-record task 4.2 (design.md "Live Apply
* Resource Fence"): engage the NEW, live, self-healing fence --
* distinct from recovery_unrecoverable -- so the promotion gate
* (task 4.3) can refuse this node while it stays alive and
* keeps retrying. Cleared below, from inside this same walk,
* the instant a retry of this exact LSN succeeds (Req 9.5). */
persist_mark_apply_resource_fenced(p, next);
free(payload);
pthread_mutex_unlock(&p->apply_pump_lock);
return apply_rc;
@ -3385,6 +3402,21 @@ int persist_apply_pump_run(persist_t* p)
atomic_store_explicit(&p->last_applied_lsn, next, memory_order_relaxed);
applied++;
/* job-array-single-wal-record task 4.2 (design.md "Live Apply
* Resource Fence", Req 9.5): self-heal. Called unconditionally on
* EVERY successful apply, not just after a known-fenced LSN --
* persist_clear_apply_resource_fence() is a cheap no-op when nothing
* is fenced (the overwhelming common case). This is correct and
* sufficient because the walk above is strictly sequential and
* returns immediately on any failure (see the halt branch just
* above): it can never advance PAST a fenced LSN without that exact
* LSN applying successfully first, so "the next entry to apply
* successfully after a fence engages" and "a retry of the SAME
* stuck LSN succeeding" are the same event by construction. No
* election, promotion check, or external/admin action is involved --
* the fence clears purely because this walk itself made progress. */
persist_clear_apply_resource_fence(p);
/* raft-apply-decoupling task 6.2 (Req 1.1, 2.1-2.3): consume this
* LSN's command hand-off, if any (task 3.2's
* persist_register_pending_apply(), e.g. handle_jobsubmit's
@ -4357,6 +4389,43 @@ void persist_clear_recovery_unrecoverable(persist_t* p)
}
}
/* job-array-single-wal-record task 4.2 (design.md "Live Apply Resource
* Fence", Req 9.4/9.5). See the apply_resource_fenced_lsn field's doc
* comment (persist.h) and this trio's own doc comment (persist.h) for the
* full rationale and the deliberate distinction from
* persist_recovery_unrecoverable/persist_clear_recovery_unrecoverable just
* above -- that boot-only mechanism is NOT reused here on purpose. */
bool persist_apply_resource_fenced(persist_t* p)
{
return p && atomic_load_explicit(&p->apply_resource_fenced_lsn,
memory_order_acquire) != 0;
}
void persist_mark_apply_resource_fenced(persist_t* p, uint64_t lsn)
{
if (!p)
return;
atomic_store_explicit(&p->apply_resource_fenced_lsn, lsn,
memory_order_release);
}
void persist_clear_apply_resource_fence(persist_t* p)
{
if (!p)
return;
/* Clear only on a genuine transition so the self-heal log fires once per
* episode, not on every ordinary successful apply (the overwhelming
* common case: nothing was fenced, so this is a cheap no-op). */
uint64_t prev = atomic_exchange_explicit(&p->apply_resource_fenced_lsn, 0,
memory_order_acq_rel);
if (prev != 0) {
INFO("persist_apply_pump_run: Live Apply Resource Fence RELEASED — "
"lsn=%" PRIu64 " applied successfully on retry; promotable "
"again (job-array-single-wal-record Req 9.5)",
prev);
}
}
recover_decision_t
persist_recover_loud_fence_if_unrecoverable(persist_t* p,
const recover_policy_inputs_t* in)