feat(job-array-single-wal-record): add live, self-healing apply resource fence

Adds a new atomic LSN field on persist_t (distinct from the existing
boot-only unrecoverable-fence flag) plus three primitives:
persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(),
persist_clear_apply_resource_fence(). Wired into
persist_apply_pump_run(): engages on task 4.1's halt condition, clears
unconditionally on the walk's next successful advance — provably
equivalent to "clears only when the same stuck LSN's retry succeeds"
since the walk is strictly sequential and never advances past a
failed position.

Known residual gap, out of this task's boundary (documented in
tasks.md Implementation Notes): replay_wal() during boot/restart
replay does not yet share this protection, so a node crashing while
fenced can silently re-pass the same failure on restart. Flagged for
a follow-up task, not fixed here.

Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
This commit is contained in:
Rongsong Shen 2026-07-15 03:54:20 +00:00
parent cc7e99c2f7
commit b47cc660ba
4 changed files with 296 additions and 2 deletions

View file

@ -486,6 +486,41 @@ struct persist_s {
uint64_t apply_pump_cache_seg_lsn;
uint32_t apply_pump_cache_off;
uint64_t apply_pump_cache_next_lsn;
/* job-array-single-wal-record task 4.2 (design.md "Live Apply Resource
* Fence"): a LIVE, self-healing fence recording "an LSN is currently
* stuck failing to apply" -- deliberately a SEPARATE mechanism from the
* sticky, boot-only `recovery_unrecoverable` flag above. The two must
* never be conflated: recovery_unrecoverable is set only during boot
* replay/recovery and cleared only via the WAL-durability catch-up path
* (persist_clear_recovery_unrecoverable), which is wrong for THIS
* condition -- a resource-exhaustion apply failure (e.g. OOM mid array
* expansion, Req 9.4) can happen at ANY time on a long-lived running
* primary or standby, has nothing to do with WAL durability catch-up,
* and must clear itself the instant a RETRY of the exact same stuck LSN
* succeeds, not on some unrelated periodic check.
*
* Semantics: 0 = clear (not fenced); non-zero = the LSN currently stuck
* failing to apply. Storing the actual LSN (rather than a bare bool)
* gives the loud ERROR log and any diagnostic reader (e.g. ctrlstatus)
* the exact stuck position for free. LSN 0 is reserved/never a real
* committed entry (first real entry is LSN 1), so 0 is a safe "clear"
* sentinel needing no separate has-value flag.
*
* Concurrency: set ONLY from persist_apply_pump_run()'s single-threaded
* walk on a negative apply_committed_entry() return (mirrors task 4.1's
* halt branch); cleared ONLY from that same walk, the instant a retry of
* the SAME LSN applies successfully -- never by an election, a promotion
* check, or any other periodic/external path (Req 9.5). Both set and
* clear happen under apply_pump_lock (the walk's own serialization), so
* the atomic ops here are belt-and-suspenders for readers (ha.c's
* promotion-gate call sites, task 4.3) that consult this without holding
* that lock. Not persisted -- like last_applied_lsn itself, recomputed
* by construction: a crash/restart re-derives "stuck" the next time
* replay/live-apply actually fails to apply the same LSN again; it is
* not itself a correctness-critical persisted fact. Defaults to 0
* (clear) in persist_init -- a fresh store has nothing stuck. */
_Atomic uint64_t apply_resource_fenced_lsn;
#ifdef XCE_TESTING
/* Fault-injection seam (ha-commit-durability task 2.1 error-path test):
* when non-zero, every fdatasync site treats the sync as FAILED — it
@ -798,6 +833,42 @@ persist_recover_loud_fence_if_unrecoverable(persist_t* p,
bool persist_recovery_unrecoverable(persist_t* p);
void persist_clear_recovery_unrecoverable(persist_t* p);
/* persist_apply_resource_fenced / persist_mark_apply_resource_fenced /
* persist_clear_apply_resource_fence (job-array-single-wal-record task 4.2,
* design.md "Live Apply Resource Fence", Req 9.4/9.5).
*
* A LIVE, self-healing fence recording "an LSN is currently stuck failing
* to apply" -- deliberately SEPARATE from persist_recovery_unrecoverable
* above (see the apply_resource_fenced_lsn field's doc comment on `persist_t`
* for the full rationale: that flag is boot-only and clears on WAL-durability
* catch-up, which is the wrong trigger for a resource-exhaustion apply
* failure that can occur at any time on a long-lived node).
*
* persist_mark_apply_resource_fenced(p, lsn) is called ONLY from
* persist_apply_pump_run()'s own halt branch, the moment a negative
* apply_committed_entry() return stops the walk at `lsn` -- mirroring
* persist_recover_loud_fence_if_unrecoverable's "log loud, then set the
* sticky flag" shape, but scoped to this narrower, live condition and with
* no abort()/exit() anywhere (Process Lifecycle invariant,
* src/controller/CLAUDE.md): the node stays alive, refuses promotion
* (task 4.3 wires the promotion-gate read), and keeps retrying.
*
* persist_clear_apply_resource_fence(p) is called ONLY from inside that same
* apply-pump walk, unconditionally after ANY entry applies successfully --
* safe and correct because the walk is strictly sequential and never
* advances past a fenced LSN without that exact LSN applying first, so "the
* next successful apply" and "a retry of the SAME stuck LSN succeeding" are
* the same event by construction. No election, promotion check, or any
* other periodic/external path may ever clear this (Req 9.5) -- the node
* self-heals purely by the apply pump itself eventually succeeding.
*
* persist_apply_resource_fenced(p) is the read-side accessor (task 4.3's
* promotion-gate call sites, ctrlstatus, tests). All three are safe on a
* NULL p (accessor returns false; mutators no-op). */
bool persist_apply_resource_fenced(persist_t* p);
void persist_mark_apply_resource_fenced(persist_t* p, uint64_t lsn);
void persist_clear_apply_resource_fence(persist_t* p);
/* persist_recovery_complete (bug-1416, raft-apply-decoupling task 7.2 round
* 2) -- true unless a persist_recover() call is CURRENTLY in progress on
* this persist_t (see the recovery_complete field doc comment). Consulted