forked from lei.chen/myxce
feat(job-array-single-wal-record): add live, self-healing apply resource fence
Adds a new atomic LSN field on persist_t (distinct from the existing boot-only unrecoverable-fence flag) plus three primitives: persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(), persist_clear_apply_resource_fence(). Wired into persist_apply_pump_run(): engages on task 4.1's halt condition, clears unconditionally on the walk's next successful advance — provably equivalent to "clears only when the same stuck LSN's retry succeeds" since the walk is strictly sequential and never advances past a failed position. Known residual gap, out of this task's boundary (documented in tasks.md Implementation Notes): replay_wal() during boot/restart replay does not yet share this protection, so a node crashing while fenced can silently re-pass the same failure on restart. Flagged for a follow-up task, not fixed here. Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
This commit is contained in:
parent
cc7e99c2f7
commit
b47cc660ba
4 changed files with 296 additions and 2 deletions
|
|
@ -486,6 +486,41 @@ struct persist_s {
|
|||
uint64_t apply_pump_cache_seg_lsn;
|
||||
uint32_t apply_pump_cache_off;
|
||||
uint64_t apply_pump_cache_next_lsn;
|
||||
|
||||
/* job-array-single-wal-record task 4.2 (design.md "Live Apply Resource
|
||||
* Fence"): a LIVE, self-healing fence recording "an LSN is currently
|
||||
* stuck failing to apply" -- deliberately a SEPARATE mechanism from the
|
||||
* sticky, boot-only `recovery_unrecoverable` flag above. The two must
|
||||
* never be conflated: recovery_unrecoverable is set only during boot
|
||||
* replay/recovery and cleared only via the WAL-durability catch-up path
|
||||
* (persist_clear_recovery_unrecoverable), which is wrong for THIS
|
||||
* condition -- a resource-exhaustion apply failure (e.g. OOM mid array
|
||||
* expansion, Req 9.4) can happen at ANY time on a long-lived running
|
||||
* primary or standby, has nothing to do with WAL durability catch-up,
|
||||
* and must clear itself the instant a RETRY of the exact same stuck LSN
|
||||
* succeeds, not on some unrelated periodic check.
|
||||
*
|
||||
* Semantics: 0 = clear (not fenced); non-zero = the LSN currently stuck
|
||||
* failing to apply. Storing the actual LSN (rather than a bare bool)
|
||||
* gives the loud ERROR log and any diagnostic reader (e.g. ctrlstatus)
|
||||
* the exact stuck position for free. LSN 0 is reserved/never a real
|
||||
* committed entry (first real entry is LSN 1), so 0 is a safe "clear"
|
||||
* sentinel needing no separate has-value flag.
|
||||
*
|
||||
* Concurrency: set ONLY from persist_apply_pump_run()'s single-threaded
|
||||
* walk on a negative apply_committed_entry() return (mirrors task 4.1's
|
||||
* halt branch); cleared ONLY from that same walk, the instant a retry of
|
||||
* the SAME LSN applies successfully -- never by an election, a promotion
|
||||
* check, or any other periodic/external path (Req 9.5). Both set and
|
||||
* clear happen under apply_pump_lock (the walk's own serialization), so
|
||||
* the atomic ops here are belt-and-suspenders for readers (ha.c's
|
||||
* promotion-gate call sites, task 4.3) that consult this without holding
|
||||
* that lock. Not persisted -- like last_applied_lsn itself, recomputed
|
||||
* by construction: a crash/restart re-derives "stuck" the next time
|
||||
* replay/live-apply actually fails to apply the same LSN again; it is
|
||||
* not itself a correctness-critical persisted fact. Defaults to 0
|
||||
* (clear) in persist_init -- a fresh store has nothing stuck. */
|
||||
_Atomic uint64_t apply_resource_fenced_lsn;
|
||||
#ifdef XCE_TESTING
|
||||
/* Fault-injection seam (ha-commit-durability task 2.1 error-path test):
|
||||
* when non-zero, every fdatasync site treats the sync as FAILED — it
|
||||
|
|
@ -798,6 +833,42 @@ persist_recover_loud_fence_if_unrecoverable(persist_t* p,
|
|||
bool persist_recovery_unrecoverable(persist_t* p);
|
||||
void persist_clear_recovery_unrecoverable(persist_t* p);
|
||||
|
||||
/* persist_apply_resource_fenced / persist_mark_apply_resource_fenced /
|
||||
* persist_clear_apply_resource_fence (job-array-single-wal-record task 4.2,
|
||||
* design.md "Live Apply Resource Fence", Req 9.4/9.5).
|
||||
*
|
||||
* A LIVE, self-healing fence recording "an LSN is currently stuck failing
|
||||
* to apply" -- deliberately SEPARATE from persist_recovery_unrecoverable
|
||||
* above (see the apply_resource_fenced_lsn field's doc comment on `persist_t`
|
||||
* for the full rationale: that flag is boot-only and clears on WAL-durability
|
||||
* catch-up, which is the wrong trigger for a resource-exhaustion apply
|
||||
* failure that can occur at any time on a long-lived node).
|
||||
*
|
||||
* persist_mark_apply_resource_fenced(p, lsn) is called ONLY from
|
||||
* persist_apply_pump_run()'s own halt branch, the moment a negative
|
||||
* apply_committed_entry() return stops the walk at `lsn` -- mirroring
|
||||
* persist_recover_loud_fence_if_unrecoverable's "log loud, then set the
|
||||
* sticky flag" shape, but scoped to this narrower, live condition and with
|
||||
* no abort()/exit() anywhere (Process Lifecycle invariant,
|
||||
* src/controller/CLAUDE.md): the node stays alive, refuses promotion
|
||||
* (task 4.3 wires the promotion-gate read), and keeps retrying.
|
||||
*
|
||||
* persist_clear_apply_resource_fence(p) is called ONLY from inside that same
|
||||
* apply-pump walk, unconditionally after ANY entry applies successfully --
|
||||
* safe and correct because the walk is strictly sequential and never
|
||||
* advances past a fenced LSN without that exact LSN applying first, so "the
|
||||
* next successful apply" and "a retry of the SAME stuck LSN succeeding" are
|
||||
* the same event by construction. No election, promotion check, or any
|
||||
* other periodic/external path may ever clear this (Req 9.5) -- the node
|
||||
* self-heals purely by the apply pump itself eventually succeeding.
|
||||
*
|
||||
* persist_apply_resource_fenced(p) is the read-side accessor (task 4.3's
|
||||
* promotion-gate call sites, ctrlstatus, tests). All three are safe on a
|
||||
* NULL p (accessor returns false; mutators no-op). */
|
||||
bool persist_apply_resource_fenced(persist_t* p);
|
||||
void persist_mark_apply_resource_fenced(persist_t* p, uint64_t lsn);
|
||||
void persist_clear_apply_resource_fence(persist_t* p);
|
||||
|
||||
/* persist_recovery_complete (bug-1416, raft-apply-decoupling task 7.2 round
|
||||
* 2) -- true unless a persist_recover() call is CURRENTLY in progress on
|
||||
* this persist_t (see the recovery_complete field doc comment). Consulted
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue