lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(job-array-single-wal-record): extend Live Apply Resource Fence to boot replay (task 4.2's gap)

Browse source 
replay_wal() now halts logical apply and engages persist_mark_apply_resource_fenced()
on a negative apply_committed_entry() return during boot/restart replay, mirroring
persist_apply_pump_run()'s existing live-path halt-and-fence behavior (task 4.1).
Physical scan bookkeeping (current_lsn, CRC chain) stays unconditional per entry,
consistent with persist_recover()'s own established current_lsn/last_applied_lsn
divergence precedent.

Went through one remediation round: round 1 exercised the wrong ha.c gate (periodic
re-alert timer); round 2 corrected the test to drive the real election-time gate
(raft_cb_become_leader_with_cluster_for_test()), with teeth independently confirmed
via a temporary fence-clear RED/GREEN cycle. Round-2 reviewer APPROVED.

Documents a real, NOT-fixed residual gap found during this task: a node's very
first-ever promotion evaluation in a fresh process runs before that node's own
first persist_recover()/replay_wal() call, so this fence is invisible to that one
promotion check (ha.c:1303-1408 gate runs before ha.c:1427-1439's first-boot
recovery). Every promotion evaluation after a process's first one is protected.
Out of this task's persist.c-only boundary; flagged for a dedicated ha.c follow-up.
This commit is contained in:
Rongsong Shen 2026-07-16 16:39:07 +00:00
parent 5a8136263e
commit ea8d9d3c4f
5 changed files with 669 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

229
src/controller/persist.c
View file

@ -3731,6 +3731,21 @@ static int replay_wal(persist_t* p, uint64_t from_lsn,
int replayed = 0;
uint32_t last_crc = WAL_CRC_NONE;
/* job-array-single-wal-record task 7.2 (design.md "Live Apply Resource
* Fence", task 4.2's self-flagged gap): once a negative
* apply_committed_entry() return is hit during this replay pass, every
* SUBSEQUENT entry is skipped for LOGICAL apply purposes -- mirroring
* persist_apply_pump_run()'s halt-and-fence branch, which never
* dispatches past a failed LSN either. This preserves the strict
* sequential-no-skip apply invariant (task 4.1): a later entry may be
* causally dependent on the one that never actually applied (e.g. a
* job-state fact for an array element that was never constructed).
* The PHYSICAL scan (CRC chain, current_lsn, seg_off) below is
* deliberately NOT gated on this flag -- see the apply_rc < 0 branch's
* doc comment for why current_lsn/last_crc must stay accurate for the
* entire on-disk log regardless of the logical apply outcome. */
bool apply_halted = false;
for (int si = 0; si < nseg; si++) {
char seg_path[4096];
snprintf(seg_path, sizeof(seg_path),
@ -3900,66 +3915,176 @@ static int replay_wal(persist_t* p, uint64_t from_lsn,
* connected endpoint that is empty at first-boot replay time.
* MUST be re-verified before tasks 4.1/6.1/6.2/6.3 make this
* path broadly live. */
int apply_rc = apply_committed_entry(p, hdr.type, hdr.lsn, payload,
hdr.payload_len);
int apply_rc = 0;
if (!apply_halted) {
apply_rc = apply_committed_entry(p, hdr.type, hdr.lsn,
payload, hdr.payload_len);
}
free(payload);
/* raft-apply-decoupling task 5.1: track the replay-derived
* LOGICAL apply cursor separately from current_lsn (the physical
* WAL position owned by this loop, bug-778) — last_applied_lsn is
* the "highest entry actually dispatched via
* apply_committed_entry()" position the post-replay verification
* compares against. Segments are scanned in ascending LSN order
* (sorted above) and entries within a segment are sequential, so
* a plain store (no monotonic max needed) is correct here. */
atomic_store_explicit(&p->last_applied_lsn, hdr.lsn,
memory_order_relaxed);
if (apply_halted) {
/* job-array-single-wal-record task 7.2: an earlier entry in
* THIS replay pass already fenced (see the apply_rc < 0
* branch below) -- this entry and every later one is
* skipped for logical apply, exactly like the
* WAL_CHECKPOINT/unknown-type skip branches above. Physical
* bookkeeping (current_lsn/seg_off) still runs
* unconditionally below. */
} else if (apply_rc < 0) {
/* job-array-single-wal-record task 7.2 (design.md "Live
* Apply Resource Fence", task 4.2's self-flagged residual
* gap): mirror persist_apply_pump_run()'s halt-and-fence
* branch exactly. This is never a content-derived failure
* (Req 9.1-9.3's pre-commit validation already made those
* impossible post-commit) -- it is the SAME node-local
* resource-exhaustion class (Req 9.4) the live pump already
* guards against, just hit during boot/restart replay
* instead of live apply. Do NOT advance last_applied_lsn
* past this LSN (skip that store entirely -- last_applied_lsn
* stays parked at this entry's predecessor) and do NOT tally
* this entry's job count; engage the SAME
* persist_mark_apply_resource_fenced() primitive the live
* path uses -- reused verbatim, no parallel mechanism. This
* closes task 4.2's documented residual gap: without this, a
* node that crashes while fenced and re-hits the identical
* resource-exhaustion condition on restart would silently
* discard the failure here (as this branch previously did,
* unconditionally advancing last_applied_lsn regardless of
* apply_rc's sign) and boot up looking fully promotable
* while actually divergent/behind.
*
* Engaging synchronously HERE, inside this boot-time call
* (persist_recover(), called synchronously from
* cluster_init_phase2() before it returns), matters:
* persist_apply_pump_kick()'s post-recovery catch-up kick
* dispatches ASYNCHRONOUSLY onto a pool worker vtask (or
* only runs inline as its own documented fallback) -- relying
* on that kick alone to eventually re-fail and set the fence
* would leave a real window, between persist_recover()
* returning and that async retry actually running, where a
* promotion-gate check could observe "not fenced" and
* wrongly treat this node as eligible.
*
* KNOWN BOUNDARY (task 7.2 adversarial review, NOT fixed by
* this task -- ha.c change required, out of this function's
* scope): this fence, once engaged here, protects every
* SUBSEQUENT promotion evaluation in this process's life --
* a later election's raft_cb_become_leader() gate (ha.c:1337
* reads persist_apply_resource_fenced(), inside the
* eligibility block at ha.c:1303-1408) and the periodic
* re-alert timer (promotion_fence_timeout_fn, ha.c ~876)
* both correctly observe it. It does NOT protect this SAME
* boot's own first-ever promotion evaluation: on a node's
* first-ever election evaluation in its process's life (i.e.
* BEFORE cluster_init_phase2() has ever run), ha.c's
* raft_cb_become_leader() runs its eligibility gate at
* ha.c:1303-1408 strictly BEFORE the is_first_boot block at
* ha.c:1427-1439 that calls cluster_init_phase2() -- which is
* what calls persist_recover()/replay_wal() (cluster.c:1482)
* and is what would engage this fence for the first time.
* There is no re-check of persist_apply_resource_fenced()
* after cluster_init_phase2() returns; the code proceeds
* straight to CTRL_PRIMARY. So a fence engaged by THIS SAME
* first replay is invisible to the one promotion check
* gating that specific boot -- exactly the "all nodes
* restart simultaneously" race this whole task exists to
* guard against. Closing this requires an ha.c change (a
* re-check of persist_apply_resource_fenced() after
* cluster_init_phase2() returns, or reordering recovery
* ahead of the promotion decision), which is out of this
* spec's current scope -- see tasks.md's task 7.2
* Implementation Notes for the full finding. */
ERROR("replay_wal: apply_committed_entry failed for "
"committed entry lsn=%" PRIu64 " wal_type=%" PRIu32
" (rc=%d) during boot/restart replay -- halting this "
"replay's LOGICAL apply at this position (the entry "
"stays committed and will be retried by the live apply "
"pump's post-recovery kick, from the same position); "
"Live Apply Resource Fence ENGAGED "
"(job-array-single-wal-record Req 9.4/9.5)",
hdr.lsn, hdr.type, apply_rc);
persist_mark_apply_resource_fenced(p, hdr.lsn);
apply_halted = true;
} else {
/* raft-apply-decoupling task 5.1: track the replay-derived
* LOGICAL apply cursor separately from current_lsn (the
* physical WAL position owned by this loop, bug-778) --
* last_applied_lsn is the "highest entry actually dispatched
* via apply_committed_entry()" position the post-replay
* verification compares against. Segments are scanned in
* ascending LSN order (sorted above) and entries within a
* segment are sequential, so a plain store (no monotonic max
* needed) is correct here. */
atomic_store_explicit(&p->last_applied_lsn, hdr.lsn,
memory_order_relaxed);
/* Log-derived half of the post-replay apply-correctness
* expectation (Req 8.1): every WAL_JOB_SUBMIT that reaches this
* point is, by construction, above the loaded snapshot's
* checkpoint_lsn (the from_lsn skip above already filtered out
* anything the snapshot covers) — so it names a job that must now
* exist in the live job table that did not exist before this
* replay pass began. A duplicate WAL_JOB_SUBMIT for an
* already-existing jobid (bug-1419: timeout-then-late-commit
* retry) is a legitimate no-op — apply_committed_entry() returns
* 0 for it, not 1 — so it must NOT be tallied here or the
* post-replay comparison against jobmgr's real growth spuriously
* fails (APPLY_DIVERGENT). */
if (hdr.type == WAL_JOB_SUBMIT && new_job_submits_out &&
apply_rc == 1)
(*new_job_submits_out)++;
/* Log-derived half of the post-replay apply-correctness
* expectation (Req 8.1): every WAL_JOB_SUBMIT that reaches
* this point is, by construction, above the loaded
* snapshot's checkpoint_lsn (the from_lsn skip above already
* filtered out anything the snapshot covers) — so it names a
* job that must now exist in the live job table that did not
* exist before this replay pass began. A duplicate
* WAL_JOB_SUBMIT for an already-existing jobid (bug-1419:
* timeout-then-late-commit retry) is a legitimate no-op —
* apply_committed_entry() returns 0 for it, not 1 — so it
* must NOT be tallied here or the post-replay comparison
* against jobmgr's real growth spuriously fails
* (APPLY_DIVERGENT). */
if (hdr.type == WAL_JOB_SUBMIT && new_job_submits_out &&
apply_rc == 1)
(*new_job_submits_out)++;
/* job-array-single-wal-record task 6.0a: the array analog of the
* WAL_JOB_SUBMIT tally immediately above. apply_committed_entry()'s
* WAL_JOB_ARRAY_SUBMIT case (restore_job_array()'s return,
* generalized to N elements) returns apply_rc == the genuine
* count of NEWLY created elements (1..array_size) on a real
* apply, or 0 for an idempotent whole-record re-apply (every
* index already existed -- restore_job_array() returned
* -EEXIST, which this same dispatch maps to 0, mirroring
* WAL_JOB_SUBMIT's own dedup return). Tallying apply_rc (not a
* flat +1) is required here, unlike the single-job case above,
* because one WAL_JOB_ARRAY_SUBMIT entry can materialize many
* jobmgr entries at once; guarding on apply_rc > 0 excludes the
* idempotent 0 case from double-counting a re-apply, exactly as
* apply_rc == 1 excludes WAL_JOB_SUBMIT's duplicate-jobid case
* (bug-1419) above. Without this branch, replay_wal() silently
* undercounts every array record's real N-element jobmgr growth
* to 0, and persist_verify_apply_correctness() then reports a
* false APPLY_DIVERGENT verdict that engages the PERMANENT,
* non-self-healing persist_recover_loud_fence_if_unrecoverable()
* fence (live-reproduced task 6.1 soak, 2026-07-15). */
if (hdr.type == WAL_JOB_ARRAY_SUBMIT && new_job_submits_out &&
apply_rc > 0)
(*new_job_submits_out) += (uint64_t)apply_rc;
/* job-array-single-wal-record task 6.0a: the array analog of
* the WAL_JOB_SUBMIT tally immediately above.
* apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case
* (restore_job_array()'s return, generalized to N elements)
* returns apply_rc == the genuine count of NEWLY created
* elements (1..array_size) on a real apply, or 0 for an
* idempotent whole-record re-apply (every index already
* existed -- restore_job_array() returned -EEXIST, which
* this same dispatch maps to 0, mirroring WAL_JOB_SUBMIT's
* own dedup return). Tallying apply_rc (not a flat +1) is
* required here, unlike the single-job case above, because
* one WAL_JOB_ARRAY_SUBMIT entry can materialize many jobmgr
* entries at once; guarding on apply_rc > 0 excludes the
* idempotent 0 case from double-counting a re-apply, exactly
* as apply_rc == 1 excludes WAL_JOB_SUBMIT's duplicate-jobid
* case (bug-1419) above. Without this branch, replay_wal()
* silently undercounts every array record's real N-element
* jobmgr growth to 0, and persist_verify_apply_correctness()
* then reports a false APPLY_DIVERGENT verdict that engages
* the PERMANENT, non-self-healing
* persist_recover_loud_fence_if_unrecoverable() fence
* (live-reproduced task 6.1 soak, 2026-07-15). */
if (hdr.type == WAL_JOB_ARRAY_SUBMIT && new_job_submits_out &&
apply_rc > 0)
(*new_job_submits_out) += (uint64_t)apply_rc;
/* Advance current_lsn to one past this entry */
replayed++;
}
/* Advance current_lsn to one past this entry -- unconditional,
* regardless of the LOGICAL apply outcome above (mirrors the
* WAL_CHECKPOINT/unknown-type skip branches earlier in this
* loop, which also advance current_lsn without ever dispatching
* through apply_committed_entry()). job-array-single-wal-record
* task 7.2: the physical WAL scan must account for every
* on-disk entry so current_lsn/last_crc stay correct for any
* future append this process makes (as a promoted primary, or
* as a replication standby receiving new entries) -- halting
* this advance too (treating a fenced entry as physically
* absent) would understate the true on-disk tail and risk a
* later append colliding with/overwriting already-durable
* entries beyond the fenced position (persist_recover()'s own
* "Round-2 fix" comment above, for the durable_lsn/apply_cursor
* split, already establishes that current_lsn and
* last_applied_lsn are expected to diverge -- this is the same
* principle, applied to a halted apply instead of a skipped
* checkpoint marker). */
if (hdr.lsn + 1 > p->current_lsn)
p->current_lsn = hdr.lsn + 1;
replayed++;
seg_off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
}