lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(job-array-single-wal-record): replay_wal's job-count tally was blind to array submissions

Browse source 
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).

Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.

Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.

Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.

Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
This commit is contained in:
Rongsong Shen 2026-07-16 04:09:43 +00:00
parent b7b90e9d21
commit 16747aa96d
3 changed files with 292 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

24
src/controller/persist.c
View file

@ -3903,6 +3903,30 @@ static int replay_wal(persist_t* p, uint64_t from_lsn,
apply_rc == 1)
(*new_job_submits_out)++;
/* job-array-single-wal-record task 6.0a: the array analog of the
* WAL_JOB_SUBMIT tally immediately above. apply_committed_entry()'s
* WAL_JOB_ARRAY_SUBMIT case (restore_job_array()'s return,
* generalized to N elements) returns apply_rc == the genuine
* count of NEWLY created elements (1..array_size) on a real
* apply, or 0 for an idempotent whole-record re-apply (every
* index already existed -- restore_job_array() returned
* -EEXIST, which this same dispatch maps to 0, mirroring
* WAL_JOB_SUBMIT's own dedup return). Tallying apply_rc (not a
* flat +1) is required here, unlike the single-job case above,
* because one WAL_JOB_ARRAY_SUBMIT entry can materialize many
* jobmgr entries at once; guarding on apply_rc > 0 excludes the
* idempotent 0 case from double-counting a re-apply, exactly as
* apply_rc == 1 excludes WAL_JOB_SUBMIT's duplicate-jobid case
* (bug-1419) above. Without this branch, replay_wal() silently
* undercounts every array record's real N-element jobmgr growth
* to 0, and persist_verify_apply_correctness() then reports a
* false APPLY_DIVERGENT verdict that engages the PERMANENT,
* non-self-healing persist_recover_loud_fence_if_unrecoverable()
* fence (live-reproduced task 6.1 soak, 2026-07-15). */
if (hdr.type == WAL_JOB_ARRAY_SUBMIT && new_job_submits_out &&
apply_rc > 0)
(*new_job_submits_out) += (uint64_t)apply_rc;
/* Advance current_lsn to one past this entry */
if (hdr.lsn + 1 > p->current_lsn)
p->current_lsn = hdr.lsn + 1;