forked from lei.chen/myxce
Rongsong Shen
16747aa96d
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were dispatched and correctly materialized N real jobmgr entries via apply_committed_entry()'s array case, but never counted -- persist_verify_apply_correctness() then saw real jobmgr growth exceed the undercounted expectation and declared APPLY_DIVERGENT, engaging a PERMANENT, non-self-healing fence (distinct from task 4.2's self-healing Live Apply Resource Fence). Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a single small-array (N=2) sub-round via two different trigger paths. Deterministic, not size-dependent -- the first array submission to a live cluster would fence any node that restarts or hits an election-path recovery fallback before its next checkpoint, shrinking the promotable pool. A pre-existing comment at persist.c ~3863-3874 (predating this spec) had already flagged this exact code region as needing re-verification before tasks 4.1/6.1+ went live; this is that re-verification, and it failed. Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine per-record element count apply_committed_entry()'s array case already returns) when apply_rc > 0, mirroring the existing dedup-safe pattern for WAL_JOB_SUBMIT. Traced the full return-value space of restore_job_array()/apply_committed_entry()'s array case to confirm every tallied value is provably equal to real jobmgr growth -- the fix cannot overcount and so cannot mask a genuine divergence. Verified via a from-scratch live 3-node HA soak re-run (--array-soak), independently by both implementer and reviewer: zero APPLY_DIVERGENT across all three original trigger paths, one-WAL-record/one-commit- barrier confirmed for N=2 and N=10000, per-element post-failover status/cancel/completion confirmed. Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1, 9.4). Inserted mid-run after live soak testing surfaced the bug; user confirmed sequencing (fix now, blocking task 6.1+). |
||
|---|---|---|
| .. | ||
| autoscaler | ||
| controller | ||
| docs | ||
| frontend | ||
| include | ||
| lib | ||
| nixSupport | ||
| scheduler | ||
| schema | ||
| tools | ||
| worker | ||
| flake.lock | ||
| flake.nix | ||
| meson.build | ||
| meson.options | ||