Existing worker-rejoin-resync coverage (sequential/chaotic) only proves a
worker-dispatched job survives a real failover -- it never re-checks the
worker's own resource accounting afterward, which is exactly where
bug-1841 lived. This mode over-submits jobs to empirically discover the
worker's real concurrency ceiling (self-verifying via a PEND overflow
probe, not assumed from resval), fails over the primary while jobs are
RUN, waits for genuine rejoin + DONE, then hard-asserts the probe job
(still PEND from before the kill) now reaches RUN -- proving the
scheduler's resInUse actually unfroze.
Live-validated against a real 3-node cluster + xcecd_virtual worker:
settled at 3 RUN / 41 PEND (limits.yaml's limit1, not raw worker
capacity, was the actual ceiling -- exactly the kind of environment
detail this design avoids needing to assume), all 3 recovered to DONE via
genuine rejoin, probe reached RUN 219ms after recovery. Two earlier
design iterations are documented in the round's own comments: sequential
submit-then-wait let early jobs finish before late ones were even
submitted, and assuming resval's raw capacity as the concurrency ceiling
doesn't hold when queue/user limits bind first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rongsong Shen
e592e92178