lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): sim resource over-commit, byte-unit display, sync job-state-at-capture

Browse source 
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
  per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
  core_reset_workers already does (bug-1751), closing the gap that let a
  worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
  a captured job's job->state is set to RUN synchronously at dispatch capture,
  mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
  of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
  same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
  live feedback that it over-fires inside busy sessions; replace it with a
  prescan-based "next non-dummy session" hint in info summary instead

Full scheduler suite: 77/77 tests pass (asan profile).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-12 14:06:41 +00:00
parent ffca959ed9
commit 25fb5c115a
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
22 changed files with 1827 additions and 303 deletions
Download patch file Download diff file Expand all files Collapse all files

185
src/scheduler/testsuite/test_sim_contention_parity.c
View file

@ -1380,49 +1380,70 @@ test_contention_parity_matches_real_normal_mode_recording_round_by_round(
}
/* ======================================================================
* scheduler-sim-eager-reconciliation task 3.1 (Req 1.1-1.4): a resource-
* contention fixture where a job's recorded DISPATCH decision and its
* JOB_STATE_CHANGE-to-RUN confirmation are separated by SEVERAL rounds --
* not the same-round shape the test above already covers (there,
* scheduler-sim-eager-reconciliation task 3.1 (Req 1.1-1.4), REVISED
* 2026-09-11 (user-directed, requirements.md Requirement 1's revision):
* a resource-contention fixture where a job's recorded DISPATCH decision
* and its JOB_STATE_CHANGE-to-RUN confirmation are separated by SEVERAL
* rounds -- not the same-round shape the test above already covers (there,
* JOB_STATE_CHANGE lands in the SAME round window as the matching DISPATCH
* decision, which is exactly the shape the OLD, pre-task-1.1 trigger gate
* (`!JOB_IS_RUN(job->state)` in sim_reconcile_unaccounted_jobs(), removed
* by task 1.1) already handled correctly). This is the genuinely NEW
* scenario the OLD gate could never reconcile: jobs A and C are PEND at
* the exact round their own DISPATCH decision is recorded (round 1), and
* only reach RUN several rounds later (rounds 4 and 5 respectively) via a
* bare JOB_STATE_CHANGE event with no DISPATCH decision of its own -- by
* which point round 1's own decoded_session (the only place their
* DISPATCH ever existed) is long freed. Under the OLD gate, job->accounts
* would stay NULL forever for both jobs, and sched_account_jobs()'s own
* JOB_IS_RUN + accounts!=NULL charging gate would therefore never charge
* them -- resInUse would read a PERMANENTLY-ZERO 0.0 for the rest of the
* replay, wrongly making the worker's full WORKER_CAPACITY look available
* even once both jobs are genuinely running. See this task's own Status
* Report for the live revert-rebuild confirmation that this exact fixture
* reproduces that failure against a reverted tree.
* by task 1.1) already handled correctly). Jobs A and C are PEND at the
* exact round their own DISPATCH decision is recorded (round 1), and only
* reach a SEPARATELY-RECORDED RUN confirmation several rounds later
* (rounds 4 and 5 respectively) via a bare JOB_STATE_CHANGE event with no
* DISPATCH decision of its own -- by which point round 1's own
* decoded_session (the only place their DISPATCH ever existed) is long
* freed.
*
* Pre-task-1.1: job->accounts stayed NULL forever for both jobs (the OLD
* `!JOB_IS_RUN` gate never re-searched a PEND job past its own round), so
* resInUse read a PERMANENTLY-ZERO 0.0 for the rest of the replay.
*
* Task 1.1 through 2026-09-07: `sim_reconcile_unaccounted_jobs()` captured
* job->allocs/job->accounts at round 1 regardless of state, but LEFT
* job->state untouched -- charging stayed deferred to sched_account_jobs()'s
* unchanged JOB_IS_RUN gate until the SEPARATELY-recorded JOB_STATE_CHANGE
* event replayed at round 4/5, several rounds after capture.
*
* 2026-09-11 revision (current, this fixture): capture now ALSO sets
* job->state = JOB_STAT_RUN synchronously (mirroring scheduler.c's
* sched_commit_decision(), bug-1624) -- so BOTH jobs A and C are
* genuinely RUN and charged starting round 1 itself, the SAME round their
* DISPATCH decision is captured, not deferred to rounds 4/5. The later,
* separately-recorded JOB_STATE_CHANGE-to-RUN events at rounds 4/5 are
* still replayed (this fixture keeps them, unchanged, to prove they
* remain safe/idempotent) but no longer change resInUse at all -- they
* re-affirm an already-RUN job. This test's core value survives
* unchanged: prove capacity accounting is CORRECT across a realistic
* multi-round dispatch-to-confirmation gap, not permanently wrong for the
* rest of the replay (the pre-task-1.1 bug) -- but "correct" now means
* "already fully accounted from round 1", not "deferred until
* confirmation."
*
* Fixture shape (reuses this file's own established constants verbatim --
* WORKER_CAPACITY=10.0, JOBA_WANT=8.0, JOBB_WANT=5.0, JOBC_WANT=2.0,
* JOBD_WANT=1.0 -- never re-typed):
* Round 1 (begin_round=2000): A/B/C pushed PEND; A(8.0) and C(2.0)
* dispatched (A+C=10.0, exactly saturating capacity), B genuinely
* denied. NEITHER A NOR C's JOB_STATE_CHANGE appears anywhere near
* this round.
* dispatched (A+C=10.0, exactly saturating capacity) -- BOTH
* genuinely RUN and charged from this round onward (2026-09-11
* revision), B genuinely denied. NEITHER A NOR C's JOB_STATE_CHANGE
* appears anywhere near this round.
* Round 2, 3 (begin_round=3000/4000): pure filler -- B is genuinely
* re-offered and re-denied each round (builtin_fcfs's own documented
* "no-op, retried next round" contract), widening the gap before A/C's
* eventual confirmations below to genuinely "several rounds", and
* proving the deferred-zero state from round 1 persists correctly
* across MULTIPLE idle rounds, not just one.
* Round 4 (begin_round=5000): job A's JOB_STATE_CHANGE to RUN -- 3
* rounds after its own round-1 DISPATCH. Job C is still PEND -- only
* PARTIAL usage (8.0 of 10.0) must be reflected from this round on.
* Round 5 (begin_round=6000): job C's JOB_STATE_CHANGE to RUN -- 4
* rounds after ITS OWN round-1 DISPATCH (a deliberately DIFFERENT
* separation than job A's 3 rounds, proving this isn't a single-job
* coincidence). Both jobs now genuinely RUN -- FULL usage (10.0 of
* 10.0) must be reflected from this round on.
* eventual (now redundant) confirmations below to genuinely "several
* rounds", and proving the round-1 charge persists correctly across
* MULTIPLE idle rounds, not just one.
* Round 4 (begin_round=5000): job A's SEPARATELY-recorded JOB_STATE_
* CHANGE to RUN replays -- 3 rounds after its own round-1 DISPATCH.
* A safe no-op re-affirmation (already RUN since round 1); resInUse
* stays at the SAME full 10.0 it already reached at round 1.
* Round 5 (begin_round=6000): job C's SEPARATELY-recorded JOB_STATE_
* CHANGE to RUN replays -- 4 rounds after ITS OWN round-1 DISPATCH (a
* deliberately DIFFERENT separation than job A's 3 rounds, proving
* this isn't a single-job coincidence). Also a safe no-op; resInUse
* is unchanged, still 10.0.
*
* The single test function below drives this SAME fixture shape three
* times, truncated after round 1, round 4, and round 5 respectively (via
@ -1597,41 +1618,43 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
g_capturing = 0;
g_round_idx = 0;
/* ---- Checkpoint 1: right after round 1 -- A and C are dispatched but
* still PEND (neither has reached RUN yet). Capacity must look FULLY
* free: a hypothetical job wanting the entire WORKER_CAPACITY must be
* granted, since nothing is genuinely running yet (Requirement 1.2, at
* the capacity-check OUTCOME level, not just a raw resInUse read). ---- */
/* ---- Checkpoint 1 (REVISED 2026-09-11): right after round 1 -- A and
* C are captured AND already genuinely RUN (synchronous capture-time
* transition, mirroring sched_commit_decision()/bug-1624). Capacity
* must already look FULLY SATURATED: a hypothetical job wanting the
* entire WORKER_CAPACITY must be DENIED, since both A and C are
* already counted from this round onward (Requirement 1.2/1.3, at the
* capacity-check OUTCOME level, not just a raw resInUse read). Under
* the PRE-2026-09-11 design this same check was granted (1) -- the
* inversion itself is the proof the revision landed. ---- */
run_multiround_contention_fixture(&sim1, 1, "_mr1");
rc = check_host_satisfy_want(&sim1, WORKER_CAPACITY);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, rc,
"round 1: A and C are dispatched-but-still-PEND -- capacity must "
"look FULLY free (a job wanting the entire WORKER_CAPACITY must be "
"granted), since neither job has genuinely reached RUN yet "
"(Requirement 1.2)");
0, rc,
"round 1 (2026-09-11 revision): A and C are captured AND already "
"genuinely RUN, synchronously with capture -- capacity must "
"already look FULLY SATURATED (a job wanting the entire "
"WORKER_CAPACITY must be DENIED), not deferred until a later "
"confirmation round (Requirement 1.2/1.3)");
/* ---- Checkpoint 2: right after round 4 -- job A confirmed RUN, 3
* rounds after its own round-1 DISPATCH; job C still PEND. Capacity
* must reflect PARTIAL usage: exactly JOBC_WANT (2.0) of headroom
* remains -- a job wanting exactly that must be granted, one wanting
* more (JOBB_WANT, 5.0) must be denied. This is the strongest single
* proof that combined accounting is correct, not just "some job got
* counted": an implementation that dropped job A's contribution (the
* OLD same-round-only bug) would wrongly grant JOBB_WANT here too. ---- */
/* ---- Checkpoint 2 (REVISED 2026-09-11): right after round 4 -- job
* A's separately-recorded confirmation event has now ALSO replayed,
* but changes nothing: A and C were both already RUN since round 1.
* Capacity must STILL be fully saturated -- even JOBC_WANT (2.0),
* previously grantable under the deferred design, must now be denied,
* since C itself has been counted since round 1 too. ---- */
run_multiround_contention_fixture(&sim4, 4, "_mr4");
rc = check_host_satisfy_want(&sim4, JOBC_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, rc,
"round 4: only job A (8.0) is genuinely RUN -- exactly JOBC_WANT "
"(2.0) of headroom must remain and must be granted");
0, rc,
"round 4 (2026-09-11 revision): both A and C have been genuinely "
"RUN and fully charged since round 1 -- even JOBC_WANT (2.0) must "
"now be denied, since there is no remaining headroom");
rc = check_host_satisfy_want(&sim4, JOBB_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 4: JOBB_WANT (5.0) exceeds the 2.0 remaining headroom -- "
"must be denied. A capacity check that wrongly dropped job A's "
"already-RUN 8.0 usage (the OLD same-round-only bug) would wrongly "
"grant this");
"round 4: JOBB_WANT (5.0) must also be denied -- capacity has been "
"fully saturated by A+C since round 1");
/* ---- Checkpoint 3: the full 5-round replay -- per-round resInUse
* snapshots (Requirement 1.1-1.4 across EVERY round, not just
@ -1661,37 +1684,41 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
"starting snapshot -- resInUse must read 0.0 (task 4.6's own "
"load-time call, not a replayed round)");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[1],
"round 1: A/C dispatched but still PEND -- resInUse must read 0.0, "
"not fabricated or prematurely charged (Requirement 1.2)");
WORKER_CAPACITY, g_round_resinuse[1],
"round 1 (2026-09-11 revision): A(8.0) and C(2.0) are BOTH "
"captured and synchronously transitioned to RUN this round -- "
"resInUse must already reflect the FULL combined 10.0, not "
"deferred to a later confirmation round (Requirement 1.2/1.3)");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[2],
"round 2: idle filler round -- the deferred-zero state from round "
"1 must persist, not drift, across an EXTRA round");
WORKER_CAPACITY, g_round_resinuse[2],
"round 2: idle filler round -- the round-1 charge must persist "
"unchanged across an EXTRA round, not drift or reset");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[3],
"round 3: a SECOND idle filler round -- still 0.0, widening the "
"gap before A's confirmation below to genuinely several rounds");
WORKER_CAPACITY, g_round_resinuse[3],
"round 3: a SECOND idle filler round -- still the full 10.0, "
"widening the gap before A's now-redundant confirmation below to "
"genuinely several rounds");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
JOBA_WANT, g_round_resinuse[4],
"round 4: job A's JOB_STATE_CHANGE to RUN just replayed, 3 rounds "
"after A's own round-1 DISPATCH -- resInUse must now reflect A's "
"8.0 usage (Requirement 1.3), and ONLY A's -- job C is still PEND");
WORKER_CAPACITY, g_round_resinuse[4],
"round 4: job A's separately-recorded JOB_STATE_CHANGE-to-RUN "
"event replays, 3 rounds after A's own round-1 DISPATCH -- a safe "
"no-op re-affirmation (A has been RUN and charged since round 1); "
"resInUse must remain exactly 10.0, not drop, not double-charge");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[5],
"round 5: job C's JOB_STATE_CHANGE to RUN just replayed, 4 rounds "
"after C's own round-1 DISPATCH (a DIFFERENT separation than A's 3 "
"rounds, proving this isn't a single-job coincidence) -- resInUse "
"must now reflect the FULL combined 10.0 usage of both A and C "
"together");
"round 5: job C's separately-recorded JOB_STATE_CHANGE-to-RUN "
"event replays, 4 rounds after C's own round-1 DISPATCH (a "
"DIFFERENT separation than A's 3 rounds, proving this isn't a "
"single-job coincidence) -- also a safe no-op; resInUse must "
"remain exactly 10.0, not double-charge to 20.0");
rc = check_host_satisfy_want(&sim5, JOBD_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 5: capacity is fully saturated by A+C (10.0 of 10.0) -- "
"even the smallest positive want (JOBD_WANT) must be denied. Under "
"the OLD same-round-only reconciliation gate, neither A nor C "
"would EVER have been captured (their DISPATCH and RUN-"
"round 5: capacity is fully saturated by A+C (10.0 of 10.0, since "
"round 1) -- even the smallest positive want (JOBD_WANT) must be "
"denied. Under the PRE-task-1.1 reconciliation gate, neither A nor "
"C would EVER have been captured (their DISPATCH and RUN-"
"confirmation rounds never coincide), so resInUse would read a "
"PERMANENTLY-ZERO 0.0 here and this check would WRONGLY grant "
"admission");