lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838)

Browse source 
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:

- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
  Reconciliation" backstop) force-applied any recorded DISPATCH decision
  it found, with zero live capacity check. Added
  sim_reconciled_alloc_fits_capacity(), reusing the same
  sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
  the live picker/allocator path already uses.

- bug-1838: even with that gate, reconciliation still preempted every
  freshly-PEND job before the live picker/allocator got a turn each round
  -- unnecessary now that scheduler-sim-live-allocation gave sim mode the
  same real, capacity-checked picker/allocator threads normal mode uses.
  Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
  2026-09-11 widening whose own justification no longer holds), so a
  freshly-pushed job now always flows through the same live, verified
  decision path as normal mode. Reconciliation now only ever fires for a
  job already RUN from before the replay's own observation window --
  the one case structurally invisible to the live picker.

Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.

Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.

Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-21 09:50:27 +00:00
parent cd9ecb4159
commit 94115b0024
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
44 changed files with 2769 additions and 333 deletions
Download patch file Download diff file Expand all files Collapse all files

143
src/scheduler/testsuite/test_sim_contention_parity.c
View file

@ -452,6 +452,7 @@ int sched_account_jobs(xce_sched_core_t* core)
return rc;
}
/* scheduler-sim-live-allocation spec, task 4.1: sim_loop.c's real
* sched_sim_driver_run() (white-box #include'd into this TU above) now
* spawns sched_job_picker()/sched_allocator() (sched_election.c) directly
@ -1815,6 +1816,31 @@ static int check_host_satisfy_want(xce_sched_sim_driver_t* sim, float want)
return rc;
}
/*
* CORRECTED (2026-09-21, bug-1838, scheduler-driver-equivalence): this
* test's entire fixture (write_multiround_contention_rounds() above) is
* built around jobs A/B/C pushed fresh PEND at round 1 with a matching
* RECORDED DISPATCH decision for A and C -- relying on
* sim_reconcile_unaccounted_jobs() to bridge them, exactly the mechanism
* bug-1838 narrowed back to JOB_IS_RUN-only. Compounding this, THIS file's
* own sched_allocator() override (used whenever sched_sim_driver_run() is
* called, as it is here) is a deliberate PEND-only stub (see that
* function's own header comment) -- "genuine resource-based dispatch... is
* proved separately by test_sim_live_allocation.c". With reconciliation no
* longer touching PEND jobs, and this file's own live path structurally
* unable to dispatch anything, jobs A/B/C now stay PEND for the entire
* fixture, in every round -- resInUse never leaves 0.0, and every
* assertion this test used to make (capacity SATURATED by round 1, stays
* saturated through rounds 2-5) is now backwards. Inverted below to lock
* in the new, correct behavior instead of deleting the coverage outright.
*
* The genuine "capacity is correctly tracked across a real dispatch-then-
* later-confirmation gap" property is now proved via test_sim_live_
* allocation.c's test_sim_verify_no_false_mismatch_after_live_dispatch_
* and_later_confirmation, which links the real scheduler_srcs and can
* genuinely dispatch a job -- something this file's PEND-only stub never
* could, reconciliation or not.
*/
static void
test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
void)
@ -1827,48 +1853,42 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
g_capturing = 0;
g_round_idx = 0;
/* ---- Checkpoint 1 (REVISED 2026-09-11): right after round 1 -- A and
* C are captured AND already genuinely RUN (synchronous capture-time
* transition, mirroring sched_commit_decision()/bug-1624). Capacity
* must already look FULLY SATURATED: a hypothetical job wanting the
* entire WORKER_CAPACITY must be DENIED, since both A and C are
* already counted from this round onward (Requirement 1.2/1.3, at the
* capacity-check OUTCOME level, not just a raw resInUse read). Under
* the PRE-2026-09-11 design this same check was granted (1) -- the
* inversion itself is the proof the revision landed. ---- */
/* ---- Checkpoint 1: right after round 1 -- A and C are pushed PEND
* with a matching recorded DISPATCH, but bug-1838's RUN-only
* reconciliation trigger does not touch them (still PEND), and this
* file's own PEND-only sched_allocator() stub cannot dispatch them
* either. Capacity must therefore be WIDE OPEN: even a job wanting
* the entire WORKER_CAPACITY must be GRANTED, since nothing has
* genuinely been charged. ---- */
run_multiround_contention_fixture(&sim1, 1, "_mr1");
rc = check_host_satisfy_want(&sim1, WORKER_CAPACITY);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 1 (2026-09-11 revision): A and C are captured AND already "
"genuinely RUN, synchronously with capture -- capacity must "
"already look FULLY SATURATED (a job wanting the entire "
"WORKER_CAPACITY must be DENIED), not deferred until a later "
"confirmation round (Requirement 1.2/1.3)");
1, rc,
"bug-1838: A and C stay PEND (neither reconciliation -- now "
"RUN-only -- nor this file's PEND-only live stub can dispatch "
"them), so resInUse stays 0.0 and a job wanting the entire "
"WORKER_CAPACITY must be GRANTED, not denied");
/* ---- Checkpoint 2 (REVISED 2026-09-11): right after round 4 -- job
* A's separately-recorded confirmation event has now ALSO replayed,
* but changes nothing: A and C were both already RUN since round 1.
* Capacity must STILL be fully saturated -- even JOBC_WANT (2.0),
* previously grantable under the deferred design, must now be denied,
* since C itself has been counted since round 1 too. ---- */
/* ---- Checkpoint 2: right after round 4 -- job A's separately-
* recorded confirmation event has replayed, but A was never captured
* in the first place (still PEND), so this is a WARN-and-skip no-op
* (core_events.c logs an unknown-jobid warning for a job it has no
* prior RUN record to reconcile against, per that handler's existing
* contract) -- it does not fabricate a charge. Capacity remains wide
* open. ---- */
run_multiround_contention_fixture(&sim4, 4, "_mr4");
rc = check_host_satisfy_want(&sim4, JOBC_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 4 (2026-09-11 revision): both A and C have been genuinely "
"RUN and fully charged since round 1 -- even JOBC_WANT (2.0) must "
"now be denied, since there is no remaining headroom");
1, rc, "bug-1838: A/C are still PEND at round 4 -- JOBC_WANT (2.0) "
"must be granted against a fully-open worker");
rc = check_host_satisfy_want(&sim4, JOBB_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 4: JOBB_WANT (5.0) must also be denied -- capacity has been "
"fully saturated by A+C since round 1");
1, rc, "bug-1838: JOBB_WANT (5.0) must likewise be granted -- "
"nothing has ever been charged against this worker");
/* ---- Checkpoint 3: the full 5-round replay -- per-round resInUse
* snapshots (Requirement 1.1-1.4 across EVERY round, not just
* endpoints) plus a final capacity check proving FULL saturation once
* both jobs are genuinely RUN. ---- */
* snapshots proving the round-1-through-5 gap stays permanently
* zero-charged, not just at the two checkpoints above. ---- */
g_capturing = 1;
g_round_idx = 0;
run_multiround_contention_fixture(&sim5, 5, "_mr5");
@ -1877,10 +1897,7 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
/* 6, not 5: scheduler-snapshot-job-allocation task 4.6's extra
* snapshot-load-time sched_account_jobs() call (see the sibling test
* above for the full rationale) lands at g_round_resinuse[0], pushing
* every round's own snapshot to index+1 below. This fixture's initial
* snapshot also has "worker-a" registered but no RUN-state jobs yet
* (same fixture_init_seed() as the sibling test), so index 0 reads
* 0.0. */
* every round's own snapshot to index+1 below. */
TEST_ASSERT_EQUAL_INT_MESSAGE(
6, g_round_idx,
"sched_account_jobs() must have been called exactly once per "
@ -1889,48 +1906,36 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[0],
"snapshot load: no RUN-state jobs exist yet at the replay's own "
"starting snapshot -- resInUse must read 0.0 (task 4.6's own "
"load-time call, not a replayed round)");
"snapshot load: no RUN-state jobs exist yet -- resInUse reads 0.0");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[1],
"round 1 (2026-09-11 revision): A(8.0) and C(2.0) are BOTH "
"captured and synchronously transitioned to RUN this round -- "
"resInUse must already reflect the FULL combined 10.0, not "
"deferred to a later confirmation round (Requirement 1.2/1.3)");
0.0f, g_round_resinuse[1],
"bug-1838: round 1 -- A and C are pushed PEND with a matching "
"recorded DISPATCH, but RUN-only reconciliation ignores them and "
"this file's live stub is PEND-only -- resInUse stays 0.0, not "
"the old 10.0 (WORKER_CAPACITY)");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[2],
"round 2: idle filler round -- the round-1 charge must persist "
"unchanged across an EXTRA round, not drift or reset");
0.0f, g_round_resinuse[2],
"round 2: idle filler round -- still 0.0, nothing to charge");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[3],
"round 3: a SECOND idle filler round -- still the full 10.0, "
"widening the gap before A's now-redundant confirmation below to "
"genuinely several rounds");
0.0f, g_round_resinuse[3],
"round 3: a second idle filler round -- still 0.0");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[4],
"round 4: job A's separately-recorded JOB_STATE_CHANGE-to-RUN "
"event replays, 3 rounds after A's own round-1 DISPATCH -- a safe "
"no-op re-affirmation (A has been RUN and charged since round 1); "
"resInUse must remain exactly 10.0, not drop, not double-charge");
0.0f, g_round_resinuse[4],
"bug-1838: round 4 -- job A's separately-recorded JOB_STATE_"
"CHANGE-to-RUN event replays, but A was never captured (still "
"PEND, no job->accounts) -- core_events.c's own unknown-jobid "
"warn-and-skip path applies; no charge is fabricated");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[5],
"round 5: job C's separately-recorded JOB_STATE_CHANGE-to-RUN "
"event replays, 4 rounds after C's own round-1 DISPATCH (a "
"DIFFERENT separation than A's 3 rounds, proving this isn't a "
"single-job coincidence) -- also a safe no-op; resInUse must "
"remain exactly 10.0, not double-charge to 20.0");
0.0f, g_round_resinuse[5],
"bug-1838: round 5 -- job C's separately-recorded confirmation "
"event likewise finds C never captured -- still 0.0, not 10.0");
rc = check_host_satisfy_want(&sim5, JOBD_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 5: capacity is fully saturated by A+C (10.0 of 10.0, since "
"round 1) -- even the smallest positive want (JOBD_WANT) must be "
"denied. Under the PRE-task-1.1 reconciliation gate, neither A nor "
"C would EVER have been captured (their DISPATCH and RUN-"
"confirmation rounds never coincide), so resInUse would read a "
"PERMANENTLY-ZERO 0.0 here and this check would WRONGLY grant "
"admission");
1, rc,
"bug-1838: capacity remains wide open through round 5 -- even "
"JOBD_WANT must be granted, since A/B/C were never genuinely "
"dispatched in this PEND-only-stub environment");
printf("test_contention_capacity_check_correct_across_multi_round_"
"dispatch_confirm_gap: PASS\n");