lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838)

Browse source 
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:

- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
  Reconciliation" backstop) force-applied any recorded DISPATCH decision
  it found, with zero live capacity check. Added
  sim_reconciled_alloc_fits_capacity(), reusing the same
  sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
  the live picker/allocator path already uses.

- bug-1838: even with that gate, reconciliation still preempted every
  freshly-PEND job before the live picker/allocator got a turn each round
  -- unnecessary now that scheduler-sim-live-allocation gave sim mode the
  same real, capacity-checked picker/allocator threads normal mode uses.
  Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
  2026-09-11 widening whose own justification no longer holds), so a
  freshly-pushed job now always flows through the same live, verified
  decision path as normal mode. Reconciliation now only ever fires for a
  job already RUN from before the replay's own observation window --
  the one case structurally invisible to the live picker.

Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.

Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.

Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-21 09:50:27 +00:00
parent cd9ecb4159
commit 94115b0024
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
44 changed files with 2769 additions and 333 deletions
Download patch file Download diff file Expand all files Collapse all files

229
src/scheduler/testsuite/test_sim_verify.c
View file

@ -222,6 +222,55 @@ int sched_account_jobs(xce_sched_core_t* core)
return 0;
}
/* bug-1836 (scheduler-driver-equivalence fix, Recorded-Decision
* Reconciliation capacity gate): link-satisfying stubs, mirroring this
* file's existing sched_account_jobs() no-op precedent immediately above --
* sim_loop.c's sim_reconcile_unaccounted_jobs() now also calls
* sched_check_host_satisfy()/sched_check_shared_satisfy() (resource.c) to
* re-validate a recorded decision against live capacity, and
* sched_charge_account() (resource.c) to keep that check's own view of
* capacity up to date across multiple candidates in the same round. This
* file never exercises real resource contention (same rationale as the
* existing sched_account_jobs()/sched_ctx_get_uncomit_host() no-ops), so
* the satisfy checks trivially always succeed and the charge call is a
* no-op. */
int sched_check_host_satisfy(xce_sched_core_t* core, xce_sched_worker_t* w,
xvec_t* uncomitted, xvec_idval_t* wanted,
xvec_idval_t* unmeet)
{
(void)core;
(void)w;
(void)uncomitted;
(void)wanted;
(void)unmeet;
return 1;
}
int sched_check_shared_satisfy(xce_sched_core_t* core, xvec_t* uncomitted,
xvec_idval_t* wanted, xvec_idval_t* unmeet)
{
(void)core;
(void)uncomitted;
(void)wanted;
(void)unmeet;
return 1;
}
int sched_charge_account(xce_sched_core_t* core, xce_job_account_t* acct)
{
(void)core;
(void)acct;
return 0;
}
int sched_uncharge_account(xce_sched_core_t* core, xce_job_account_t* acct)
{
(void)core;
(void)acct;
return 0;
}
/* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same
* copy test_sim_event_loop.c/test_sim_startup.c/test_core_full_cycle.c each
* carry (schedobj.c is not linked here: it would need plugin_register(),
@ -750,6 +799,29 @@ static void make_tmp_dir(char* out, size_t out_len, const char* suffix)
* require a resolvable picker (Req 5 AC 21: "Picker state is NOT stored in
* snapshot").
*/
/* bug-1836 fix: sim_reconcile_unaccounted_jobs()'s new capacity gate calls
* sched_get_worker() for every host named in a reconciled job's recorded
* allocation, and treats "host not found" as a capacity miss -- this file's
* own sched_check_host_satisfy() stub (link-satisfying only, resource.c is
* not white-box #included here) always reports "satisfied" once reached,
* but a bare, unregistered host name is now rejected BEFORE that stub is
* even called. This helper registers a bare worker (no real resource
* values needed -- the stub never reads them) so tests whose fixtures name
* "hostA"/"hostC" in a recorded DISPATCH decision resolve correctly. */
static void add_bare_worker(xce_sched_core_t* core, const char* name)
{
xce_sched_worker_t* w = (xce_sched_worker_t*)calloc(1, sizeof(*w));
TEST_ASSERT_NOT_NULL(w);
w->name = strdup(name);
TEST_ASSERT_NOT_NULL(w->name);
TEST_ASSERT_EQUAL_INT(0, zarray_init(&(w->resvals), 8, NULL));
TEST_ASSERT_EQUAL_INT(0, zarray_init(&(w->resInUse), 8, NULL));
TEST_ASSERT_TRUE(sched_add_worker(core, w) >= 0);
}
static void build_fixture_scheduler(xce_scheduler_t* sched, char* record_dir)
{
int rc;
@ -762,6 +834,13 @@ static void build_fixture_scheduler(xce_scheduler_t* sched, char* record_dir)
sched->core.job_pick_plugin = (char*)"builtin_fcfs";
sched->record_dir = record_dir;
/* bug-1836 fix: registered once here (captured into every fixture's
* initial snapshot) rather than per-test, since several tests in this
* file build a recorded DISPATCH decision naming "hostA"/"hostC" --
* harmless for tests that never reference these hosts. */
add_bare_worker(&(sched->core), "hostA");
add_bare_worker(&(sched->core), "hostC");
}
/* Same "leave the file open, skip sched_record_stop()'s own endSession
@ -1543,74 +1622,45 @@ test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatc
}
/*
* test_sim_verify_no_false_mismatch_across_round_separated_dispatch --
* scheduler-sim-eager-reconciliation spec, task 3.2 (Req 2.1, 2.2, 2.3,
* 4.2): --sim-verify against a replay where a job's recorded DISPATCH
* decision (round 1) and its recorded JOB_STATE_CHANGE-to-RUN confirmation
* (round 4, three rounds later) are genuinely separated -- the multi-round
* shape task 1.1 (this spec) fixed sim_reconcile_unaccounted_jobs() to
* capture correctly. Under the OLD gate (`!JOB_IS_RUN(job->state)` in
* addition to `job->accounts != NULL`), job 601 was still PEND at round 1,
* so reconciliation skipped it entirely -- decoded_session's own
* DISPATCH(601) entry was NEVER consumed, left over for --sim-verify's
* positional walk to compare against a live stream that never produces a
* DISPATCH at all (sim mode's own picker is PEND-only) -- a false
* MISMATCH. research.md's own "Interaction with --sim-verify" research log
* entry documents this exact latent risk and why widening the gate to
* `job->accounts == NULL` (state-independent) removes it rather than
* introducing a new one.
* test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_mismatch --
* RENAMED and INVERTED (2026-09-21, bug-1838, scheduler-driver-equivalence):
* this test used to be test_sim_verify_no_false_mismatch_across_round_
* separated_dispatch, proving scheduler-sim-eager-reconciliation's Req 4.2
* ("a dispatch decision and its RUN-state confirmation recorded several
* rounds apart must not produce a false --sim-verify mismatch") via
* sim_reconcile_unaccounted_jobs() force-applying job 601's recorded round-1
* DISPATCH decision while it was still PEND (the gate was state-independent
* at the time).
*
* **Correction (2026-09-07, bug-1759/bug-1779, scheduler-sim-decision-
* parity):** this fixture originally recorded a PEND(601) entry alongside
* round 1's DISPATCH(601), and again for rounds 2-3, reasoning that
* sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch()
* to retire job 601 from the live picker's own pendJobs -- so the live
* picker allegedly HAD to keep re-offering and re-PENDing it every round
* until the round-4 RUN confirmation removed it. That turned out to be a
* genuine, live-observed bug (bug-1759: a 50-job batch where 22
* already-dispatched jobs also received a PEND in the same round;
* bug-1779: a 10-element array job, 5 already-dispatched elements same
* shape), not an unavoidable consequence of the design -- it directly
* contradicts this same spec's own design.md "Integration with
* --sim-verify" section, which already documented the correct assumption
* ("the picker never re-decides an already-resolved job"). Fixed:
* sim_reconcile_unaccounted_jobs() now calls on_dispatch() the moment it
* reconstructs a job's allocation, mirroring sched_allocator()'s own live
* dispatch ordering (scheduler.c) exactly. The fixture below reflects the
* corrected shape.
* bug-1838 narrowed that gate back to JOB_IS_RUN-only (see sim_loop.c's own
* updated header comment on sim_reconcile_unaccounted_jobs() for the full
* rationale: scheduler-sim-live-allocation, completed after this spec, gave
* sim mode a genuine capacity-checked live picker/allocator path, making
* reconciliation's PEND-inclusive widening redundant for an ordinary
* cross-round gap -- the live path now independently re-derives the SAME
* decision the recording made). Job 601 here is freshly PEND at round 1, so
* it is no longer a reconciliation candidate at all -- but THIS FILE's own
* sched_job_picker()/sched_allocator() pair (see this file's header comment,
* "Genuine resource-based dispatch... is proved separately by
* test_sim_live_allocation.c") is a deliberate PEND-only stub, incapable of
* ever producing a live DISPATCH. With neither mechanism able to dispatch
* job 601 anymore, comparing the live stream's PEND(601) against the
* recorded DISPATCH(601) now correctly reports a MISMATCH -- not a false
* one: this fixture's own environment genuinely cannot reproduce what the
* recording claims happened, and --sim-verify is supposed to say so.
*
* Round shape (mirrors test_sim_contention_parity.c's own task 3.1 fixture
* numbers -- job A: dispatch round 1, RUN-confirmed round 4, a 3-round
* gap):
* Round 1 (begin_round=2000): job 601 pushed PEND; recorded decisions =
* [DISPATCH(601)]. sim_reconcile_unaccounted_jobs() retires job 601
* from the live picker's own pendJobs (on_dispatch()) BEFORE this
* round's picker session runs (sim_loop.c's own fixed ordering:
* reconcile, then xce_core_signal_go(), then drain/compare), so the
* live picker has nothing left to offer this round either -- no
* same-round PEND echo to record.
* Round 2, 3 (begin_round=3000/4000): job 601 is retired from the live
* picker's pendJobs (round 1's on_dispatch() above) -- nothing left for
* the real picker to offer or re-PEND. Recorded decisions = [] (empty
* ReplaySessions), the same shape round 4 below already uses for the
* identical reason. Still genuinely widens the dispatch-to-confirmation
* GAP to "several rounds" (Requirement 4.2) -- only the picker-echo
* noise in between changed, not the gap itself.
* Round 4 (begin_round=5000): job 601's JOB_STATE_CHANGE to RUN is
* applied as part of this round's own pushed event batch. Recorded
* decisions = [] (an empty ReplaySession, the same shape a genuinely
* idle round produces in a real recording).
*
* Zero mismatches across all four rounds is the Observable (Requirement
* 4.2). It is also consistent evidence for 2.1 (the DISPATCH decision IS
* reported, via reconciliation's own three-call sequence, at round 1 --
* not silently dropped or deferred to round 4) and 2.3 (no duplicate
* report: the live stream never independently emits a DISPATCH for job
* 601 at all, so there is nothing for reconciliation's own report to
* collide with).
* The genuine, mismatch-free proof of Req 4.2 (a real dispatch-then-later-
* confirmation gap causing no false mismatch) now lives in
* test_sim_live_allocation.c's test_sim_verify_no_false_mismatch_after_
* live_dispatch_and_later_confirmation, which links the real scheduler_srcs
* and can genuinely dispatch job 601 the way this file's stub never could
* -- see that test's own header comment for the full mirrored rationale.
* This test is kept, inverted, as a regression lock on the NEW behavior:
* reconciliation must not "help" a PEND job here anymore, even when a
* matching recorded DISPATCH entry exists.
*/
static void
test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void)
test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_mismatch(void)
{
char path[PATH_MAX];
FILE* f;
@ -1630,31 +1680,22 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void)
host[0] = make_test_host_alloc("hostA", res, 1);
alloc = make_test_job_alloc(601ULL, host, 1);
/* Round 1: DISPATCH recorded. bug-1759/bug-1779 correction:
* sim_reconcile_unaccounted_jobs() now calls the picker's on_dispatch()
* the moment it reconstructs job 601's allocation this round, retiring
* it from the live picker's own pendJobs BEFORE that round's picker
* session runs (sim_loop.c's fixed ordering: reconcile, then
* xce_core_signal_go()) -- so there is no longer a same-round PEND
* echo to record. */
/* Round 1: job 601 pushed PEND this same round; recorded decisions =
* [DISPATCH(601)]. bug-1838: sim_reconcile_unaccounted_jobs() is now
* JOB_IS_RUN-only, so it does not touch job 601 (still PEND) at all --
* this file's own PEND-only sched_allocator() stub (see this file's
* header comment) is the only thing that CAN decide it, and it always
* PENDs. The live stream therefore produces PEND(601) against a
* recorded DISPATCH(601) -- a genuine, correctly-reported mismatch. */
append_replay_session_with_dispatch(f, 2000ULL, alloc);
/* Round 2, 3: job 601 is now retired from the live picker's pendJobs
* (bug-1759/bug-1779 correction, same reasoning as round 4's own
* comment below) -- nothing left for the live picker to offer, so these
* rounds are recorded empty, the same shape a genuinely idle round
* produces. Still genuinely widens the dispatch-to-confirmation gap to
* "several rounds" (Requirement 4.2) -- only the picker-echo noise in
* between changed, not the gap itself. */
/* Round 2: job 601 is STILL a picker candidate (nothing retired it --
* reconciliation never touched it, and the stub allocator's on_pend()
* path doesn't retire candidates either), so the live stream produces
* ANOTHER PEND(601) here against this round's empty recorded
* decisions -- also a genuine mismatch, not asserted on individually
* (this test only checks that at least one mismatch is reported). */
append_empty_replay_session(f, 3000ULL);
append_empty_replay_session(f, 4000ULL);
/* Round 4: job 601's RUN confirmation, 3 rounds after its own round-1
* DISPATCH -- applied before this round's picker session runs, so the
* live picker has nothing left to offer. */
append_job_state_change_event(f, 4500ULL, 601ULL, JOB_STAT_PEND,
JOB_STAT_RUN);
append_empty_replay_session(f, 5000ULL);
append_end_of_stream_marker(f);
fclose(f);
@ -1662,15 +1703,17 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void)
captured = run_sim_and_capture_stdout(path, true, &rc);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NULL_MESSAGE(
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, "MISMATCH"),
"a dispatch decision and its RUN-state confirmation recorded "
"several rounds apart must not produce a false --sim-verify "
"mismatch (Requirement 4.2)");
"a PEND-state job's recorded DISPATCH decision must no longer be "
"force-applied by reconciliation (bug-1838) -- this file's "
"PEND-only live stub cannot reproduce it either, so --sim-verify "
"must correctly report a real mismatch here, not silently agree "
"with a decision nothing in this environment actually made");
free(captured);
printf("test_sim_verify_no_false_mismatch_across_round_separated_"
"dispatch: PASS\n");
printf("test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_"
"mismatch: PASS\n");
}
int main(void)
@ -1695,7 +1738,7 @@ int main(void)
RUN_TEST(
test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatch);
RUN_TEST(
test_sim_verify_no_false_mismatch_across_round_separated_dispatch);
test_reconciliation_no_longer_masks_a_pend_jobs_dispatch_mismatch);
return UNITY_END();
}