lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): retire reconciled jobs from picker candidates (bug-1759/1779)

Browse source 
sim_reconcile_unaccounted_jobs() (the mechanism that replays historical
DISPATCH decisions in --sim mode) reconstructed a job's allocation but
never called the picker's on_dispatch() hook, unlike the live allocator
path which always does. The job therefore stayed a live candidate in the
picker's own pendJobs bookkeeping, and sim_picker_thread_fn() (which
unconditionally PENDs whatever it is still offered) kept re-offering and
re-PENDing it every subsequent round -- live-observed as an already-
dispatched job receiving a bogus PEND decision (bug-1759: 22/50 jobs in
one batch; bug-1779: 5/10 array-job elements).

This directly contradicts scheduler-sim-decision-parity's own design.md,
which already documented "the picker never re-decides an already-resolved
job" as the correct assumption -- the implementation had drifted from it.
Fixed by calling on_dispatch() immediately after reconstructing the job's
allocation, mirroring sched_allocator()'s own live-dispatch ordering.

test_sim_verify_no_false_mismatch_across_round_separated_dispatch's own
fixture had baked in the stale-PEND symptom as "expected" data; updated to
match the corrected behavior (dispatch-only round 1, empty filler rounds).
Both design docs updated in place with correction notes.
This commit is contained in:
Rongsong Shen 2026-09-07 07:32:18 +00:00
parent ef3c4670a6
commit 0d3f925c4d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 213 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/scheduler/src/sim_loop.c
View file

@ -688,13 +688,36 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
replay_session_t* decoded_session,
sched_record_session_ctx_t* record_ctx)
{
iterator_t it;
int rc;
iterator_t it;
int rc;
xce_sched_plugin_t* picker = NULL;
if ((sim == NULL) || (decoded_session == NULL)) {
return;
}
/* bug-1759/bug-1779 (scheduler-sim-decision-parity correction, see that
* spec's design.md "Integration with --sim-verify" section, which
* already documented the assumption this restores: "the picker never
* re-decides an already-resolved job"): a match found below
* reconstructs this job's dispatch purely from replay data, but that
* must ALSO remove the job from the picker's OWN candidate bookkeeping
* (fcfs_queue_t.pendJobs), exactly like a LIVE dispatch does
* (sched_allocator()'s job_picker->picker_iface->on_dispatch() call,
* scheduler.c, BEFORE the decision is even built). Without this,
* sim_picker_thread_fn() -- a separate, always-running thread that
* walks next_job() every round and unconditionally PENDs whatever it is
* still offered, since it has no allocation awareness of its own --
* walked right past this "already reconciled" job on every subsequent
* round and emitted a bogus PEND decision for a job that had already
* been dispatched (live-observed: bug-1759's 50-job batch, bug-1779's
* 10-element array job). A NULL/unresolvable picker degrades to the
* pre-existing behavior (reconcile job->allocs/accounts as before, just
* without picker sync) rather than skipping reconciliation altogether --
* same "never load-bearing" posture every other picker lookup in this
* file family uses. */
(void)sched_resolve_job_picker(&(sim->core), &picker);
rc = iter_make(&it, sim->core.jobs);
if (rc < 0) {
return;
@ -751,6 +774,15 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
entry->body.dispatch->alloc = NULL;
job->accounts = make_job_account(job->allocs);
/* bug-1759/bug-1779: mirrors sched_allocator()'s own ordering
* (scheduler.c) as closely as this reconstruction allows --
* remove the job from the picker's candidate set now that it is
* confirmed dispatched, so a later round's sim_picker_thread_fn()
* never re-offers (and re-PENDs) it. */
if (picker != NULL) {
picker->picker_iface->on_dispatch(picker, job->jobid);
}
found = 1;
/* Memory ownership (design.md's Implementation Notes): this

85
src/scheduler/testsuite/test_sim_reconcile_unaccounted.c
View file

@ -2020,6 +2020,90 @@ test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch(
"stray_dispatch: PASS\n");
}
/*
* bug-1759/bug-1779: a reconciled (replay-reconstructed) dispatch must
* remove the job from the picker's OWN candidate bookkeeping (fcfs_queue_t.
* pendJobs), exactly like a LIVE dispatch does -- sched_allocator()
* (scheduler.c) always calls job_picker->picker_iface->on_dispatch() before
* building the decision, but sim_reconcile_unaccounted_jobs() (this file's
* subject) never did. Left unfixed, sim_picker_thread_fn() (sim_loop.c) --
* a separate, always-running thread that walks next_job() every round and
* unconditionally PENDs whatever it is still offered (it predates this
* reconciliation mechanism and was never taught about it) -- walks right
* past the "already reconciled" job on the very next round and emits a
* bogus PEND decision for a job that has already been dispatched. This is
* the live-observed root cause of both bug-1759 (50-job batch, 22
* already-dispatched jobs also received a PEND) and bug-1779 (10-element
* array job, 5 already-dispatched elements also received a PEND).
*
* Fixture: sim_bootstrap_picker() (sim_picker.c, bug-1567) is the same
* retroactive registration a real replay's initial snapshot bootstrap
* performs -- register builtin_fcfs, then replay the existing queue/job as
* picker candidates, so `job` genuinely sits in the picker's pendJobs
* exactly as it would after a live JOBS_PUSH, without needing a full
* xce_core_push_events() event round trip.
*/
static void
test_reconcile_removes_reconciled_job_from_picker_candidates(void)
{
xce_sched_sim_driver_t sim;
replay_decision_t* dispatch_entry;
replay_session_t* decoded_session;
xce_sched_job_t* job;
xce_sched_plugin_t* picker = NULL;
xce_sched_job_t* offered;
int rc;
fixture_init(&sim);
job = add_job(&(sim.core), 9101ULL, JOB_STAT_PEND, NULL);
TEST_ASSERT_NOT_NULL(job);
sim_bootstrap_picker(&sim);
rc = sched_resolve_job_picker(&(sim.core), &picker);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(picker);
/* Fixture sanity: the job is genuinely a live picker candidate before
* reconciliation runs. */
picker->picker_iface->begin_session(picker);
offered = picker->picker_iface->next_job(picker);
picker->picker_iface->end_session(picker);
TEST_ASSERT_NOT_NULL_MESSAGE(
offered, "fixture sanity: job must be a picker candidate before "
"reconciliation runs");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
9101ULL, offered->jobid,
"fixture sanity: the offered candidate must be this test's own job");
dispatch_entry = make_dispatch_entry(9101ULL);
decoded_session = make_decoded_session(0, dispatch_entry);
sim_reconcile_unaccounted_jobs(&sim, decoded_session, NULL);
TEST_ASSERT_NOT_NULL_MESSAGE(
job->allocs,
"reconciliation must still populate job->allocs exactly as before "
"this fix");
picker->picker_iface->begin_session(picker);
offered = picker->picker_iface->next_job(picker);
picker->picker_iface->end_session(picker);
TEST_ASSERT_NULL_MESSAGE(
offered,
"bug-1759/bug-1779: a job reconciled (dispatched) via replayed "
"decision data must be removed from the picker's candidate set, "
"exactly like a live dispatch (sched_allocator()'s on_dispatch() "
"call, scheduler.c) -- otherwise sim_picker_thread_fn()'s "
"unconditional-PEND thread re-offers and re-PENDs it on a later "
"round even though it has already been dispatched");
printf("test_reconcile_removes_reconciled_job_from_picker_candidates: "
"PASS\n");
}
int main(void)
{
UNITY_BEGIN();
@ -2053,5 +2137,6 @@ int main(void)
test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry);
RUN_TEST(
test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch);
RUN_TEST(test_reconcile_removes_reconciled_job_from_picker_candidates);
return UNITY_END();
}

129
src/scheduler/testsuite/test_sim_verify.c
View file

@ -845,6 +845,45 @@ static void append_replay_session_with_dispatch_and_pend(
capn_free(&c);
}
/* Appends one ReplaySession carrying a SINGLE recorded DISPATCH decision,
* with no accompanying PEND -- bug-1759/bug-1779 correction (scheduler-sim-
* decision-parity): sim_reconcile_unaccounted_jobs() now calls the picker's
* on_dispatch() the moment it reconstructs a job's allocation, retiring it
* from the picker's own pendJobs immediately -- so the live picker no
* longer independently re-offers/re-PENDs a job in the SAME round its
* DISPATCH is reconciled (previously append_replay_session_with_dispatch_
* and_pend(), just below, modeled that now-corrected stale-PEND echo). */
static void append_replay_session_with_dispatch(FILE* f,
uint64_t begin_round,
xce_job_alloc_t* dispatch_alloc)
{
replay_decision_t d_dispatch;
replay_decision_t* decisions[1];
replay_session_t session;
struct capn c;
struct capn_segment* cs;
ReplaySession_ptr sp;
d_dispatch = make_replay_dispatch_decision(dispatch_alloc);
d_dispatch.ts = begin_round;
decisions[0] = &d_dispatch;
memset(&session, 0, sizeof(session));
session.begin_round = begin_round;
session.decisions = decisions;
session.n_decisions = 1;
session.summary = NULL;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_ReplaySession_ptr(cs, &sp, &session);
capn_setp(capn_root(&c), 0, sp.p);
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
capn_free(&c);
}
/* Appends one ReplaySession carrying THREE recorded decisions -- a DISPATCH
* for `dispatch_alloc_a`'s jobid, a PEND for `pend_jobid`, and a DISPATCH
* for `dispatch_alloc_c`'s jobid (in that order: the still-PEND job's own
@ -1338,41 +1377,46 @@ test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatc
* `job->accounts == NULL` (state-independent) removes it rather than
* introducing a new one.
*
* **Correction (2026-09-07, bug-1759/bug-1779, scheduler-sim-decision-
* parity):** this fixture originally recorded a PEND(601) entry alongside
* round 1's DISPATCH(601), and again for rounds 2-3, reasoning that
* sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch()
* to retire job 601 from the live picker's own pendJobs -- so the live
* picker allegedly HAD to keep re-offering and re-PENDing it every round
* until the round-4 RUN confirmation removed it. That turned out to be a
* genuine, live-observed bug (bug-1759: a 50-job batch where 22
* already-dispatched jobs also received a PEND in the same round;
* bug-1779: a 10-element array job, 5 already-dispatched elements same
* shape), not an unavoidable consequence of the design -- it directly
* contradicts this same spec's own design.md "Integration with
* --sim-verify" section, which already documented the correct assumption
* ("the picker never re-decides an already-resolved job"). Fixed:
* sim_reconcile_unaccounted_jobs() now calls on_dispatch() the moment it
* reconstructs a job's allocation, mirroring sched_allocator()'s own live
* dispatch ordering (scheduler.c) exactly. The fixture below reflects the
* corrected shape.
*
* Round shape (mirrors test_sim_contention_parity.c's own task 3.1 fixture
* numbers -- job A: dispatch round 1, RUN-confirmed round 4, a 3-round
* gap):
* Round 1 (begin_round=2000): job 601 pushed PEND; recorded decisions =
* [DISPATCH(601), PEND(601)]. The recorded PEND(601) entry is not a
* contrivance: sim mode's own live picker (builtin_fcfs) has no way to
* learn a job was already dispatched in the original recording --
* sim_reconcile_unaccounted_jobs() captures job->accounts but never
* calls picker_iface->on_dispatch() to retire the job from the
* picker's own internal pendJobs (design.md's Non-Goals/Revalidation
* Triggers explicitly leave live-picker DISPATCH emission to a future
* scheduler-allocation spec) -- so job 601, still genuinely
* JOB_STAT_PEND at this round's picker session, IS independently
* re-offered and re-PENDed by the real picker thread even in the very
* round its DISPATCH is recorded. The recorded stream must include
* that same PEND(601) entry to stay verify-clean; reconciliation
* consumes the DISPATCH(601) entry ahead of the verify walk
* (sim_loop.c's own fixed ordering: reconcile, then
* xce_core_signal_go(), then drain/compare), leaving exactly the
* PEND(601) entry to align with the live picker's own single decision
* this round.
* Round 2, 3 (begin_round=3000/4000): job 601 is still genuinely PEND
* (no DISPATCH/state-change event touches it this round) -- the real
* picker re-offers and re-PENDs it every round, the same "no-op,
* retried next round" contract test_sim_contention_parity.c's own
* filler rounds already establish; recorded decisions = [PEND(601)]
* each round.
* [DISPATCH(601)]. sim_reconcile_unaccounted_jobs() retires job 601
* from the live picker's own pendJobs (on_dispatch()) BEFORE this
* round's picker session runs (sim_loop.c's own fixed ordering:
* reconcile, then xce_core_signal_go(), then drain/compare), so the
* live picker has nothing left to offer this round either -- no
* same-round PEND echo to record.
* Round 2, 3 (begin_round=3000/4000): job 601 is retired from the live
* picker's pendJobs (round 1's on_dispatch() above) -- nothing left for
* the real picker to offer or re-PEND. Recorded decisions = [] (empty
* ReplaySessions), the same shape round 4 below already uses for the
* identical reason. Still genuinely widens the dispatch-to-confirmation
* GAP to "several rounds" (Requirement 4.2) -- only the picker-echo
* noise in between changed, not the gap itself.
* Round 4 (begin_round=5000): job 601's JOB_STATE_CHANGE to RUN is
* applied as part of this round's own pushed event batch, strictly
* BEFORE the round's picker session begins (sim_loop.c's event-then-
* reconcile-then-signal-go ordering) -- builtin_fcfs_on_job_state_change()
* removes job 601 from pendJobs the moment it leaves PEND state, so the
* live picker has nothing left to offer this round. Recorded decisions
* = [] (an empty ReplaySession, the same shape a genuinely idle round
* produces in a real recording).
* applied as part of this round's own pushed event batch. Recorded
* decisions = [] (an empty ReplaySession, the same shape a genuinely
* idle round produces in a real recording).
*
* Zero mismatches across all four rounds is the Observable (Requirement
* 4.2). It is also consistent evidence for 2.1 (the DISPATCH decision IS
@ -1403,15 +1447,24 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void)
host[0] = make_test_host_alloc("hostA", res, 1);
alloc = make_test_job_alloc(601ULL, host, 1);
/* Round 1: DISPATCH recorded, plus the matching PEND the live picker
* independently re-produces this same round (see header comment). */
append_replay_session_with_dispatch_and_pend(f, 2000ULL, alloc, 601ULL,
0);
/* Round 1: DISPATCH recorded. bug-1759/bug-1779 correction:
* sim_reconcile_unaccounted_jobs() now calls the picker's on_dispatch()
* the moment it reconstructs job 601's allocation this round, retiring
* it from the live picker's own pendJobs BEFORE that round's picker
* session runs (sim_loop.c's fixed ordering: reconcile, then
* xce_core_signal_go()) -- so there is no longer a same-round PEND
* echo to record. */
append_replay_session_with_dispatch(f, 2000ULL, alloc);
/* Round 2, 3: job 601 still genuinely PEND -- filler rounds widening
* the dispatch-to-confirmation gap to genuinely "several rounds". */
append_replay_session_with_pend(f, 3000ULL, 601ULL, 0);
append_replay_session_with_pend(f, 4000ULL, 601ULL, 0);
/* Round 2, 3: job 601 is now retired from the live picker's pendJobs
* (bug-1759/bug-1779 correction, same reasoning as round 4's own
* comment below) -- nothing left for the live picker to offer, so these
* rounds are recorded empty, the same shape a genuinely idle round
* produces. Still genuinely widens the dispatch-to-confirmation gap to
* "several rounds" (Requirement 4.2) -- only the picker-echo noise in
* between changed, not the gap itself. */
append_empty_replay_session(f, 3000ULL);
append_empty_replay_session(f, 4000ULL);
/* Round 4: job 601's RUN confirmation, 3 rounds after its own round-1
* DISPATCH -- applied before this round's picker session runs, so the