lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): retire reconciled jobs from picker candidates (bug-1759/1779)

Browse source 
sim_reconcile_unaccounted_jobs() (the mechanism that replays historical
DISPATCH decisions in --sim mode) reconstructed a job's allocation but
never called the picker's on_dispatch() hook, unlike the live allocator
path which always does. The job therefore stayed a live candidate in the
picker's own pendJobs bookkeeping, and sim_picker_thread_fn() (which
unconditionally PENDs whatever it is still offered) kept re-offering and
re-PENDing it every subsequent round -- live-observed as an already-
dispatched job receiving a bogus PEND decision (bug-1759: 22/50 jobs in
one batch; bug-1779: 5/10 array-job elements).

This directly contradicts scheduler-sim-decision-parity's own design.md,
which already documented "the picker never re-decides an already-resolved
job" as the correct assumption -- the implementation had drifted from it.
Fixed by calling on_dispatch() immediately after reconstructing the job's
allocation, mirroring sched_allocator()'s own live-dispatch ordering.

test_sim_verify_no_false_mismatch_across_round_separated_dispatch's own
fixture had baked in the stale-PEND symptom as "expected" data; updated to
match the corrected behavior (dispatch-only round 1, empty filler rounds).
Both design docs updated in place with correction notes.
This commit is contained in:
Rongsong Shen 2026-09-07 07:32:18 +00:00
parent ef3c4670a6
commit 0d3f925c4d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 213 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

129
src/scheduler/testsuite/test_sim_verify.c
View file

@ -845,6 +845,45 @@ static void append_replay_session_with_dispatch_and_pend(
capn_free(&c);
}
/* Appends one ReplaySession carrying a SINGLE recorded DISPATCH decision,
* with no accompanying PEND -- bug-1759/bug-1779 correction (scheduler-sim-
* decision-parity): sim_reconcile_unaccounted_jobs() now calls the picker's
* on_dispatch() the moment it reconstructs a job's allocation, retiring it
* from the picker's own pendJobs immediately -- so the live picker no
* longer independently re-offers/re-PENDs a job in the SAME round its
* DISPATCH is reconciled (previously append_replay_session_with_dispatch_
* and_pend(), just below, modeled that now-corrected stale-PEND echo). */
static void append_replay_session_with_dispatch(FILE* f,
uint64_t begin_round,
xce_job_alloc_t* dispatch_alloc)
{
replay_decision_t d_dispatch;
replay_decision_t* decisions[1];
replay_session_t session;
struct capn c;
struct capn_segment* cs;
ReplaySession_ptr sp;
d_dispatch = make_replay_dispatch_decision(dispatch_alloc);
d_dispatch.ts = begin_round;
decisions[0] = &d_dispatch;
memset(&session, 0, sizeof(session));
session.begin_round = begin_round;
session.decisions = decisions;
session.n_decisions = 1;
session.summary = NULL;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_ReplaySession_ptr(cs, &sp, &session);
capn_setp(capn_root(&c), 0, sp.p);
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
capn_free(&c);
}
/* Appends one ReplaySession carrying THREE recorded decisions -- a DISPATCH
* for `dispatch_alloc_a`'s jobid, a PEND for `pend_jobid`, and a DISPATCH
* for `dispatch_alloc_c`'s jobid (in that order: the still-PEND job's own
@ -1338,41 +1377,46 @@ test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatc
* `job->accounts == NULL` (state-independent) removes it rather than
* introducing a new one.
*
* **Correction (2026-09-07, bug-1759/bug-1779, scheduler-sim-decision-
* parity):** this fixture originally recorded a PEND(601) entry alongside
* round 1's DISPATCH(601), and again for rounds 2-3, reasoning that
* sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch()
* to retire job 601 from the live picker's own pendJobs -- so the live
* picker allegedly HAD to keep re-offering and re-PENDing it every round
* until the round-4 RUN confirmation removed it. That turned out to be a
* genuine, live-observed bug (bug-1759: a 50-job batch where 22
* already-dispatched jobs also received a PEND in the same round;
* bug-1779: a 10-element array job, 5 already-dispatched elements same
* shape), not an unavoidable consequence of the design -- it directly
* contradicts this same spec's own design.md "Integration with
* --sim-verify" section, which already documented the correct assumption
* ("the picker never re-decides an already-resolved job"). Fixed:
* sim_reconcile_unaccounted_jobs() now calls on_dispatch() the moment it
* reconstructs a job's allocation, mirroring sched_allocator()'s own live
* dispatch ordering (scheduler.c) exactly. The fixture below reflects the
* corrected shape.
*
* Round shape (mirrors test_sim_contention_parity.c's own task 3.1 fixture
* numbers -- job A: dispatch round 1, RUN-confirmed round 4, a 3-round
* gap):
* Round 1 (begin_round=2000): job 601 pushed PEND; recorded decisions =
* [DISPATCH(601), PEND(601)]. The recorded PEND(601) entry is not a
* contrivance: sim mode's own live picker (builtin_fcfs) has no way to
* learn a job was already dispatched in the original recording --
* sim_reconcile_unaccounted_jobs() captures job->accounts but never
* calls picker_iface->on_dispatch() to retire the job from the
* picker's own internal pendJobs (design.md's Non-Goals/Revalidation
* Triggers explicitly leave live-picker DISPATCH emission to a future
* scheduler-allocation spec) -- so job 601, still genuinely
* JOB_STAT_PEND at this round's picker session, IS independently
* re-offered and re-PENDed by the real picker thread even in the very
* round its DISPATCH is recorded. The recorded stream must include
* that same PEND(601) entry to stay verify-clean; reconciliation
* consumes the DISPATCH(601) entry ahead of the verify walk
* (sim_loop.c's own fixed ordering: reconcile, then
* xce_core_signal_go(), then drain/compare), leaving exactly the
* PEND(601) entry to align with the live picker's own single decision
* this round.
* Round 2, 3 (begin_round=3000/4000): job 601 is still genuinely PEND
* (no DISPATCH/state-change event touches it this round) -- the real
* picker re-offers and re-PENDs it every round, the same "no-op,
* retried next round" contract test_sim_contention_parity.c's own
* filler rounds already establish; recorded decisions = [PEND(601)]
* each round.
* [DISPATCH(601)]. sim_reconcile_unaccounted_jobs() retires job 601
* from the live picker's own pendJobs (on_dispatch()) BEFORE this
* round's picker session runs (sim_loop.c's own fixed ordering:
* reconcile, then xce_core_signal_go(), then drain/compare), so the
* live picker has nothing left to offer this round either -- no
* same-round PEND echo to record.
* Round 2, 3 (begin_round=3000/4000): job 601 is retired from the live
* picker's pendJobs (round 1's on_dispatch() above) -- nothing left for
* the real picker to offer or re-PEND. Recorded decisions = [] (empty
* ReplaySessions), the same shape round 4 below already uses for the
* identical reason. Still genuinely widens the dispatch-to-confirmation
* GAP to "several rounds" (Requirement 4.2) -- only the picker-echo
* noise in between changed, not the gap itself.
* Round 4 (begin_round=5000): job 601's JOB_STATE_CHANGE to RUN is
* applied as part of this round's own pushed event batch, strictly
* BEFORE the round's picker session begins (sim_loop.c's event-then-
* reconcile-then-signal-go ordering) -- builtin_fcfs_on_job_state_change()
* removes job 601 from pendJobs the moment it leaves PEND state, so the
* live picker has nothing left to offer this round. Recorded decisions
* = [] (an empty ReplaySession, the same shape a genuinely idle round
* produces in a real recording).
* applied as part of this round's own pushed event batch. Recorded
* decisions = [] (an empty ReplaySession, the same shape a genuinely
* idle round produces in a real recording).
*
* Zero mismatches across all four rounds is the Observable (Requirement
* 4.2). It is also consistent evidence for 2.1 (the DISPATCH decision IS
@ -1403,15 +1447,24 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void)
host[0] = make_test_host_alloc("hostA", res, 1);
alloc = make_test_job_alloc(601ULL, host, 1);
/* Round 1: DISPATCH recorded, plus the matching PEND the live picker
* independently re-produces this same round (see header comment). */
append_replay_session_with_dispatch_and_pend(f, 2000ULL, alloc, 601ULL,
0);
/* Round 1: DISPATCH recorded. bug-1759/bug-1779 correction:
* sim_reconcile_unaccounted_jobs() now calls the picker's on_dispatch()
* the moment it reconstructs job 601's allocation this round, retiring
* it from the live picker's own pendJobs BEFORE that round's picker
* session runs (sim_loop.c's fixed ordering: reconcile, then
* xce_core_signal_go()) -- so there is no longer a same-round PEND
* echo to record. */
append_replay_session_with_dispatch(f, 2000ULL, alloc);
/* Round 2, 3: job 601 still genuinely PEND -- filler rounds widening
* the dispatch-to-confirmation gap to genuinely "several rounds". */
append_replay_session_with_pend(f, 3000ULL, 601ULL, 0);
append_replay_session_with_pend(f, 4000ULL, 601ULL, 0);
/* Round 2, 3: job 601 is now retired from the live picker's pendJobs
* (bug-1759/bug-1779 correction, same reasoning as round 4's own
* comment below) -- nothing left for the live picker to offer, so these
* rounds are recorded empty, the same shape a genuinely idle round
* produces. Still genuinely widens the dispatch-to-confirmation gap to
* "several rounds" (Requirement 4.2) -- only the picker-echo noise in
* between changed, not the gap itself. */
append_empty_replay_session(f, 3000ULL);
append_empty_replay_session(f, 4000ULL);
/* Round 4: job 601's RUN confirmation, 3 rounds after its own round-1
* DISPATCH -- applied before this round's picker session runs, so the