fix(scheduler): retire reconciled jobs from picker candidates (bug-1759/1779)

sim_reconcile_unaccounted_jobs() (the mechanism that replays historical
DISPATCH decisions in --sim mode) reconstructed a job's allocation but
never called the picker's on_dispatch() hook, unlike the live allocator
path which always does. The job therefore stayed a live candidate in the
picker's own pendJobs bookkeeping, and sim_picker_thread_fn() (which
unconditionally PENDs whatever it is still offered) kept re-offering and
re-PENDing it every subsequent round -- live-observed as an already-
dispatched job receiving a bogus PEND decision (bug-1759: 22/50 jobs in
one batch; bug-1779: 5/10 array-job elements).

This directly contradicts scheduler-sim-decision-parity's own design.md,
which already documented "the picker never re-decides an already-resolved
job" as the correct assumption -- the implementation had drifted from it.
Fixed by calling on_dispatch() immediately after reconstructing the job's
allocation, mirroring sched_allocator()'s own live-dispatch ordering.

test_sim_verify_no_false_mismatch_across_round_separated_dispatch's own
fixture had baked in the stale-PEND symptom as "expected" data; updated to
match the corrected behavior (dispatch-only round 1, empty filler rounds).
Both design docs updated in place with correction notes.
This commit is contained in:
Rongsong Shen 2026-09-07 07:32:18 +00:00
parent ef3c4670a6
commit 0d3f925c4d
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 213 additions and 40 deletions

View file

@ -218,6 +218,7 @@ void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
**Implementation Notes**
- Integration with `--sim-verify` (CONFIRMED during design validation, resolving the prior "read the comparison driver at implementation time" placeholder): `sim_loop.c`'s verify branch (current code: the `while ((cbuf_get(&(sim->core.decisions), &d) == 0) ...)` loop) walks `decoded_session->decisions` *positionally* via an index `di`, comparing each live-drained decision `d` against `decoded_session->decisions[di]` and advancing `di` once per live decision. A recorded decision belonging to a job this round's reconciliation step consumes (i.e., a job whose allocation/decision visibility is being restored from the recording rather than freshly re-derived by the live picker) will **never** produce a corresponding live-drained `d` — the picker never re-decides an already-resolved job. Comparing positionally without accounting for this would misalign every subsequent entry and produce false mismatches (Requirement 5.2 violation), independent of any bug in the comparison logic itself.
- **Correction (2026-09-07, bug-1759/bug-1779):** "the picker never re-decides an already-resolved job" was stated here as an assumption but was not actually mechanically enforced — the reconciliation step reconstructed the job's allocation/accounts but never called `picker_iface->on_dispatch()`, so the job remained a live candidate in the picker's own `pendJobs` and WAS re-offered (and re-PENDed, via `sim_picker_thread_fn()`'s unconditional-PEND behavior) on every subsequent round until an unrelated RUN-confirmation event happened to remove it. Live-observed (bug-1759, bug-1759's 50-job batch; bug-1779, a 10-element array job) as a job showing a PEND decision in the same or later rounds despite already having been dispatched. Fixed: the reconciliation step (`sim_loop.c`'s FOUND branch) now calls `picker_iface->on_dispatch()` immediately after reconstructing `job->allocs`/`job->accounts`, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly. This does not change this section's positional --sim-verify reasoning (still correct, now actually true rather than merely assumed) and does not touch scheduling/allocation *policy* (this spec's own Non-Goals) — only picker candidate-bookkeeping consistency for a job whose dispatch was already decided (by the original live recording, not re-decided here).
- **Resolution**: the reconciliation step removes each recorded decision it consumes from `decoded_session->decisions` *before* the verify drain loop runs for that round — it already has the exact jobid-matched entry in hand from its own lookup, so this is a natural side effect of the same pass, not a second scan. This keeps the remaining (unconsumed) recorded entries positionally aligned with the live-drained stream exactly as they are today, when no reconciliation was needed for a job. No change to `sim_decision_eq()`/`sim_verify_job_alloc_eq()`/`sim_verify_job_preason_eq()` (the comparison predicates in `sim_verify.c`) is required — only `sim_loop.c`'s bookkeeping of which array entries reach that comparison.
- **Memory ownership (must not be overlooked)**: `decoded_session->decisions`/`n_decisions` (`replay_session_t`, `typemap.h`) are a plain heap-owned array — `free_ReplaySession_ptr()` frees it via `free_ReplayDecision_list(d->n_decisions, d->decisions)`, which walks and frees exactly the first `n_decisions` entries, then frees the array itself. Consequently, simply decrementing `n_decisions` to "drop" a consumed entry (without separately freeing that entry) is **not** safe compaction — it silently leaks that entry's payload every round a reconciliation happens, since the round-end free walk will never reach it again. The reconciliation step must instead: (1) extract the data it needs (see ownership sub-point immediately below — this is NOT a plain read for a DISPATCH entry), (2) call the existing `free_ReplayDecision(entry)` + `free(entry)` (the same two calls `free_ReplayDecision_list()` already makes per-element) on the one consumed entry, then (3) compact the array by shifting the remaining entries down over the removed slot (`memmove`), and decrement `n_decisions` by one. Shift-down, not swap-with-last: swap-with-last is the usual O(1) removal trick, but it would relocate the last entry into the removed slot and change its position relative to the other remaining entries — corrupting the very relative ordering the positional `di` comparison in the point above depends on. `decoded_session` is decoded fresh once per round regardless, so this compaction is confined to the current round with no cross-round effect.
- **Ownership hazard in step (1) above — a DISPATCH entry's `alloc` must be moved, not aliased, before step (2) frees the entry**: the established live-path precedent (`scheduler.c:1440`, `sched_commit_decision()`) is `job->allocs = decision->u.alloc;` — a pointer *takeover*, not a deep copy; `job->accounts` (`make_job_account(job->allocs)`) is then built directly on top of that same, now job-owned, memory. If reconciliation reconstructs `job->allocs` the same way from a recorded DISPATCH entry — `job->allocs = entry->body.dispatch->alloc;` (`sched_dispatch_decision_t.alloc`, an `xce_job_alloc_t*`, `typemap.h`) — that pointer is now aliased by BOTH `job->allocs` and the entry about to be freed in step (2): `free_ReplayDecision()` → `free_DispatchDecision()` → `free_JobAlloc_ptr(&d->alloc)` would free that exact `xce_job_alloc_t`, leaving `job->allocs` dangling immediately (and a double-free whenever the job's own allocation is later released through its normal lifecycle). **Required fix**: immediately after the takeover assignment, null out the source field — `entry->body.dispatch->alloc = NULL;` — before calling `free_ReplayDecision(entry)`; every `_ptr`-style free helper in this codebase (including `free_JobAlloc_ptr`) already no-ops on `NULL`, so this makes the subsequent free of the rest of the entry safe without a special case. This is DISPATCH-specific and, by construction, the *only* case this component needs to handle: the reconciliation trigger is a RUN-state job, and a job can only reach RUN-state via a recorded DISPATCH decision — a recorded PEND (or PREEMPT) entry for the same job in the same round is structurally impossible to match here, since PEND/PREEMPT are exactly the decisions that do *not* put a job into RUN-state. (Independently confirmed no `job->preason`/equivalent field exists to alias in the first place — `job->accounts`/`job->allocs` are DISPATCH-only concepts, consistent with this.) No PEND/PREEMPT ownership case needs to be designed or implemented.

View file

@ -19,6 +19,8 @@
- Any change to `--sim-verify`'s comparison semantics beyond not introducing a new false mismatch.
- Reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks, or any normal-mode (`driver.c`) behavior.
**Correction (2026-09-07, bug-1759/bug-1779):** the "live picker" Non-Goal above was never meant to license `sim_reconcile_unaccounted_jobs()` leaving a reconciled job as a live picker candidate — it means "do not give the live picker real resource-matching allocation logic" (still out of scope, still `scheduler-allocation`'s territory). Whether a reconciled job stays in the picker's own `pendJobs` bookkeeping is a `scheduler-sim-decision-parity` concern (see that spec's design.md, "Integration with `--sim-verify`", which already documented "the picker never re-decides an already-resolved job" as the correct assumption). The implementation had drifted from that documented assumption: `sim_reconcile_unaccounted_jobs()` never called `picker_iface->on_dispatch()`, so a reconciled job stayed a live candidate and was live-observed being re-offered and re-PENDed every subsequent round (bug-1759: 22/50 already-dispatched jobs in one batch; bug-1779: 5/10 array elements). Fixed by adding that missing `on_dispatch()` call, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly — a correction to already-agreed-upon behavior, not a new capability, so it did not need to reopen this spec's own requirements/tasks. `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` (`test_sim_verify.c`) was updated to match: rounds 2-3 of its fixture are now empty `ReplaySession`s (no more picker-echoed PEND for an already-reconciled job) rather than baking in the pre-fix stale-PEND behavior as "expected."
## Boundary Commitments
### This Spec Owns

View file

@ -688,13 +688,36 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
replay_session_t* decoded_session,
sched_record_session_ctx_t* record_ctx)
{
iterator_t it;
int rc;
iterator_t it;
int rc;
xce_sched_plugin_t* picker = NULL;
if ((sim == NULL) || (decoded_session == NULL)) {
return;
}
/* bug-1759/bug-1779 (scheduler-sim-decision-parity correction, see that
* spec's design.md "Integration with --sim-verify" section, which
* already documented the assumption this restores: "the picker never
* re-decides an already-resolved job"): a match found below
* reconstructs this job's dispatch purely from replay data, but that
* must ALSO remove the job from the picker's OWN candidate bookkeeping
* (fcfs_queue_t.pendJobs), exactly like a LIVE dispatch does
* (sched_allocator()'s job_picker->picker_iface->on_dispatch() call,
* scheduler.c, BEFORE the decision is even built). Without this,
* sim_picker_thread_fn() -- a separate, always-running thread that
* walks next_job() every round and unconditionally PENDs whatever it is
* still offered, since it has no allocation awareness of its own --
* walked right past this "already reconciled" job on every subsequent
* round and emitted a bogus PEND decision for a job that had already
* been dispatched (live-observed: bug-1759's 50-job batch, bug-1779's
* 10-element array job). A NULL/unresolvable picker degrades to the
* pre-existing behavior (reconcile job->allocs/accounts as before, just
* without picker sync) rather than skipping reconciliation altogether --
* same "never load-bearing" posture every other picker lookup in this
* file family uses. */
(void)sched_resolve_job_picker(&(sim->core), &picker);
rc = iter_make(&it, sim->core.jobs);
if (rc < 0) {
return;
@ -751,6 +774,15 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
entry->body.dispatch->alloc = NULL;
job->accounts = make_job_account(job->allocs);
/* bug-1759/bug-1779: mirrors sched_allocator()'s own ordering
* (scheduler.c) as closely as this reconstruction allows --
* remove the job from the picker's candidate set now that it is
* confirmed dispatched, so a later round's sim_picker_thread_fn()
* never re-offers (and re-PENDs) it. */
if (picker != NULL) {
picker->picker_iface->on_dispatch(picker, job->jobid);
}
found = 1;
/* Memory ownership (design.md's Implementation Notes): this

View file

@ -2020,6 +2020,90 @@ test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch(
"stray_dispatch: PASS\n");
}
/*
* bug-1759/bug-1779: a reconciled (replay-reconstructed) dispatch must
* remove the job from the picker's OWN candidate bookkeeping (fcfs_queue_t.
* pendJobs), exactly like a LIVE dispatch does -- sched_allocator()
* (scheduler.c) always calls job_picker->picker_iface->on_dispatch() before
* building the decision, but sim_reconcile_unaccounted_jobs() (this file's
* subject) never did. Left unfixed, sim_picker_thread_fn() (sim_loop.c) --
* a separate, always-running thread that walks next_job() every round and
* unconditionally PENDs whatever it is still offered (it predates this
* reconciliation mechanism and was never taught about it) -- walks right
* past the "already reconciled" job on the very next round and emits a
* bogus PEND decision for a job that has already been dispatched. This is
* the live-observed root cause of both bug-1759 (50-job batch, 22
* already-dispatched jobs also received a PEND) and bug-1779 (10-element
* array job, 5 already-dispatched elements also received a PEND).
*
* Fixture: sim_bootstrap_picker() (sim_picker.c, bug-1567) is the same
* retroactive registration a real replay's initial snapshot bootstrap
* performs -- register builtin_fcfs, then replay the existing queue/job as
* picker candidates, so `job` genuinely sits in the picker's pendJobs
* exactly as it would after a live JOBS_PUSH, without needing a full
* xce_core_push_events() event round trip.
*/
static void
test_reconcile_removes_reconciled_job_from_picker_candidates(void)
{
xce_sched_sim_driver_t sim;
replay_decision_t* dispatch_entry;
replay_session_t* decoded_session;
xce_sched_job_t* job;
xce_sched_plugin_t* picker = NULL;
xce_sched_job_t* offered;
int rc;
fixture_init(&sim);
job = add_job(&(sim.core), 9101ULL, JOB_STAT_PEND, NULL);
TEST_ASSERT_NOT_NULL(job);
sim_bootstrap_picker(&sim);
rc = sched_resolve_job_picker(&(sim.core), &picker);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(picker);
/* Fixture sanity: the job is genuinely a live picker candidate before
* reconciliation runs. */
picker->picker_iface->begin_session(picker);
offered = picker->picker_iface->next_job(picker);
picker->picker_iface->end_session(picker);
TEST_ASSERT_NOT_NULL_MESSAGE(
offered, "fixture sanity: job must be a picker candidate before "
"reconciliation runs");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
9101ULL, offered->jobid,
"fixture sanity: the offered candidate must be this test's own job");
dispatch_entry = make_dispatch_entry(9101ULL);
decoded_session = make_decoded_session(0, dispatch_entry);
sim_reconcile_unaccounted_jobs(&sim, decoded_session, NULL);
TEST_ASSERT_NOT_NULL_MESSAGE(
job->allocs,
"reconciliation must still populate job->allocs exactly as before "
"this fix");
picker->picker_iface->begin_session(picker);
offered = picker->picker_iface->next_job(picker);
picker->picker_iface->end_session(picker);
TEST_ASSERT_NULL_MESSAGE(
offered,
"bug-1759/bug-1779: a job reconciled (dispatched) via replayed "
"decision data must be removed from the picker's candidate set, "
"exactly like a live dispatch (sched_allocator()'s on_dispatch() "
"call, scheduler.c) -- otherwise sim_picker_thread_fn()'s "
"unconditional-PEND thread re-offers and re-PENDs it on a later "
"round even though it has already been dispatched");
printf("test_reconcile_removes_reconciled_job_from_picker_candidates: "
"PASS\n");
}
int main(void)
{
UNITY_BEGIN();
@ -2053,5 +2137,6 @@ int main(void)
test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry);
RUN_TEST(
test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch);
RUN_TEST(test_reconcile_removes_reconciled_job_from_picker_candidates);
return UNITY_END();
}

View file

@ -845,6 +845,45 @@ static void append_replay_session_with_dispatch_and_pend(
capn_free(&c);
}
/* Appends one ReplaySession carrying a SINGLE recorded DISPATCH decision,
* with no accompanying PEND -- bug-1759/bug-1779 correction (scheduler-sim-
* decision-parity): sim_reconcile_unaccounted_jobs() now calls the picker's
* on_dispatch() the moment it reconstructs a job's allocation, retiring it
* from the picker's own pendJobs immediately -- so the live picker no
* longer independently re-offers/re-PENDs a job in the SAME round its
* DISPATCH is reconciled (previously append_replay_session_with_dispatch_
* and_pend(), just below, modeled that now-corrected stale-PEND echo). */
static void append_replay_session_with_dispatch(FILE* f,
uint64_t begin_round,
xce_job_alloc_t* dispatch_alloc)
{
replay_decision_t d_dispatch;
replay_decision_t* decisions[1];
replay_session_t session;
struct capn c;
struct capn_segment* cs;
ReplaySession_ptr sp;
d_dispatch = make_replay_dispatch_decision(dispatch_alloc);
d_dispatch.ts = begin_round;
decisions[0] = &d_dispatch;
memset(&session, 0, sizeof(session));
session.begin_round = begin_round;
session.decisions = decisions;
session.n_decisions = 1;
session.summary = NULL;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_ReplaySession_ptr(cs, &sp, &session);
capn_setp(capn_root(&c), 0, sp.p);
TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c));
capn_free(&c);
}
/* Appends one ReplaySession carrying THREE recorded decisions -- a DISPATCH
* for `dispatch_alloc_a`'s jobid, a PEND for `pend_jobid`, and a DISPATCH
* for `dispatch_alloc_c`'s jobid (in that order: the still-PEND job's own
@ -1338,41 +1377,46 @@ test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatc
* `job->accounts == NULL` (state-independent) removes it rather than
* introducing a new one.
*
* **Correction (2026-09-07, bug-1759/bug-1779, scheduler-sim-decision-
* parity):** this fixture originally recorded a PEND(601) entry alongside
* round 1's DISPATCH(601), and again for rounds 2-3, reasoning that
* sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch()
* to retire job 601 from the live picker's own pendJobs -- so the live
* picker allegedly HAD to keep re-offering and re-PENDing it every round
* until the round-4 RUN confirmation removed it. That turned out to be a
* genuine, live-observed bug (bug-1759: a 50-job batch where 22
* already-dispatched jobs also received a PEND in the same round;
* bug-1779: a 10-element array job, 5 already-dispatched elements same
* shape), not an unavoidable consequence of the design -- it directly
* contradicts this same spec's own design.md "Integration with
* --sim-verify" section, which already documented the correct assumption
* ("the picker never re-decides an already-resolved job"). Fixed:
* sim_reconcile_unaccounted_jobs() now calls on_dispatch() the moment it
* reconstructs a job's allocation, mirroring sched_allocator()'s own live
* dispatch ordering (scheduler.c) exactly. The fixture below reflects the
* corrected shape.
*
* Round shape (mirrors test_sim_contention_parity.c's own task 3.1 fixture
* numbers -- job A: dispatch round 1, RUN-confirmed round 4, a 3-round
* gap):
* Round 1 (begin_round=2000): job 601 pushed PEND; recorded decisions =
* [DISPATCH(601), PEND(601)]. The recorded PEND(601) entry is not a
* contrivance: sim mode's own live picker (builtin_fcfs) has no way to
* learn a job was already dispatched in the original recording --
* sim_reconcile_unaccounted_jobs() captures job->accounts but never
* calls picker_iface->on_dispatch() to retire the job from the
* picker's own internal pendJobs (design.md's Non-Goals/Revalidation
* Triggers explicitly leave live-picker DISPATCH emission to a future
* scheduler-allocation spec) -- so job 601, still genuinely
* JOB_STAT_PEND at this round's picker session, IS independently
* re-offered and re-PENDed by the real picker thread even in the very
* round its DISPATCH is recorded. The recorded stream must include
* that same PEND(601) entry to stay verify-clean; reconciliation
* consumes the DISPATCH(601) entry ahead of the verify walk
* (sim_loop.c's own fixed ordering: reconcile, then
* xce_core_signal_go(), then drain/compare), leaving exactly the
* PEND(601) entry to align with the live picker's own single decision
* this round.
* Round 2, 3 (begin_round=3000/4000): job 601 is still genuinely PEND
* (no DISPATCH/state-change event touches it this round) -- the real
* picker re-offers and re-PENDs it every round, the same "no-op,
* retried next round" contract test_sim_contention_parity.c's own
* filler rounds already establish; recorded decisions = [PEND(601)]
* each round.
* [DISPATCH(601)]. sim_reconcile_unaccounted_jobs() retires job 601
* from the live picker's own pendJobs (on_dispatch()) BEFORE this
* round's picker session runs (sim_loop.c's own fixed ordering:
* reconcile, then xce_core_signal_go(), then drain/compare), so the
* live picker has nothing left to offer this round either -- no
* same-round PEND echo to record.
* Round 2, 3 (begin_round=3000/4000): job 601 is retired from the live
* picker's pendJobs (round 1's on_dispatch() above) -- nothing left for
* the real picker to offer or re-PEND. Recorded decisions = [] (empty
* ReplaySessions), the same shape round 4 below already uses for the
* identical reason. Still genuinely widens the dispatch-to-confirmation
* GAP to "several rounds" (Requirement 4.2) -- only the picker-echo
* noise in between changed, not the gap itself.
* Round 4 (begin_round=5000): job 601's JOB_STATE_CHANGE to RUN is
* applied as part of this round's own pushed event batch, strictly
* BEFORE the round's picker session begins (sim_loop.c's event-then-
* reconcile-then-signal-go ordering) -- builtin_fcfs_on_job_state_change()
* removes job 601 from pendJobs the moment it leaves PEND state, so the
* live picker has nothing left to offer this round. Recorded decisions
* = [] (an empty ReplaySession, the same shape a genuinely idle round
* produces in a real recording).
* applied as part of this round's own pushed event batch. Recorded
* decisions = [] (an empty ReplaySession, the same shape a genuinely
* idle round produces in a real recording).
*
* Zero mismatches across all four rounds is the Observable (Requirement
* 4.2). It is also consistent evidence for 2.1 (the DISPATCH decision IS
@ -1403,15 +1447,24 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void)
host[0] = make_test_host_alloc("hostA", res, 1);
alloc = make_test_job_alloc(601ULL, host, 1);
/* Round 1: DISPATCH recorded, plus the matching PEND the live picker
* independently re-produces this same round (see header comment). */
append_replay_session_with_dispatch_and_pend(f, 2000ULL, alloc, 601ULL,
0);
/* Round 1: DISPATCH recorded. bug-1759/bug-1779 correction:
* sim_reconcile_unaccounted_jobs() now calls the picker's on_dispatch()
* the moment it reconstructs job 601's allocation this round, retiring
* it from the live picker's own pendJobs BEFORE that round's picker
* session runs (sim_loop.c's fixed ordering: reconcile, then
* xce_core_signal_go()) -- so there is no longer a same-round PEND
* echo to record. */
append_replay_session_with_dispatch(f, 2000ULL, alloc);
/* Round 2, 3: job 601 still genuinely PEND -- filler rounds widening
* the dispatch-to-confirmation gap to genuinely "several rounds". */
append_replay_session_with_pend(f, 3000ULL, 601ULL, 0);
append_replay_session_with_pend(f, 4000ULL, 601ULL, 0);
/* Round 2, 3: job 601 is now retired from the live picker's pendJobs
* (bug-1759/bug-1779 correction, same reasoning as round 4's own
* comment below) -- nothing left for the live picker to offer, so these
* rounds are recorded empty, the same shape a genuinely idle round
* produces. Still genuinely widens the dispatch-to-confirmation gap to
* "several rounds" (Requirement 4.2) -- only the picker-echo noise in
* between changed, not the gap itself. */
append_empty_replay_session(f, 3000ULL);
append_empty_replay_session(f, 4000ULL);
/* Round 4: job 601's RUN confirmation, 3 rounds after its own round-1
* DISPATCH -- applied before this round's picker session runs, so the