From 0d3f925c4dce50be1d125e6d47f733600198e384 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 7 Sep 2026 07:32:18 +0000 Subject: [PATCH] fix(scheduler): retire reconciled jobs from picker candidates (bug-1759/1779) sim_reconcile_unaccounted_jobs() (the mechanism that replays historical DISPATCH decisions in --sim mode) reconstructed a job's allocation but never called the picker's on_dispatch() hook, unlike the live allocator path which always does. The job therefore stayed a live candidate in the picker's own pendJobs bookkeeping, and sim_picker_thread_fn() (which unconditionally PENDs whatever it is still offered) kept re-offering and re-PENDing it every subsequent round -- live-observed as an already- dispatched job receiving a bogus PEND decision (bug-1759: 22/50 jobs in one batch; bug-1779: 5/10 array-job elements). This directly contradicts scheduler-sim-decision-parity's own design.md, which already documented "the picker never re-decides an already-resolved job" as the correct assumption -- the implementation had drifted from it. Fixed by calling on_dispatch() immediately after reconstructing the job's allocation, mirroring sched_allocator()'s own live-dispatch ordering. test_sim_verify_no_false_mismatch_across_round_separated_dispatch's own fixture had baked in the stale-PEND symptom as "expected" data; updated to match the corrected behavior (dispatch-only round 1, empty filler rounds). Both design docs updated in place with correction notes. --- .../scheduler-sim-decision-parity/design.md | 1 + .../design.md | 2 + src/scheduler/src/sim_loop.c | 36 ++++- .../test_sim_reconcile_unaccounted.c | 85 ++++++++++++ src/scheduler/testsuite/test_sim_verify.c | 129 ++++++++++++------ 5 files changed, 213 insertions(+), 40 deletions(-) diff --git a/.kiro/specs/scheduler-sim-decision-parity/design.md b/.kiro/specs/scheduler-sim-decision-parity/design.md index e23aa6a1..53ed0bd4 100644 --- a/.kiro/specs/scheduler-sim-decision-parity/design.md +++ b/.kiro/specs/scheduler-sim-decision-parity/design.md @@ -218,6 +218,7 @@ void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, **Implementation Notes** - Integration with `--sim-verify` (CONFIRMED during design validation, resolving the prior "read the comparison driver at implementation time" placeholder): `sim_loop.c`'s verify branch (current code: the `while ((cbuf_get(&(sim->core.decisions), &d) == 0) ...)` loop) walks `decoded_session->decisions` *positionally* via an index `di`, comparing each live-drained decision `d` against `decoded_session->decisions[di]` and advancing `di` once per live decision. A recorded decision belonging to a job this round's reconciliation step consumes (i.e., a job whose allocation/decision visibility is being restored from the recording rather than freshly re-derived by the live picker) will **never** produce a corresponding live-drained `d` — the picker never re-decides an already-resolved job. Comparing positionally without accounting for this would misalign every subsequent entry and produce false mismatches (Requirement 5.2 violation), independent of any bug in the comparison logic itself. + - **Correction (2026-09-07, bug-1759/bug-1779):** "the picker never re-decides an already-resolved job" was stated here as an assumption but was not actually mechanically enforced — the reconciliation step reconstructed the job's allocation/accounts but never called `picker_iface->on_dispatch()`, so the job remained a live candidate in the picker's own `pendJobs` and WAS re-offered (and re-PENDed, via `sim_picker_thread_fn()`'s unconditional-PEND behavior) on every subsequent round until an unrelated RUN-confirmation event happened to remove it. Live-observed (bug-1759, bug-1759's 50-job batch; bug-1779, a 10-element array job) as a job showing a PEND decision in the same or later rounds despite already having been dispatched. Fixed: the reconciliation step (`sim_loop.c`'s FOUND branch) now calls `picker_iface->on_dispatch()` immediately after reconstructing `job->allocs`/`job->accounts`, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly. This does not change this section's positional --sim-verify reasoning (still correct, now actually true rather than merely assumed) and does not touch scheduling/allocation *policy* (this spec's own Non-Goals) — only picker candidate-bookkeeping consistency for a job whose dispatch was already decided (by the original live recording, not re-decided here). - **Resolution**: the reconciliation step removes each recorded decision it consumes from `decoded_session->decisions` *before* the verify drain loop runs for that round — it already has the exact jobid-matched entry in hand from its own lookup, so this is a natural side effect of the same pass, not a second scan. This keeps the remaining (unconsumed) recorded entries positionally aligned with the live-drained stream exactly as they are today, when no reconciliation was needed for a job. No change to `sim_decision_eq()`/`sim_verify_job_alloc_eq()`/`sim_verify_job_preason_eq()` (the comparison predicates in `sim_verify.c`) is required — only `sim_loop.c`'s bookkeeping of which array entries reach that comparison. - **Memory ownership (must not be overlooked)**: `decoded_session->decisions`/`n_decisions` (`replay_session_t`, `typemap.h`) are a plain heap-owned array — `free_ReplaySession_ptr()` frees it via `free_ReplayDecision_list(d->n_decisions, d->decisions)`, which walks and frees exactly the first `n_decisions` entries, then frees the array itself. Consequently, simply decrementing `n_decisions` to "drop" a consumed entry (without separately freeing that entry) is **not** safe compaction — it silently leaks that entry's payload every round a reconciliation happens, since the round-end free walk will never reach it again. The reconciliation step must instead: (1) extract the data it needs (see ownership sub-point immediately below — this is NOT a plain read for a DISPATCH entry), (2) call the existing `free_ReplayDecision(entry)` + `free(entry)` (the same two calls `free_ReplayDecision_list()` already makes per-element) on the one consumed entry, then (3) compact the array by shifting the remaining entries down over the removed slot (`memmove`), and decrement `n_decisions` by one. Shift-down, not swap-with-last: swap-with-last is the usual O(1) removal trick, but it would relocate the last entry into the removed slot and change its position relative to the other remaining entries — corrupting the very relative ordering the positional `di` comparison in the point above depends on. `decoded_session` is decoded fresh once per round regardless, so this compaction is confined to the current round with no cross-round effect. - **Ownership hazard in step (1) above — a DISPATCH entry's `alloc` must be moved, not aliased, before step (2) frees the entry**: the established live-path precedent (`scheduler.c:1440`, `sched_commit_decision()`) is `job->allocs = decision->u.alloc;` — a pointer *takeover*, not a deep copy; `job->accounts` (`make_job_account(job->allocs)`) is then built directly on top of that same, now job-owned, memory. If reconciliation reconstructs `job->allocs` the same way from a recorded DISPATCH entry — `job->allocs = entry->body.dispatch->alloc;` (`sched_dispatch_decision_t.alloc`, an `xce_job_alloc_t*`, `typemap.h`) — that pointer is now aliased by BOTH `job->allocs` and the entry about to be freed in step (2): `free_ReplayDecision()` → `free_DispatchDecision()` → `free_JobAlloc_ptr(&d->alloc)` would free that exact `xce_job_alloc_t`, leaving `job->allocs` dangling immediately (and a double-free whenever the job's own allocation is later released through its normal lifecycle). **Required fix**: immediately after the takeover assignment, null out the source field — `entry->body.dispatch->alloc = NULL;` — before calling `free_ReplayDecision(entry)`; every `_ptr`-style free helper in this codebase (including `free_JobAlloc_ptr`) already no-ops on `NULL`, so this makes the subsequent free of the rest of the entry safe without a special case. This is DISPATCH-specific and, by construction, the *only* case this component needs to handle: the reconciliation trigger is a RUN-state job, and a job can only reach RUN-state via a recorded DISPATCH decision — a recorded PEND (or PREEMPT) entry for the same job in the same round is structurally impossible to match here, since PEND/PREEMPT are exactly the decisions that do *not* put a job into RUN-state. (Independently confirmed no `job->preason`/equivalent field exists to alias in the first place — `job->accounts`/`job->allocs` are DISPATCH-only concepts, consistent with this.) No PEND/PREEMPT ownership case needs to be designed or implemented. diff --git a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md index 12dcbbfc..b0841f48 100644 --- a/.kiro/specs/scheduler-sim-eager-reconciliation/design.md +++ b/.kiro/specs/scheduler-sim-eager-reconciliation/design.md @@ -19,6 +19,8 @@ - Any change to `--sim-verify`'s comparison semantics beyond not introducing a new false mismatch. - Reopening or modifying `scheduler-sim-decision-parity`'s own requirements, design, or tasks, or any normal-mode (`driver.c`) behavior. +**Correction (2026-09-07, bug-1759/bug-1779):** the "live picker" Non-Goal above was never meant to license `sim_reconcile_unaccounted_jobs()` leaving a reconciled job as a live picker candidate — it means "do not give the live picker real resource-matching allocation logic" (still out of scope, still `scheduler-allocation`'s territory). Whether a reconciled job stays in the picker's own `pendJobs` bookkeeping is a `scheduler-sim-decision-parity` concern (see that spec's design.md, "Integration with `--sim-verify`", which already documented "the picker never re-decides an already-resolved job" as the correct assumption). The implementation had drifted from that documented assumption: `sim_reconcile_unaccounted_jobs()` never called `picker_iface->on_dispatch()`, so a reconciled job stayed a live candidate and was live-observed being re-offered and re-PENDed every subsequent round (bug-1759: 22/50 already-dispatched jobs in one batch; bug-1779: 5/10 array elements). Fixed by adding that missing `on_dispatch()` call, mirroring `sched_allocator()`'s own live-dispatch ordering (`scheduler.c`) exactly — a correction to already-agreed-upon behavior, not a new capability, so it did not need to reopen this spec's own requirements/tasks. `test_sim_verify_no_false_mismatch_across_round_separated_dispatch` (`test_sim_verify.c`) was updated to match: rounds 2-3 of its fixture are now empty `ReplaySession`s (no more picker-echoed PEND for an already-reconciled job) rather than baking in the pre-fix stale-PEND behavior as "expected." + ## Boundary Commitments ### This Spec Owns diff --git a/src/scheduler/src/sim_loop.c b/src/scheduler/src/sim_loop.c index ce9ccdff..7f47837a 100644 --- a/src/scheduler/src/sim_loop.c +++ b/src/scheduler/src/sim_loop.c @@ -688,13 +688,36 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, replay_session_t* decoded_session, sched_record_session_ctx_t* record_ctx) { - iterator_t it; - int rc; + iterator_t it; + int rc; + xce_sched_plugin_t* picker = NULL; if ((sim == NULL) || (decoded_session == NULL)) { return; } + /* bug-1759/bug-1779 (scheduler-sim-decision-parity correction, see that + * spec's design.md "Integration with --sim-verify" section, which + * already documented the assumption this restores: "the picker never + * re-decides an already-resolved job"): a match found below + * reconstructs this job's dispatch purely from replay data, but that + * must ALSO remove the job from the picker's OWN candidate bookkeeping + * (fcfs_queue_t.pendJobs), exactly like a LIVE dispatch does + * (sched_allocator()'s job_picker->picker_iface->on_dispatch() call, + * scheduler.c, BEFORE the decision is even built). Without this, + * sim_picker_thread_fn() -- a separate, always-running thread that + * walks next_job() every round and unconditionally PENDs whatever it is + * still offered, since it has no allocation awareness of its own -- + * walked right past this "already reconciled" job on every subsequent + * round and emitted a bogus PEND decision for a job that had already + * been dispatched (live-observed: bug-1759's 50-job batch, bug-1779's + * 10-element array job). A NULL/unresolvable picker degrades to the + * pre-existing behavior (reconcile job->allocs/accounts as before, just + * without picker sync) rather than skipping reconciliation altogether -- + * same "never load-bearing" posture every other picker lookup in this + * file family uses. */ + (void)sched_resolve_job_picker(&(sim->core), &picker); + rc = iter_make(&it, sim->core.jobs); if (rc < 0) { return; @@ -751,6 +774,15 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim, entry->body.dispatch->alloc = NULL; job->accounts = make_job_account(job->allocs); + /* bug-1759/bug-1779: mirrors sched_allocator()'s own ordering + * (scheduler.c) as closely as this reconstruction allows -- + * remove the job from the picker's candidate set now that it is + * confirmed dispatched, so a later round's sim_picker_thread_fn() + * never re-offers (and re-PENDs) it. */ + if (picker != NULL) { + picker->picker_iface->on_dispatch(picker, job->jobid); + } + found = 1; /* Memory ownership (design.md's Implementation Notes): this diff --git a/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c b/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c index 76e1cc03..bd1e2f35 100644 --- a/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c +++ b/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c @@ -2020,6 +2020,90 @@ test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch( "stray_dispatch: PASS\n"); } +/* + * bug-1759/bug-1779: a reconciled (replay-reconstructed) dispatch must + * remove the job from the picker's OWN candidate bookkeeping (fcfs_queue_t. + * pendJobs), exactly like a LIVE dispatch does -- sched_allocator() + * (scheduler.c) always calls job_picker->picker_iface->on_dispatch() before + * building the decision, but sim_reconcile_unaccounted_jobs() (this file's + * subject) never did. Left unfixed, sim_picker_thread_fn() (sim_loop.c) -- + * a separate, always-running thread that walks next_job() every round and + * unconditionally PENDs whatever it is still offered (it predates this + * reconciliation mechanism and was never taught about it) -- walks right + * past the "already reconciled" job on the very next round and emits a + * bogus PEND decision for a job that has already been dispatched. This is + * the live-observed root cause of both bug-1759 (50-job batch, 22 + * already-dispatched jobs also received a PEND) and bug-1779 (10-element + * array job, 5 already-dispatched elements also received a PEND). + * + * Fixture: sim_bootstrap_picker() (sim_picker.c, bug-1567) is the same + * retroactive registration a real replay's initial snapshot bootstrap + * performs -- register builtin_fcfs, then replay the existing queue/job as + * picker candidates, so `job` genuinely sits in the picker's pendJobs + * exactly as it would after a live JOBS_PUSH, without needing a full + * xce_core_push_events() event round trip. + */ +static void +test_reconcile_removes_reconciled_job_from_picker_candidates(void) +{ + xce_sched_sim_driver_t sim; + replay_decision_t* dispatch_entry; + replay_session_t* decoded_session; + xce_sched_job_t* job; + xce_sched_plugin_t* picker = NULL; + xce_sched_job_t* offered; + int rc; + + fixture_init(&sim); + + job = add_job(&(sim.core), 9101ULL, JOB_STAT_PEND, NULL); + TEST_ASSERT_NOT_NULL(job); + + sim_bootstrap_picker(&sim); + + rc = sched_resolve_job_picker(&(sim.core), &picker); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_NOT_NULL(picker); + + /* Fixture sanity: the job is genuinely a live picker candidate before + * reconciliation runs. */ + picker->picker_iface->begin_session(picker); + offered = picker->picker_iface->next_job(picker); + picker->picker_iface->end_session(picker); + TEST_ASSERT_NOT_NULL_MESSAGE( + offered, "fixture sanity: job must be a picker candidate before " + "reconciliation runs"); + TEST_ASSERT_EQUAL_UINT64_MESSAGE( + 9101ULL, offered->jobid, + "fixture sanity: the offered candidate must be this test's own job"); + + dispatch_entry = make_dispatch_entry(9101ULL); + decoded_session = make_decoded_session(0, dispatch_entry); + + sim_reconcile_unaccounted_jobs(&sim, decoded_session, NULL); + + TEST_ASSERT_NOT_NULL_MESSAGE( + job->allocs, + "reconciliation must still populate job->allocs exactly as before " + "this fix"); + + picker->picker_iface->begin_session(picker); + offered = picker->picker_iface->next_job(picker); + picker->picker_iface->end_session(picker); + + TEST_ASSERT_NULL_MESSAGE( + offered, + "bug-1759/bug-1779: a job reconciled (dispatched) via replayed " + "decision data must be removed from the picker's candidate set, " + "exactly like a live dispatch (sched_allocator()'s on_dispatch() " + "call, scheduler.c) -- otherwise sim_picker_thread_fn()'s " + "unconditional-PEND thread re-offers and re-PENDs it on a later " + "round even though it has already been dispatched"); + + printf("test_reconcile_removes_reconciled_job_from_picker_candidates: " + "PASS\n"); +} + int main(void) { UNITY_BEGIN(); @@ -2053,5 +2137,6 @@ int main(void) test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry); RUN_TEST( test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch); + RUN_TEST(test_reconcile_removes_reconciled_job_from_picker_candidates); return UNITY_END(); } diff --git a/src/scheduler/testsuite/test_sim_verify.c b/src/scheduler/testsuite/test_sim_verify.c index df747ee6..0c76dac2 100644 --- a/src/scheduler/testsuite/test_sim_verify.c +++ b/src/scheduler/testsuite/test_sim_verify.c @@ -845,6 +845,45 @@ static void append_replay_session_with_dispatch_and_pend( capn_free(&c); } +/* Appends one ReplaySession carrying a SINGLE recorded DISPATCH decision, + * with no accompanying PEND -- bug-1759/bug-1779 correction (scheduler-sim- + * decision-parity): sim_reconcile_unaccounted_jobs() now calls the picker's + * on_dispatch() the moment it reconstructs a job's allocation, retiring it + * from the picker's own pendJobs immediately -- so the live picker no + * longer independently re-offers/re-PENDs a job in the SAME round its + * DISPATCH is reconciled (previously append_replay_session_with_dispatch_ + * and_pend(), just below, modeled that now-corrected stale-PEND echo). */ +static void append_replay_session_with_dispatch(FILE* f, + uint64_t begin_round, + xce_job_alloc_t* dispatch_alloc) +{ + replay_decision_t d_dispatch; + replay_decision_t* decisions[1]; + replay_session_t session; + struct capn c; + struct capn_segment* cs; + ReplaySession_ptr sp; + + d_dispatch = make_replay_dispatch_decision(dispatch_alloc); + d_dispatch.ts = begin_round; + + decisions[0] = &d_dispatch; + + memset(&session, 0, sizeof(session)); + session.begin_round = begin_round; + session.decisions = decisions; + session.n_decisions = 1; + session.summary = NULL; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + encode_ReplaySession_ptr(cs, &sp, &session); + capn_setp(capn_root(&c), 0, sp.p); + + TEST_ASSERT_EQUAL_INT(0, write_capnp_message(f, &c)); + capn_free(&c); +} + /* Appends one ReplaySession carrying THREE recorded decisions -- a DISPATCH * for `dispatch_alloc_a`'s jobid, a PEND for `pend_jobid`, and a DISPATCH * for `dispatch_alloc_c`'s jobid (in that order: the still-PEND job's own @@ -1338,41 +1377,46 @@ test_verify_with_multiple_already_resolved_jobs_across_rounds_reports_no_mismatc * `job->accounts == NULL` (state-independent) removes it rather than * introducing a new one. * + * **Correction (2026-09-07, bug-1759/bug-1779, scheduler-sim-decision- + * parity):** this fixture originally recorded a PEND(601) entry alongside + * round 1's DISPATCH(601), and again for rounds 2-3, reasoning that + * sim_reconcile_unaccounted_jobs() never called picker_iface->on_dispatch() + * to retire job 601 from the live picker's own pendJobs -- so the live + * picker allegedly HAD to keep re-offering and re-PENDing it every round + * until the round-4 RUN confirmation removed it. That turned out to be a + * genuine, live-observed bug (bug-1759: a 50-job batch where 22 + * already-dispatched jobs also received a PEND in the same round; + * bug-1779: a 10-element array job, 5 already-dispatched elements same + * shape), not an unavoidable consequence of the design -- it directly + * contradicts this same spec's own design.md "Integration with + * --sim-verify" section, which already documented the correct assumption + * ("the picker never re-decides an already-resolved job"). Fixed: + * sim_reconcile_unaccounted_jobs() now calls on_dispatch() the moment it + * reconstructs a job's allocation, mirroring sched_allocator()'s own live + * dispatch ordering (scheduler.c) exactly. The fixture below reflects the + * corrected shape. + * * Round shape (mirrors test_sim_contention_parity.c's own task 3.1 fixture * numbers -- job A: dispatch round 1, RUN-confirmed round 4, a 3-round * gap): * Round 1 (begin_round=2000): job 601 pushed PEND; recorded decisions = - * [DISPATCH(601), PEND(601)]. The recorded PEND(601) entry is not a - * contrivance: sim mode's own live picker (builtin_fcfs) has no way to - * learn a job was already dispatched in the original recording -- - * sim_reconcile_unaccounted_jobs() captures job->accounts but never - * calls picker_iface->on_dispatch() to retire the job from the - * picker's own internal pendJobs (design.md's Non-Goals/Revalidation - * Triggers explicitly leave live-picker DISPATCH emission to a future - * scheduler-allocation spec) -- so job 601, still genuinely - * JOB_STAT_PEND at this round's picker session, IS independently - * re-offered and re-PENDed by the real picker thread even in the very - * round its DISPATCH is recorded. The recorded stream must include - * that same PEND(601) entry to stay verify-clean; reconciliation - * consumes the DISPATCH(601) entry ahead of the verify walk - * (sim_loop.c's own fixed ordering: reconcile, then - * xce_core_signal_go(), then drain/compare), leaving exactly the - * PEND(601) entry to align with the live picker's own single decision - * this round. - * Round 2, 3 (begin_round=3000/4000): job 601 is still genuinely PEND - * (no DISPATCH/state-change event touches it this round) -- the real - * picker re-offers and re-PENDs it every round, the same "no-op, - * retried next round" contract test_sim_contention_parity.c's own - * filler rounds already establish; recorded decisions = [PEND(601)] - * each round. + * [DISPATCH(601)]. sim_reconcile_unaccounted_jobs() retires job 601 + * from the live picker's own pendJobs (on_dispatch()) BEFORE this + * round's picker session runs (sim_loop.c's own fixed ordering: + * reconcile, then xce_core_signal_go(), then drain/compare), so the + * live picker has nothing left to offer this round either -- no + * same-round PEND echo to record. + * Round 2, 3 (begin_round=3000/4000): job 601 is retired from the live + * picker's pendJobs (round 1's on_dispatch() above) -- nothing left for + * the real picker to offer or re-PEND. Recorded decisions = [] (empty + * ReplaySessions), the same shape round 4 below already uses for the + * identical reason. Still genuinely widens the dispatch-to-confirmation + * GAP to "several rounds" (Requirement 4.2) -- only the picker-echo + * noise in between changed, not the gap itself. * Round 4 (begin_round=5000): job 601's JOB_STATE_CHANGE to RUN is - * applied as part of this round's own pushed event batch, strictly - * BEFORE the round's picker session begins (sim_loop.c's event-then- - * reconcile-then-signal-go ordering) -- builtin_fcfs_on_job_state_change() - * removes job 601 from pendJobs the moment it leaves PEND state, so the - * live picker has nothing left to offer this round. Recorded decisions - * = [] (an empty ReplaySession, the same shape a genuinely idle round - * produces in a real recording). + * applied as part of this round's own pushed event batch. Recorded + * decisions = [] (an empty ReplaySession, the same shape a genuinely + * idle round produces in a real recording). * * Zero mismatches across all four rounds is the Observable (Requirement * 4.2). It is also consistent evidence for 2.1 (the DISPATCH decision IS @@ -1403,15 +1447,24 @@ test_sim_verify_no_false_mismatch_across_round_separated_dispatch(void) host[0] = make_test_host_alloc("hostA", res, 1); alloc = make_test_job_alloc(601ULL, host, 1); - /* Round 1: DISPATCH recorded, plus the matching PEND the live picker - * independently re-produces this same round (see header comment). */ - append_replay_session_with_dispatch_and_pend(f, 2000ULL, alloc, 601ULL, - 0); + /* Round 1: DISPATCH recorded. bug-1759/bug-1779 correction: + * sim_reconcile_unaccounted_jobs() now calls the picker's on_dispatch() + * the moment it reconstructs job 601's allocation this round, retiring + * it from the live picker's own pendJobs BEFORE that round's picker + * session runs (sim_loop.c's fixed ordering: reconcile, then + * xce_core_signal_go()) -- so there is no longer a same-round PEND + * echo to record. */ + append_replay_session_with_dispatch(f, 2000ULL, alloc); - /* Round 2, 3: job 601 still genuinely PEND -- filler rounds widening - * the dispatch-to-confirmation gap to genuinely "several rounds". */ - append_replay_session_with_pend(f, 3000ULL, 601ULL, 0); - append_replay_session_with_pend(f, 4000ULL, 601ULL, 0); + /* Round 2, 3: job 601 is now retired from the live picker's pendJobs + * (bug-1759/bug-1779 correction, same reasoning as round 4's own + * comment below) -- nothing left for the live picker to offer, so these + * rounds are recorded empty, the same shape a genuinely idle round + * produces. Still genuinely widens the dispatch-to-confirmation gap to + * "several rounds" (Requirement 4.2) -- only the picker-echo noise in + * between changed, not the gap itself. */ + append_empty_replay_session(f, 3000ULL); + append_empty_replay_session(f, 4000ULL); /* Round 4: job 601's RUN confirmation, 3 rounds after its own round-1 * DISPATCH -- applied before this round's picker session runs, so the