fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838)

User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:

- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
  Reconciliation" backstop) force-applied any recorded DISPATCH decision
  it found, with zero live capacity check. Added
  sim_reconciled_alloc_fits_capacity(), reusing the same
  sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
  the live picker/allocator path already uses.

- bug-1838: even with that gate, reconciliation still preempted every
  freshly-PEND job before the live picker/allocator got a turn each round
  -- unnecessary now that scheduler-sim-live-allocation gave sim mode the
  same real, capacity-checked picker/allocator threads normal mode uses.
  Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
  2026-09-11 widening whose own justification no longer holds), so a
  freshly-pushed job now always flows through the same live, verified
  decision path as normal mode. Reconciliation now only ever fires for a
  job already RUN from before the replay's own observation window --
  the one case structurally invisible to the live picker.

Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.

Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.

Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-21 09:50:27 +00:00
parent cd9ecb4159
commit 94115b0024
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
44 changed files with 2769 additions and 333 deletions

View file

@ -49,6 +49,7 @@
#include "plugin.h"
#include "sched/types.h"
#include "xce/types.h"
#include "xvec.h"
#include "zarray.h"
/* xce_sched_core_t is also forward-declared in plugin.h (picker lifecycle
@ -403,6 +404,26 @@ extern int sched_charge_account(xce_sched_core_t*, xce_job_account_t*);
extern int sched_uncharge_account(xce_sched_core_t*, xce_job_account_t*);
extern int sched_commit_job_alloc(xce_sched_core_t*, xce_sched_job_t*);
/*
* sched_check_host_satisfy()/sched_check_shared_satisfy() - relocated here
* from scheduler.h (scheduler-driver-equivalence fix, bug-1836), following
* the exact precedent sched_account_jobs()/sched_resolve_job_picker()
* already established above: pure declaration move, body unchanged (still
* defined in resource.c) -- scheduler.h's own doc comment on these two
* already documented them as "callable identically from either driver"
* (scheduler-sim-live-allocation task 1.3), but scheduler.h itself pulls in
* controller-facing headers (client.h, xce/cluster.h) sim_loop.c must never
* include (Req 5 AC 17 / test_sim_isolation.c). Needed by
* sim_reconcile_unaccounted_jobs()'s new capacity gate, which must
* re-validate a recorded decision's resource footprint against LIVE
* resInUse/sharedInUse the same way the live picker/allocator path already
* does, instead of trusting the recording unconditionally.
*/
extern int sched_check_shared_satisfy(xce_sched_core_t*, xvec_t*,
xvec_idval_t*, xvec_idval_t*);
extern int sched_check_host_satisfy(xce_sched_core_t*, xce_sched_worker_t*,
xvec_t*, xvec_idval_t*, xvec_idval_t*);
/*
* sched_reconcile_host_resource_usage()/sched_reconcile_job_resource_usage()
* - resource-accounting.md's reservation-vs-actual-usage policy for a REAL

View file

@ -366,17 +366,18 @@ extern int sched_publish_dispatch(xce_scheduler_t*, xce_job_dispatch_t*);
extern void sched_check_pending_rejoin(xce_scheduler_t*);
extern void sched_worker_release_host_ctx(xce_sched_worker_t*);
/* scheduler-sim-live-allocation spec, task 1.3: retyped from
* xce_scheduler_t* to xce_sched_core_t* -- these three now operate on
* shared core state directly, callable identically from either driver
* (both xce_scheduler_t and xce_sched_sim_driver_t embed xce_sched_core_t
* core as their first member). Body unchanged besides sched->core.X ->
* core->X field-access updates. */
* xce_scheduler_t* to xce_sched_core_t* -- these now operate on shared core
* state directly, callable identically from either driver (both
* xce_scheduler_t and xce_sched_sim_driver_t embed xce_sched_core_t core as
* their first member). Body unchanged besides sched->core.X -> core->X
* field-access updates. sched_check_shared_satisfy()/sched_check_host_
* satisfy() relocated to core.h (scheduler-driver-equivalence fix,
* bug-1836) -- same rationale as sched_account_jobs' own relocation there:
* sim_loop.c needs to call them without pulling in scheduler.h's
* controller-facing includes (client.h, xce/cluster.h), which
* test_sim_isolation.c forbids. */
extern int sched_meet_host_selection(xce_sched_core_t*, xce_sched_worker_t*,
xce_select_t*);
extern int sched_check_shared_satisfy(xce_sched_core_t*, xvec_t*, xvec_idval_t*,
xvec_idval_t*);
extern int sched_check_host_satisfy(xce_sched_core_t*, xce_sched_worker_t*,
xvec_t*, xvec_idval_t*, xvec_idval_t*);
extern int sched_collect_workers(xce_scheduler_t*, xlist_t*,
zarray_iter_skip_fn_t,
sched_group_worker_fn_t);