fix(scheduler): sim mode force-dispatched recorded decisions past live capacity (bug-1836, bug-1838)
User-reported: replaying a captured session dispatched 44 single-slot jobs onto a 22-slot host. Root cause traced through two layers: - bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision Reconciliation" backstop) force-applied any recorded DISPATCH decision it found, with zero live capacity check. Added sim_reconciled_alloc_fits_capacity(), reusing the same sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks the live picker/allocator path already uses. - bug-1838: even with that gate, reconciliation still preempted every freshly-PEND job before the live picker/allocator got a turn each round -- unnecessary now that scheduler-sim-live-allocation gave sim mode the same real, capacity-checked picker/allocator threads normal mode uses. Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a 2026-09-11 widening whose own justification no longer holds), so a freshly-pushed job now always flows through the same live, verified decision path as normal mode. Reconciliation now only ever fires for a job already RUN from before the replay's own observation window -- the one case structurally invisible to the live picker. Also logged (bug-1837, not fixed here, needs its own design decision): sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from the driver thread instead of the allocator thread plugin.h documents it belonging to -- currently inert since the only real picker plugin happens to lock internally, but not contract-safe for a future lock-free one. Fixing bug-1838 required re-examining every test exercising the old PEND-inclusive trigger; several needed their fixtures corrected to register a real worker (harmless before any capacity check existed), and others needed their actual intent re-derived and their assertions inverted or restated now that a PEND job is never eligible for reconciliation. Dated correction notes added in place to the scheduler-sim-eager-reconciliation spec, matching its own existing 2026-09-11 correction-note precedent. Verified: full scheduler suite 85/85, independently re-confirmed by running the four most affected binaries directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
cd9ecb4159
commit
94115b0024
44 changed files with 2769 additions and 333 deletions
|
|
@ -49,6 +49,7 @@
|
|||
#include "plugin.h"
|
||||
#include "sched/types.h"
|
||||
#include "xce/types.h"
|
||||
#include "xvec.h"
|
||||
#include "zarray.h"
|
||||
|
||||
/* xce_sched_core_t is also forward-declared in plugin.h (picker lifecycle
|
||||
|
|
@ -403,6 +404,26 @@ extern int sched_charge_account(xce_sched_core_t*, xce_job_account_t*);
|
|||
extern int sched_uncharge_account(xce_sched_core_t*, xce_job_account_t*);
|
||||
extern int sched_commit_job_alloc(xce_sched_core_t*, xce_sched_job_t*);
|
||||
|
||||
/*
|
||||
* sched_check_host_satisfy()/sched_check_shared_satisfy() - relocated here
|
||||
* from scheduler.h (scheduler-driver-equivalence fix, bug-1836), following
|
||||
* the exact precedent sched_account_jobs()/sched_resolve_job_picker()
|
||||
* already established above: pure declaration move, body unchanged (still
|
||||
* defined in resource.c) -- scheduler.h's own doc comment on these two
|
||||
* already documented them as "callable identically from either driver"
|
||||
* (scheduler-sim-live-allocation task 1.3), but scheduler.h itself pulls in
|
||||
* controller-facing headers (client.h, xce/cluster.h) sim_loop.c must never
|
||||
* include (Req 5 AC 17 / test_sim_isolation.c). Needed by
|
||||
* sim_reconcile_unaccounted_jobs()'s new capacity gate, which must
|
||||
* re-validate a recorded decision's resource footprint against LIVE
|
||||
* resInUse/sharedInUse the same way the live picker/allocator path already
|
||||
* does, instead of trusting the recording unconditionally.
|
||||
*/
|
||||
extern int sched_check_shared_satisfy(xce_sched_core_t*, xvec_t*,
|
||||
xvec_idval_t*, xvec_idval_t*);
|
||||
extern int sched_check_host_satisfy(xce_sched_core_t*, xce_sched_worker_t*,
|
||||
xvec_t*, xvec_idval_t*, xvec_idval_t*);
|
||||
|
||||
/*
|
||||
* sched_reconcile_host_resource_usage()/sched_reconcile_job_resource_usage()
|
||||
* - resource-accounting.md's reservation-vs-actual-usage policy for a REAL
|
||||
|
|
|
|||
|
|
@ -366,17 +366,18 @@ extern int sched_publish_dispatch(xce_scheduler_t*, xce_job_dispatch_t*);
|
|||
extern void sched_check_pending_rejoin(xce_scheduler_t*);
|
||||
extern void sched_worker_release_host_ctx(xce_sched_worker_t*);
|
||||
/* scheduler-sim-live-allocation spec, task 1.3: retyped from
|
||||
* xce_scheduler_t* to xce_sched_core_t* -- these three now operate on
|
||||
* shared core state directly, callable identically from either driver
|
||||
* (both xce_scheduler_t and xce_sched_sim_driver_t embed xce_sched_core_t
|
||||
* core as their first member). Body unchanged besides sched->core.X ->
|
||||
* core->X field-access updates. */
|
||||
* xce_scheduler_t* to xce_sched_core_t* -- these now operate on shared core
|
||||
* state directly, callable identically from either driver (both
|
||||
* xce_scheduler_t and xce_sched_sim_driver_t embed xce_sched_core_t core as
|
||||
* their first member). Body unchanged besides sched->core.X -> core->X
|
||||
* field-access updates. sched_check_shared_satisfy()/sched_check_host_
|
||||
* satisfy() relocated to core.h (scheduler-driver-equivalence fix,
|
||||
* bug-1836) -- same rationale as sched_account_jobs' own relocation there:
|
||||
* sim_loop.c needs to call them without pulling in scheduler.h's
|
||||
* controller-facing includes (client.h, xce/cluster.h), which
|
||||
* test_sim_isolation.c forbids. */
|
||||
extern int sched_meet_host_selection(xce_sched_core_t*, xce_sched_worker_t*,
|
||||
xce_select_t*);
|
||||
extern int sched_check_shared_satisfy(xce_sched_core_t*, xvec_t*, xvec_idval_t*,
|
||||
xvec_idval_t*);
|
||||
extern int sched_check_host_satisfy(xce_sched_core_t*, xce_sched_worker_t*,
|
||||
xvec_t*, xvec_idval_t*, xvec_idval_t*);
|
||||
extern int sched_collect_workers(xce_scheduler_t*, xlist_t*,
|
||||
zarray_iter_skip_fn_t,
|
||||
sched_group_worker_fn_t);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue