Task 5.1: the 26 white-box tests that include repl.c or sim_loop.c now
include the REPL trace sources (and the core trace sources where no
hooked source brings them); sched_sim_isolation_lib carries all trace
sources and the sim isolation test scans the REPL trace files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sim_process_replay_event_message() now hands the decoded replay event to
the caller, which keeps it beside the translated core event and frees it
after xce_core_push_events() with driver.c's release_translated_payload()
rules: a pushed config is kept by the core, and a rusages report is kept
by the job it found. An untranslatable event is freed at once, and events
after the last session are freed at the end. The payloads were an
"accepted leak" per event; exit-time leak on the reproduction replay
75,495 -> 25,807 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:
- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
Reconciliation" backstop) force-applied any recorded DISPATCH decision
it found, with zero live capacity check. Added
sim_reconciled_alloc_fits_capacity(), reusing the same
sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
the live picker/allocator path already uses.
- bug-1838: even with that gate, reconciliation still preempted every
freshly-PEND job before the live picker/allocator got a turn each round
-- unnecessary now that scheduler-sim-live-allocation gave sim mode the
same real, capacity-checked picker/allocator threads normal mode uses.
Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
2026-09-11 widening whose own justification no longer holds), so a
freshly-pushed job now always flows through the same live, verified
decision path as normal mode. Reconciliation now only ever fires for a
job already RUN from before the replay's own observation window --
the one case structurally invisible to the live picker.
Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.
Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.
Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now displays accounted-usage/capacity for GLOBAL/SHARED
limit instances: iterates limit_get_usage_instances(), filters by
vres->limit == matched_limit, reads capacity from core.sharedInst[idx] and
accounted usage from core.sharedInUse[idx]. Existence-of-instance and
zero-usage are kept distinct per design.md: no matching instance (or one
with no populated capacity) prints "no active usage instance yet"; a
matched-but-never-charged instance correctly shows accounted=0 instead.
Self-caught during implementation and independently confirmed by review: a
negative or stale vres->idx (possible per add_vres()'s allocate-then-assign
pattern) is not safely rejected by zarray_get_ex()'s idx>=caps check alone
-- the new code adds an explicit idx<0 guard rather than copying
repl_info_resource_one()'s existing .count-based pre-check (which is itself
imprecise for a sparsely-populated zarray, a pre-existing issue left
out of scope here).
Calling limit_get_usage_instances() for the first time from repl_info.c
reproduced the same ~24-file link-stub ripple task 2.1 established for
limit_get_configured_limits() -- fixed with the identical pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() (repl_info.c, not yet wired into repl_cmd_info() -- that's
task 4) resolves "all" vs a specific name against limit_get_configured_limits():
a NULL/empty handle on the all path reports "no limits configured" (Req 1.7);
a NULL handle or key_find() miss on the named path reports "no such limit
'<name>'" (Req 1.3) and never collapses into the all-only message. New
white-box test_repl_limit_table.c exercises both paths directly, including a
dedicated regression guard for the NULL-scoping rule.
Making repl_info.c genuinely call limit_get_configured_limits() broke the link
for ~24 other white-box test files that #include repl_info.c without also
linking plugins/limit.c -- fixed with the same link-satisfying-stub idiom this
file family already uses elsewhere (all return NULL, verified behavior-
preserving since none of these fixtures register the limit plugin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1790's fix worked but put limit-plugin-specific logic in the wrong
place: seqjob_prepare_allocation()/parjob_prepare_allocation() (the
allocator plugins) unconditionally pre-computed a job-wide total for
every per-host resource in the job's rusages, "just in case" some
global limit needed one. That's the allocator plugins' concern leaking
into limit evaluation -- duplicated across both allocators (a third
would need it too), and wasteful (computed for resources no limit
actually references).
Moved ownership to sched_fill_limit_requests() (resource.c), called
once centrally from plugin_prepare_allocation() regardless of which
allocator ran:
- Removed calc_wanted_host_resval_job_totals() and its calls from both
allocator plugins entirely.
- Added calc_wanted_job_total_resval() (resreq.c): an on-demand,
single-resource lookup by name (no resmgr/key_find needed -- the
caller already has the resolved xce_resource_t), computing the same
job-wide-total formula only for the one (scope, underlying-resource)
combination that needs it -- a GLOBAL limit on a per-host resource.
The other three combinations are unchanged, still reading straight
out of alloc_req.
- sched_fill_limit_requests() gains rusages/nhosts/nslots parameters
back (a genuine need this time, not a re-added unused one).
plugin.c's two call sites pass obj->u.primary->resreq->rusages
(NULL for COMPOUND/ALTER, matching this codebase's existing scope
convention for those tags) and ctx->nhosts/ctx->nslots.
All 23 testsuite link-satisfying stubs mechanically updated. Removed
the two allocator-level regression tests from the first pass (the
behavior they proved no longer exists by design); rewrote
test_fill_limit_requests_global_limit_on_host_resource_reads_host to
exercise the new on-demand path directly, nslots=3 to prove the job's
WHOLE total is used.
Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions.
Live re-verified against the same ug1_global_mem deploy/scenario:
byte-identical scheduler trace to the pre-refactor result, confirming
behavior-preserving.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
sched_fill_limit_requests() used to read a limit's underlying resource's
want from the same vector its own virtual-resource entry gets written to.
That's wrong whenever the two differ in scope -- e.g. a GLOBAL limit built
on top of a per-host resource like "slots" (limits.yaml's
ug1_normal_slots): the limit's own vres entry belongs in alloc_req->shared,
but "slots"'s want is only ever staged into alloc_req->host by
seqjob/parjob_prepare_allocation(), so the old code always read a zero and
silently never enforced the cap.
Now both the read source and write destination are derived independently
from each resource's own XCE_RESF_GLOBAL flag (vres->res for the source,
vres->vres for the destination) -- the same flag
calc_wanted_global_resvals()/calc_wanted_host_resvals() (resreq.c) already
dispatch on, so this is the existing rule applied consistently rather than
a new one. Also drops sched_fill_limit_requests()'s unused xce_scheduler_t*
parameter.
Full scheduler suite: 76/77 (the one failure, test_job_rusage_reconciles_
mem_reservation_against_usage, is bug-1762, pre-existing and unrelated).
Live-verified against a fresh single-node deploy.
Filed bug-1790 to track a known, deliberately out-of-scope residual gap:
a future global limit on a per-host resource other than "slots" would need
calc_wanted_host_resvals() to also run during the global phase for that
resource, which nothing currently does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Extends test_maxtasks_wiring_ingest.c with two tests: (1) a resource-
request update never changes a job's already-dispatched task count,
because core_events.c's live JOB_UPDATE handler never rebuilds
job->schedobj (Requirement 1.4); (2) two otherwise-identical jobs
differing only in requested task count don't collide in the
unsatisfiable-resource-request cache on the fresh-ingest path
(Requirement 5.4).
Adds a new file, test_maxtasks_wiring_sim_driver_parity.c, proving the
simulation driver's real static translator chain
(sim_process_replay_event_message()/replay_event_to_core_event())
applies this feature's wiring identically to the real driver -- a
genuine white-box test of the real sim_loop.c code (following
test_sim_event_loop.c's established #include technique), not a mirror
of its logic. This was split into its own file because
test_maxtasks_wiring_ingest.c already links the full scheduler_srcs
object list for its own task-3.1 test, which collides at link time
with white-box #include-ing core_events.c/sim_loop.c in one
translation unit; see tasks.md's Implementation Notes.
No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
