User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:
- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
Reconciliation" backstop) force-applied any recorded DISPATCH decision
it found, with zero live capacity check. Added
sim_reconciled_alloc_fits_capacity(), reusing the same
sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
the live picker/allocator path already uses.
- bug-1838: even with that gate, reconciliation still preempted every
freshly-PEND job before the live picker/allocator got a turn each round
-- unnecessary now that scheduler-sim-live-allocation gave sim mode the
same real, capacity-checked picker/allocator threads normal mode uses.
Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
2026-09-11 widening whose own justification no longer holds), so a
freshly-pushed job now always flows through the same live, verified
decision path as normal mode. Reconciliation now only ever fires for a
job already RUN from before the replay's own observation window --
the one case structurally invisible to the live picker.
Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.
Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.
Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a plugin_job_election(&(sim->core), job) call to
sim_reconcile_unaccounted_jobs()'s FOUND branch, before the existing
alloc-reconstruction/state-transition code, so a job whose outcome
comes from the recording still gets its global-limit usage registered
-- the same core-safe dispatcher the live path (task 4.1) already
uses, not a separate mechanism.
The existing job->accounts != NULL trigger guard needs no change: it
already excludes any job the live path has already decided, so this
call is naturally idempotent (match_job_global_limit_'s own
obj->configver guard makes a redundant call a safe no-op) -- narrowing
this function to a genuine timing-gap safety net rather than the
general-case decision source it used to be, per design's Live/Recorded
Authority Rule.
Host-scoped limit matching is deliberately not attempted here (needs a
resolved worker context this function doesn't independently
construct) -- that gap is tracked separately as task 4.3.
TDD: new tests prove a reconciled job's global limit is genuinely
registered via the real limit_get_usage_instances() accessor, and that
a job the live path already decided is never reprocessed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now displays accounted-usage/capacity for GLOBAL/SHARED
limit instances: iterates limit_get_usage_instances(), filters by
vres->limit == matched_limit, reads capacity from core.sharedInst[idx] and
accounted usage from core.sharedInUse[idx]. Existence-of-instance and
zero-usage are kept distinct per design.md: no matching instance (or one
with no populated capacity) prints "no active usage instance yet"; a
matched-but-never-charged instance correctly shows accounted=0 instead.
Self-caught during implementation and independently confirmed by review: a
negative or stale vres->idx (possible per add_vres()'s allocate-then-assign
pattern) is not safely rejected by zarray_get_ex()'s idx>=caps check alone
-- the new code adds an explicit idx<0 guard rather than copying
repl_info_resource_one()'s existing .count-based pre-check (which is itself
imprecise for a sparsely-populated zarray, a pre-existing issue left
out of scope here).
Calling limit_get_usage_instances() for the first time from repl_info.c
reproduced the same ~24-file link-stub ripple task 2.1 established for
limit_get_configured_limits() -- fixed with the identical pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() (repl_info.c, not yet wired into repl_cmd_info() -- that's
task 4) resolves "all" vs a specific name against limit_get_configured_limits():
a NULL/empty handle on the all path reports "no limits configured" (Req 1.7);
a NULL handle or key_find() miss on the named path reports "no such limit
'<name>'" (Req 1.3) and never collapses into the all-only message. New
white-box test_repl_limit_table.c exercises both paths directly, including a
dedicated regression guard for the NULL-scoping rule.
Making repl_info.c genuinely call limit_get_configured_limits() broke the link
for ~24 other white-box test files that #include repl_info.c without also
linking plugins/limit.c -- fixed with the same link-satisfying-stub idiom this
file family already uses elsewhere (all return NULL, verified behavior-
preserving since none of these fixtures register the limit plugin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1790's fix worked but put limit-plugin-specific logic in the wrong
place: seqjob_prepare_allocation()/parjob_prepare_allocation() (the
allocator plugins) unconditionally pre-computed a job-wide total for
every per-host resource in the job's rusages, "just in case" some
global limit needed one. That's the allocator plugins' concern leaking
into limit evaluation -- duplicated across both allocators (a third
would need it too), and wasteful (computed for resources no limit
actually references).
Moved ownership to sched_fill_limit_requests() (resource.c), called
once centrally from plugin_prepare_allocation() regardless of which
allocator ran:
- Removed calc_wanted_host_resval_job_totals() and its calls from both
allocator plugins entirely.
- Added calc_wanted_job_total_resval() (resreq.c): an on-demand,
single-resource lookup by name (no resmgr/key_find needed -- the
caller already has the resolved xce_resource_t), computing the same
job-wide-total formula only for the one (scope, underlying-resource)
combination that needs it -- a GLOBAL limit on a per-host resource.
The other three combinations are unchanged, still reading straight
out of alloc_req.
- sched_fill_limit_requests() gains rusages/nhosts/nslots parameters
back (a genuine need this time, not a re-added unused one).
plugin.c's two call sites pass obj->u.primary->resreq->rusages
(NULL for COMPOUND/ALTER, matching this codebase's existing scope
convention for those tags) and ctx->nhosts/ctx->nslots.
All 23 testsuite link-satisfying stubs mechanically updated. Removed
the two allocator-level regression tests from the first pass (the
behavior they proved no longer exists by design); rewrote
test_fill_limit_requests_global_limit_on_host_resource_reads_host to
exercise the new on-demand path directly, nslots=3 to prove the job's
WHOLE total is used.
Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions.
Live re-verified against the same ug1_global_mem deploy/scenario:
byte-identical scheduler trace to the pre-refactor result, confirming
behavior-preserving.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
sched_fill_limit_requests() used to read a limit's underlying resource's
want from the same vector its own virtual-resource entry gets written to.
That's wrong whenever the two differ in scope -- e.g. a GLOBAL limit built
on top of a per-host resource like "slots" (limits.yaml's
ug1_normal_slots): the limit's own vres entry belongs in alloc_req->shared,
but "slots"'s want is only ever staged into alloc_req->host by
seqjob/parjob_prepare_allocation(), so the old code always read a zero and
silently never enforced the cap.
Now both the read source and write destination are derived independently
from each resource's own XCE_RESF_GLOBAL flag (vres->res for the source,
vres->vres for the destination) -- the same flag
calc_wanted_global_resvals()/calc_wanted_host_resvals() (resreq.c) already
dispatch on, so this is the existing rule applied consistently rather than
a new one. Also drops sched_fill_limit_requests()'s unused xce_scheduler_t*
parameter.
Full scheduler suite: 76/77 (the one failure, test_job_rusage_reconciles_
mem_reservation_against_usage, is bug-1762, pre-existing and unrelated).
Live-verified against a fresh single-node deploy.
Filed bug-1790 to track a known, deliberately out-of-scope residual gap:
a future global limit on a per-host resource other than "slots" would need
calc_wanted_host_resvals() to also run during the global phase for that
resource, which nothing currently does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Implements repl_is_bytes_valued_resource()/repl_format_bytes_human()
(mirroring xce_test's confirmed is_bytes_valued_resource()/
format_bytes_human() precedent exactly) and the units_mode setting
(explicitly defaulted to both, not left to zero-init) wired into
set units raw|human|both alongside the existing set speed/output/verify
branches (Requirements 6.1-6.3). All 8 of task 7.1's tests pass.
Split repl_format_ts_human(), the Jobid Range Compressor (task 1.2),
and these two new helpers out of repl.c into a new repl_fmt.c: adding
them in place pushed repl.c to 929 lines (16.1% over the 800-line
cap), and unlike task 6.2's single-function split attempt (which
didn't even achieve compliance), bundling these three pure-formatter
groups together does - repl.c settles at 587 lines, repl_fmt.c at 402,
both comfortably under cap. Required a matching #include line in the
20 test files that white-box #include repl.c, plus extending
test_sim_isolation.c's forbidden-include/symbol scan to cover the new
file, following the repl_rollup.c split precedent from task 2.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements the round-boundary decision summary, collapsing of repetitive
rounds, and the shared repl_round_collapse_flush() entry point
(Requirements 2.1, 2.2, 2.4, 3.1-3.4, 3.6). Fingerprint composition
matches design.md exactly: dispatch/preempt counts from the existing
per-round tally, pend-jobid set state-derived from core->jobs (not
decision events), so Requirement 2.4 holds independent of bug-1611.
All 7 of task 2.1's RED-phase tests now pass.
Went through three review rounds:
- Round 1 approved the logic but rejected repl.c's size (1160 lines,
45% over the 800-line cap). Remediated by splitting the Round Rollup
functions into a new repl_rollup.c, following this codebase's own
established repl.c/repl_break.c/repl_info.c/repl_params.c domain-split
precedent.
- Round 2 approved the split but found the new file wasn't added to the
"simulation isolation" regression guard (sched_sim_isolation_lib +
test_sim_isolation.c's forbidden-include/symbol scans), plus two stale
repl.c-location doc comments left over from the move. Both fixed.
- Round 3 found repl.c still 17 lines over cap post-split (808 after
trimming, from 817) and rejected on that basis. Verified empirically
that closing the remaining gap isn't achievable via comment trimming
alone (even deleting both comments entirely only reaches 801) without
another disproportionate file split for single-digit lines. Accepted
808 lines (1% over) as the final state, consistent with round 1's own
explicit judgment that a small residual is not a blocker once the
original 45%-over violation is resolved - documented in design.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
xce_sched_core_load_snapshot() left every restored worker's resInUse
(and core->sharedInUse) at zero -- zarray_init() in the workers-
restoration block clears them, and nothing re-summed against the
restored job allocations. A caller inspecting worker/resource state
between a snapshot load and the first scheduling session (e.g. the
REPL's `info worker` before the first `step`) saw a misleadingly empty
"used" total for jobs already known to be RUNNING. Added one
sched_account_jobs(core) call after workers/jobs are restored -- a
correct idempotent-from-zero re-derivation using the same
sched_charge_account() chain the next real session would use anyway.
Must not be called again without an intervening reset (accumulates,
does not assign) -- verified no such double-call exists in this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
sim_loop.c never called plugin_begin_session()/plugin_end_session() at
its round boundary, unlike driver.c's normal-mode round -- a driver-
equivalence gap found by an /xce-audit-invariants scheduler pass.
Currently inert (sim mode's only reachable picker, builtin_fcfs, leaves
both hooks NULL) but structurally identical to bug-1613's class of bug.
Relocated plugin_begin_session()/plugin_end_session() from an implicit
xce_scheduler_t* to xce_sched_core_t* directly (mirroring
sched_account_jobs()'s prior relocation) -- their bodies never touched
anything beyond core->plugins, so driver.c's comment claiming they
"needed the full xce_scheduler_t*" was itself stale. sim_loop.c now
calls both, same relative position as driver.c. A full-tree grep (done
after the first build attempt failed on an untouched file) turned up a
third live call site in scheduler.c's legacy sched_begin_session()/
sched_end_session(), plus four testsuite files with their own stub
definitions of the same functions -- all updated to match the new
signature.
Extended test_sim_session_ordering.c's existing bug-1613 regression
tests with new plugin_begin_session()/plugin_end_session() spies
(call-count + session_active-window proof, same technique as the
existing sched_account_jobs spy). RED/GREEN independently verified:
temporarily disabled the two new call sites, confirmed both new
assertions fail (Expected 2 Was 0 / Expected 1 Was 0), restored,
reconfirmed green. Full scheduler suite: 51/52 green, the one failure
is the pre-existing unrelated bug-1560 sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Replaces sim_loop.c's monolithic xce_core_go() with the same
xce_core_begin_session() -> sched_account_jobs() -> xce_core_signal_go()
sequence driver.c already uses for normal-mode rounds, opening the same
interleaving window and calling the now-core-level accounting service
(task 1.1) at the same relative position. Error-handling for a failed
session start is preserved byte-for-byte (same cleanup, same rc source).
Both interactive and batch simulation runs go through this one call site.
This is bug-1613's wiring half: sim_loop.c now calls sched_account_jobs()
at all (it never did before). resInUse is still zero end-to-end after this
task alone, because job->accounts is never populated by any live path in
sim mode yet -- that's Recorded-Decision Reconciliation (tasks 2.2-2.4),
not yet implemented; a comment marks where it slots in.
Adds a call-count/session-window regression test proving the wiring itself
(RED-phase reproduced live via git stash + rebuild, matching bug-1613's
symptom exactly). 8 sibling white-box test files needed a link-satisfying
sched_account_jobs() stub, discovered via actual linker errors.
45/46 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
