snapshot_load.c carried its own field-for-field copy of core_events.c's
schedobj builders (primary/compound/alter + hlimits killer), kept separate
by a "never share builders" convention. The copy drifted from the original
three times, each a field-completeness bug in restored jobs (bug-1772's
all-zero resreq_fp among them). The convention's reasons apply to the
job/queue/worker builders, not to schedobj, which is never wire-captured.
- core_build_schedobj() becomes the exported xce_core_build_schedobj(obj,
user, queue, project, resreq) (core.h), taking identity strings instead
of a wire job; core_build_job() and snapshot_load.c both call it.
- snapshot_load.c's five copied functions are removed.
- test_sim_startup.c #includes snapshot_load.c without core_events.c; it
now also #includes core_events.c, plus the same
sched_reconcile_job_resource_usage link stub test_driver_translate.c
uses.
- test_fresh_ingest_and_snapshot_restore_derive_identical_... now compares
the whole schedobj (tag, user/queue/project, configver, glimits, owner,
resreq binding, resreq_fp), guarding against future drift. No RED phase:
there is no live gap left to catch; this removes the duplication that
kept producing them.
Scheduler suite 87/87; full build links.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JOB_UPDATE (core_events.c) and the legacy sched_update_existing_job()
(schedobj.c) replaced job->resreq, but the allocator reads job->schedobj,
whose primary->resreq still pointed into the old parsed request -- so an
update never changed scheduling, for any field. The old primary was also
leaked (FREEUP only freed the container).
- core_bind_primary_resreq(): the resreq pointer + fingerprint computation
factored out of core_build_primary_schedobj(), reused on update.
- core_rebind_schedobj_resreq(): primary->primary re-points in place and
recomputes the fingerprint, keeping the schedobj's runtime state; a kind
change rebuilds the kind-specific part (the old part is left alone, as
nothing frees these today -- never a dangling pointer).
- JOB_UPDATE now applies the job's ingest-time task count to the new
resreq, as core_build_job() does, so resreq-maxtasks-wiring Req 1.4 (an
update never changes the dispatched task count) still holds -- it used to
hold only because the update never reached the schedobj at all.
- schedobj.c's legacy path gets the same re-binding.
test_resource_request_update_never_changes_dispatched_task_count asserted
the old behaviour (schedobj keeps the ingest-time object); it now asserts
the schedobj uses the updated resreq and fingerprint while the task count
stays 4/7 (RED before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every scheduler (re)join re-pushes the whole config, and both apply paths
-- sched_init_resources() (scheduler.c) and core_apply_config_resources()
(core_events.c) -- re-added each resource with makeResource(), which
refuses an existing name: one ERROR per resource per rejoin (33 in
data/xce_scheduler.log), and a genuinely changed definition was dropped
just as silently.
New libxce resourceApplyDef(): under one write lock, register a new
resource (ADDED), or compare an already-registered one on its
scheduling-relevant fields -- valtype, ordtype, flags, interval -- and
return SAME or CHANGED. Both call sites log SAME at debug, CHANGED as a
WARN that the definition is not applied to the running scheduler
(updating a registered resource in place would disturb its index and
accounting), and only real failures as ERROR.
Test test_resource_reapply_same_is_noop_changed_is_reported (EEXIST on
an identical re-push before). Scheduler suite 87/87, lib 1/1. Re-verify
of data/scheduler-replay-20260925-133708.replay: 0 add-resource ERRORs
(33 before), 0 changed-definition WARNs, 0 mismatches, same 60 DISPATCH /
908 PEND decisions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a later primary of a COMPOUND failed, sched_make_compound_alloc_
still moved the earlier primaries' merged allocation onto the schedobj,
so a job decided PEND carried a partial allocation that the next
successful round overwrote and leaked. The merged allocation is now
published only when every primary succeeds and freed otherwise. No
resource-state rollback is needed: the primaries' uncommitted debits live
in ctx.uncomitted, per-attempt scratch freed by sched_ctx_fini(), and
schedobj->alloc's only reader is the dispatch path. The ALTER loop's
per-alternative cleanup (added with bug-1769) is now redundant and removed.
Test: "1*{..} + 5*{..}" on 2x2 slots PENDs with schedobj->alloc NULL
(held the first primary's allocation before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each primary's allocator (seqjob/parjob my_alloc) sets ctx->alloc to a
fresh job allocation covering only its own hosts, so a multi-primary
COMPOUND ("1*{A} + 2*{B}") dispatched only the last primary's hosts --
leaking the earlier allocations while every primary's hosts had been
debited in ctx.uncomitted.
sched_make_compound_alloc_ now folds each primary's allocation into one
job allocation via the new libxce merge_job_alloc(): a host placed by
several primaries becomes one entry with per-resource values added
(matched on resname, instid and reserve, so distinct instances are never
summed); new hosts and shared allocations are appended/merged the same
way. Partial-failure rollback remains bug-1766.
Tests: "1*{..} + 2*{..}" on 2x2 slots now allocates 3 slots across two
distinct hosts; "1*{..} + 1*{..}" yields one host entry with 2 slots
(both kept only the last primary before). Scheduler suite 87/87, lib 1/1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seqjob_prepare_allocation() and parjob_prepare_allocation() returned 0
whenever ctx->schedobj was not PRIMARY, so a primary nested in a COMPOUND
(or ALTER) job never had its slot count or rusage checked: a 5-task
compound dispatched onto 4 slots, and an ALTER job was "placed" by its
first alternative with an empty (0-slot) allocation. Both hooks now
compute the want from ctx->primary -- the primary being allocated, set by
sched_ctx_set_primary() for nested primaries and identical to
obj->u.primary->resreq for PRIMARY jobs.
sched_ctx_set_primary() also clears ctx->worker, so the next primary of a
compound computes its job-level want rather than a per-host want left
over from the previous primary's host (PRIMARY jobs set it once, with
worker already NULL).
Tests (real ingest + live allocator): a 5-task COMPOUND on 2x2 slots now
PENDs; "{5*{..}} || {1*{..}}" falls back to the second alternative (1
slot). Both failed before. Scheduler suite 87/87.
Follow-up logged: bug-1863 (multi-primary COMPOUND keeps only the last
primary's allocation).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1769: every builder stores xce_compound_schedobj_t* (24 bytes) in
obj->u.alters, but sched_make_alter_allocation() read them as
xce_sched_obj_t* (176 bytes) and ran the compound allocation on them -- a
heap-buffer-overflow read of ->u.compound and write of ->alloc for any real
`{..} || {..}` resreq. The compound allocation now takes its resreq
explicitly (sched_make_compound_alloc_), and each ALTER alternative is
tried on behalf of the ALTER obj itself; a failed alternative's partial
allocation is dropped before the next (rollback itself is bug-1766).
sched_ctx_init() rejected ALTER objs ("bad sched obj type"), so ALTER jobs
could never allocate at all -- it now accepts them like COMPOUND (all
downstream checks only distinguish PRIMARY).
bug-1771: plugin_commit_decision() fans every DISPATCH out to the legacy
"xce_sched_fcfs" plugin even when it never tracked the job; fcfs_del_job()
(and fcfs's job-add hook) dereferenced job->queue->name with no NULL
check, SEGV'ing the allocator for a job whose queue was never registered.
Tests (test_parjob_e2e_multislot.c, real ingest + live allocator thread):
test_alter_resreq_job_dispatches_via_real_allocator (ASAN
heap-buffer-overflow before, DISPATCH after) and
test_dispatch_of_job_with_unregistered_queue_does_not_crash (SEGV at
fcfs.c:318 before). Scheduler suite 87/87.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
q->resreq holds an interned makeString() value (pointer past the xstring
object header). core_upsert_queue() and sched_update_existing_queue()
released the old value with FREEUP()/free() when a queue update carried a
new resreq, an ASAN bad-free. Seen as HA soak replay-parity errors=1: the
sim replay of a snapshot-restored queue followed by a config push aborted
in core_upsert_queue.
Both sites now dereference() the old value (NULL-guarded). Regression
tests cover each site; the existing queue-update test had skipped the
resreq branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recorder encoded each controller event batch after
xce_core_push_events() had applied it, and applying moves payload fields
out of the events: a WORKERS_PUSH's resvals (core_apply_worker_resvals
MOVE_PTR), a JOBS_PUSH's allocs, a JOB_RUSAGE's rusages, and a
CONFIG_PUSH's shared resvals (sched_apply_pending_config recorded after
sched_init_config). Recordings silently lost exactly those fields -- a
worker's resval names were written with NULL values, so --sim-verify
replayed "no slots" and turned 3 live DISPATCHes into PENDs.
driver.c now records the batch before the push; the periodic in-stream
SnapshotEntry moves out of sched_record_write_events() into new
sched_record_checkpoint_after_apply(), called after the push so a
checkpoint still reflects the events written ahead of it. The config
push is likewise recorded before sched_init_config().
Regression test: test_recorded_workers_push_keeps_resval_values (RED:
recorded value NULL; GREEN after). Periodic-checkpoint tests updated to
the write-then-checkpoint contract. meson test 109/109; 25 live soak
recordings replay with 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-ups from the bug-1846 investigation.
bug-1847: config pushes never reached a .replay recording -- handler.c
applied them synchronously, bypassing the pending-update queue the
recorder writes from. With bug-1846's deferral they're now applied at a
round boundary on the driver thread, so sched_apply_pending_config()
writes a CONFIG_PUSH event via the existing sched_record_write_events()
(no-op when not recording), ahead of the round's other recorded events.
Test: test_config_push_deferred.c's test_applied_config_push_is_recorded
(RED: 0 events; GREEN: 1). Fidelity limit, pre-existing: replay applies
configPush through core_events.c's CONFIG_PUSH path, which doesn't re-run
groupmgr_load_configs()/plugin_config_update(), so a limits/groups change
still won't take effect in sim.
bug-1848: xce_sched_obj_t.hlimits was created with a NULL zarray killer
at all three constructors, so reset_sched_obj() and schedobj teardown
leaked the per-worker xlist_t match_job_host_limit_() callocs. New
kill_hlimits_list() (schedobj.c) frees the list, never its vres entries
(owned by the limit plugin); core_events.c and snapshot_load.c use
identical static local copies, since both avoid sharing helpers with
schedobj.c and several test binaries link them without it. Test:
test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED:
killer NULL; GREEN).
bug-1849: every --sim run warned "failed to resolve job picker plugin ...
picker rebuild will be skipped". Benign: the snapshot is loaded before
the sim registers its picker, and sim_bootstrap_picker() replays the same
queue/job walk afterward. Downgraded to TRACE with an explanation.
Buglog also records bug-1850 (open): the ~100s reconnect churn behind
bug-1846 part 2 -- xcecli_retry_next() rotates/reconnects on every retry
(ETIMEDOUT after the 10s reply deadline, E_XCE_NOT_READY, redirects), so
a slow reply from a healthy primary forces a rejoin + full re-push.
Needs the live scheduler.log and a libxce spec.
Full scheduler suite 87/87 (asan). Replay of the captured file: same 184
mismatches, all attributable to the recording's pre-fix binary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.
Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.
Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.
Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
-- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
returned NULL right after the push), GREEN (session walks the remaining
jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
#include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).
Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
match_job_host_limit_() (plugins/limit.c) runs on every per-host
allocation attempt via limit_prepare_allocation(), and unconditionally
xlist_store()d the matched vres into obj->hlimits[worker->idx]. That list
is only rebuilt by limit_job_election() on a configver bump, so a job that
kept pending on the same host gained one more copy per attempt.
sched_fill_limit_requests() then staged every copy into
ctx->alloc_req.host, and seqjob's make_host_alloc() built a host alloc
carrying all of them.
Found via temporary per-decision --sim-verify diagnostics (reverted, not
committed) against scheduler-replay-20260923-085356.replay: pc13v host
allocs held mem + slots + ~70 identical limit entries (72) instead of 3.
After the fix the replay side is always 3 entries, and the LIVE recording
itself still shows the inflation (72/81/96/114/138/177 entries, resetting
only on a config push) -- so this is a production bug, not sim-only.
make_job_account() creates one res_account per entry and
sched_update_res_accounts() charges each, so the host limit was charged
(and later released) N-fold.
Fix: hlimits_contains_() pointer-identity check (vres is interned per name
in privdata->instances); a matched vres is stored once per (job, worker).
Test: test_limit_host_dedup.c (sched_limit_host_dedup_test) -- five
plugin_prepare_allocation() attempts for one job on one worker must leave
hlimits[worker]->count == 1 and one staged copy of the vres in
alloc_req.host. RED before the fix (Expected 1 Was 2), GREEN after; full
scheduler suite 86/86 (asan).
Still open (bug-1846 part 2, logged in buglog): 9 live-recorded rounds end
after 11-12 decisions while replay processes the whole candidate list; not
root-caused, needs the live scheduler.log or a fresh capture.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/xce-audit-invariants scheduler dimension-1/4 audit, prompted by a fresh
user-captured .replay file that still showed massive --sim-verify mismatch
volume after bug-1843's fix. Root-caused via temporary diagnostic
instrumentation against the real capture (reverted, never committed) rather
than trusting the surface symptom.
bug-1844: sim_verify_alloc_eq() (sim_verify.c) compared `instid`, a bare
per-process monotonic counter (scheduler.c's sched_get_id(), core->nextid)
stamped at allocation-build time -- not semantic content. The original
recording's counter and a fresh replay's counter have no reason to reach
the same value for "the same" dispatch. Live-verified: every DISPATCH
mismatch showed identical value/duration/resname/reserve, differing only
in instid (e.g. 826 vs 1). Dropped from the comparison, same treatment as
bug-1842's `from` exclusion.
bug-1845: sim_verify_job_preason_eq() required PEND decisions' full
host/resource reason-detail lists to match. core.h documents "simulation
isolation" as deliberate design: sim/replay never registers
core->unsatisfiable_lookup_cb, so it can never hit the unsatisfiable-cache
main.c's normal-mode branch registers. A live cache hit
(sched_pend_from_cached_reason_()) records only main_reason with zero
detail; replay always falls through to the full per-host computation and
always produces detail. Same semantic outcome (PEND, same main_reason) at
two architecturally-legitimate detail levels. Relaxed the comparator to
main_reason only. Separately verified the cache itself is sound (key is
safe within a round since capacity only decreases and the cache clears
every round) -- ruling it out as the cause of a second, still-open
divergence (bug-1846: one job genuinely dispatches one round early in
replay vs. the original recording -- a suspected resource-accounting
parity gap, not yet root-caused, logged for follow-up).
bug-1772 (previously deferred, incorrectly tagged "fixed"): re-found
independently by a dimension-4 snapshot-field-completeness audit subagent.
snapshot_build_primary_schedobj() (snapshot_load.c) never computed
resreq_fp, unlike its declared model core_build_primary_schedobj()
(core_events.c) -- every snapshot-restored job got a zeroed fingerprint,
which alloc_cache.c's cache key folds in directly alongside
user/queue/project, so two distinct restored jobs sharing those three
fields could silently share an incorrect unsatisfiable-cache verdict.
Added the same 4-field computation the live path uses.
TDD throughout: each fix has a dedicated regression test, and each was
verified via genuine red (revert fix, confirm the new test fails) then
green (restore, full 85/85 scheduler suite). Also ran three parallel
dimension audits (core I/O isolation, picker/allocator jbuf rendezvous) --
both clean, no violations found.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sched_record_session_add_decision() (recording_session.c) grows its
per-round decisions array via realloc(), doubling from 8. On a failed
growth it silently dropped the current decision and every one after it
that round, without marking the round incomplete -- finalize() then
wrote the short array as an ordinary, apparently-complete ReplaySession.
--sim-verify's replay comparison had no way to distinguish that from a
genuinely smaller live round, producing spurious "expected: <none>"
mismatches for jobs that were, in reality, correctly decided live and
simply never recorded. Live scheduling itself was unaffected either
way -- only the .replay file's own record of that one round lost its
tail.
This refutes the leading theory from the prior investigation (replay
seeing a job's first appearance "one round early" via coarser event
batching in sim_loop.c) -- confirmed via targeted diagnostic
instrumentation against the original captured .replay file: replay
deterministically reproduces the full candidate set every round;
the recording itself has the one-round gap.
Fix: track truncation on the session ctx; finalize() now refuses to
write an incomplete session and fails loudly through the same
sched_record_abort_on_error() path every other recording failure in
this subsystem already uses.
TDD: added a regression test forcing a real, deterministic realloc()
failure via -Wl,--wrap=realloc (scoped to this one test binary --
no prior malloc/realloc fault-injection harness existed in this
codebase). Verified genuine red -> green: the test fails without the
fix, passes with it. Full 85/85 scheduler suite green.
Note: this prevents future recordings from losing decisions this way;
it cannot retroactively repair an already-captured .replay file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_verify_alloc_eq() compared xce_alloc_t.from, which make_alloc()
(xcealloc.c) stamps with the real OS wall clock (time(NULL)), not
core.c's virtualized sim clock. The recorded (expected) value is
whatever real second the original run happened to execute in; the
freshly-rebuilt (actual) value is whatever real second the replay
executes in -- two unrelated real-world timestamps, so every
otherwise-correct DISPATCH decision failed verification whenever a
replay ran even one wall-clock second away from its own recording.
Drop `from` from the comparison -- it's reservation bookkeeping, not
part of a decision's semantic identity. Verified against a real
captured .replay file: 120/120 false DISPATCH mismatches -> 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sim_reconcile_unaccounted_jobs() (sim_loop.c) was calling
picker->picker_iface->on_dispatch() directly from the sim driver thread,
violating plugin.h's documented "allocator thread only" contract for
Post-allocation hooks. Dormant in practice only because builtin_fcfs.c
happens to lock internally around every hook -- a defensive measure the
contract text doesn't actually require of picker plugins.
Since scheduler-sim-live-allocation, sim mode spawns the same real
sched_job_picker()/sched_allocator() thread pair normal mode uses, so a
real allocator thread now exists to route through. Added a
reconcile_req/reconcile_ack cbuf pair on xce_sched_core_t and a reserved
XCE_SCHED_RECONCILE_WAKE sentinel pushed through jbuf to wake the
allocator thread without touching jbuf's pinned capacity-1/item-type
invariant (Req 5 AC 7). sched_allocator() services the request and calls
on_dispatch() itself; sim_reconcile_invoke_on_dispatch() blocks on the ack.
Falls back to a direct call when no allocator thread is live for a given
core (standalone unit tests), which is safe by construction.
TDD: new test spawns a stand-in allocator-relay thread and asserts
on_dispatch() fires there, not on the caller's thread -- confirmed RED
against the old direct-call code, GREEN after. Four other test files with
hand-rolled sched_allocator() stubs needed the same relay branch (found by
an actual hang at meson's timeout). Full scheduler suite: 85/85, zero
regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User-reported: replaying a captured session dispatched 44 single-slot jobs
onto a 22-slot host. Root cause traced through two layers:
- bug-1836: sim_reconcile_unaccounted_jobs() (the "Recorded-Decision
Reconciliation" backstop) force-applied any recorded DISPATCH decision
it found, with zero live capacity check. Added
sim_reconciled_alloc_fits_capacity(), reusing the same
sched_check_host_satisfy()/sched_check_shared_satisfy() admission checks
the live picker/allocator path already uses.
- bug-1838: even with that gate, reconciliation still preempted every
freshly-PEND job before the live picker/allocator got a turn each round
-- unnecessary now that scheduler-sim-live-allocation gave sim mode the
same real, capacity-checked picker/allocator threads normal mode uses.
Narrowed reconciliation's trigger back to JOB_IS_RUN-only (undoing a
2026-09-11 widening whose own justification no longer holds), so a
freshly-pushed job now always flows through the same live, verified
decision path as normal mode. Reconciliation now only ever fires for a
job already RUN from before the replay's own observation window --
the one case structurally invisible to the live picker.
Also logged (bug-1837, not fixed here, needs its own design decision):
sim_reconcile_unaccounted_jobs() calls a picker's on_dispatch() hook from
the driver thread instead of the allocator thread plugin.h documents it
belonging to -- currently inert since the only real picker plugin happens
to lock internally, but not contract-safe for a future lock-free one.
Fixing bug-1838 required re-examining every test exercising the old
PEND-inclusive trigger; several needed their fixtures corrected to
register a real worker (harmless before any capacity check existed), and
others needed their actual intent re-derived and their assertions
inverted or restated now that a PEND job is never eligible for
reconciliation. Dated correction notes added in place to the
scheduler-sim-eager-reconciliation spec, matching its own existing
2026-09-11 correction-note precedent.
Verified: full scheduler suite 85/85, independently re-confirmed by
running the four most affected binaries directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fmt_limit()/fmt_limit_units() formatted a percent-based limit's value
as a bare snprintf(..., "%f%%", limit->val.val) -- echoing the raw
stored value with a literal "%" appended. But xce_limit_val_t.val is
genuinely a 0-1 fraction when percent is set (confirmed end-to-end:
no /100 or *100 scaling exists anywhere in the config->capnp->runtime
pipeline; confirmed authoritative by the original hand-written C
literal this codebase's oldest percent-based limit was migrated from,
.val = 0.2 for a documented "20% mem cap"). A correctly-configured 20%
limit therefore displayed as "0.200000%" instead of "20.000000%" --
misleading, though technically an honest echo of the stored value.
Fixed by multiplying by 100.0 before formatting. Display-only --
update_limit_res_value()'s actual capacity computation was already
correct and is untouched. fmt_limit() (a thin XCE_UNITS_RAW wrapper)
inherits this fix for its own percent-branch output too, a deliberate,
intentional change from its pre-fix behavior for percent-based limits
specifically (its 5 pre-existing direct tests only assert rc > 0,
never the exact percent string, so this is safe).
Updated test_limit.c's percent-mode test fixture from an unrealistic
.val=80.0 to a realistic .val=0.8 (assertion text unchanged, since
0.8*100=80 either way) and test_repl_limit_table.c's own percent test
to expect its shared fixture's arbitrary .val=50.0 constant correctly
rendering as "5000.000000%". RED confirmed via the raw Unity testlog's
:FAIL: lines (the aggregate meson summary masks single-test failures,
a known harness quirk in this project); GREEN confirmed after, both
lib and scheduler suites, 0 Unity FAILs.
Discovered during live manual verification of scheduler-sim-diagnostics
Requirement 6 against a real deployed cluster.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Final validation task for the Human-Readable Units increment (tasks
7-10.1). Adds three tests closing gaps no prior task's own tests
covered:
- Cross-surface consistency (Requirement 6.5): a single info limit
request against a byte-valued limit with both a configured setting
and a live usage instance shows the SAME units convention in both
lines, across mode switches -- neither task 8.2's nor task 9's own
tests combined both surfaces in one capture.
- Non-byte-valued exclusion (6.3): a new "slots"-resourced fixture
(the first non-"mem" fixture in this file) proves the entire
combined display is byte-for-byte identical across all three units
modes.
- Percentage exclusion (6.3) at the REPL-dispatch level: a percent
limit's own percentage value survives real repl_dispatch()/
set-units switching unchanged, complementing test_limit.c's
existing library-level proof of the same invariant.
Full regression sweep: lib suite 368/368, scheduler suite 85/85,
including test_repl_limit_table.c (35/35), test_repl_resource_table.c,
test_repl_resource_formatting.c, and test_limit_plugin.c all
unaffected. No production code changed. This closes out Requirement 6
(ACs 6.1-6.5) and reconfirms Requirement 5 (ACs 5.1-5.3) end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Six pre-existing assertions in test_repl_limit_table.c (across 4 test
functions) used bare strstr() substring checks against whole-number
fixture values on "mem" -- e.g. strstr(captured, "accounted=4.00").
Task 9's fix changed the actual output to "accounted=4.000000 (4.00
B)", but the old substring is a literal prefix of the new one, so
these assertions kept passing by coincidence of formatting, providing
zero actual regression protection for the fix.
Strengthened each to assert the full corrected "<raw> (<human>)"
string. Proved via TDD: reverting task 9's fix now makes all 6
strengthened assertions (plus task 9's own 2 genuine tests) fail;
restoring the fix returns the suite to 85/85 green.
Confirmed via full-file scan that no other loose assertion of this
shape remains, and that info worker/info resource's own regression
suites were never vulnerable to this class of bug (they build expected
strings dynamically rather than hardcoding literal substrings).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_print_limit_usage()/repl_print_limit_usage_per_host() (repl_info.c)
passed vres->vres->name (a synthesized compound scope-instance name, e.g.
"limits2/u=rshen/q=xce/p=") to the byte-value classifier instead of the
limit's actual constrained resource name -- so a byte-valued limit's
usage amounts never got recognized as byte-valued, regardless of the
underlying formatting machinery already working correctly.
Fixed both call sites to pass the limit's own resname
(limit->val.resname / vres->limit->val.resname) instead. Added tests
proving human-readable rendering for both the cluster-wide/GLOBAL path
and a percentage-based per-host limit (Requirement 6 AC 6.4: a
percentage-of-byte-resource limit's resolved absolute amount is
human-formatted too).
Discovered and confirmed (implementer + independent reviewer): the
fix landed with zero pre-existing test failures, not the handful
design.md predicted -- those assertions used whole-number fixture
values with bare strstr() substring checks, which still match the
new, longer human-augmented output by coincidence of formatting.
Documented in tasks.md; task 10.1 will strengthen those assertions
to genuinely exercise the fix rather than pass by accident.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Switches both fmt_limit() call sites in repl_info_limit() (the "all"
path and the named-lookup path) to
fmt_limit_units(..., translate_units_mode(repl_current_units_mode(sim))),
so a byte-valued limit's configured amount now renders human-readable
under the REPL's real, currently-selected units mode -- the same
convention info worker/info resource already honor.
Adds two tests proving human-readable rendering under the default
mode and consistent switching across set units raw|human|both via
real repl_dispatch(). RED confirmed by reverting the call sites
(2 failures, 28 pre-existing tests in the same file unaffected).
Corrects an inaccurate self-reported regression-safety rationale
(there is no slots-resource fixture in this file; safety instead
comes from those tests never asserting on the formatted value line),
logged in tasks.md's Implementation Notes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
core_clock.c joined the mechanically-enforced isolation archive
(sched_core_isolation_lib) and its own doc comment claimed coverage by
the forbidden-headers/forbidden-symbols text scans, but was actually
absent from both -- only the nm(1) undefined-symbol layer covered it,
which cannot catch a forbidden #include resolving to no undefined
symbol (a macro-only header).
Added XCE_CORE_CLOCK_C_PATH (meson.build, mirroring the existing
per-file path macros) and extended both
test_core_never_includes_forbidden_headers() and
test_core_never_calls_forbidden_symbols() to scan core_clock.c. It
remains excluded only from the wall-clock scan, for its one sanctioned
clock_gettime() call.
TDD: confirmed RED by temporarily inserting a forbidden #include into
core_clock.c (84/85, correctly attributed failure), reverted
(byte-identical to HEAD), reconfirmed GREEN (85/85). No live violation
existed -- this closes a coverage gap in the mechanism itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
/xce-audit-invariants scheduler (dimension 1, driver state-equivalence,
re-run after scheduler-sim-live-allocation) found plugin_end_session()
still checked `plugin->begin_session_fn != NULL` before invoking
`plugin->end_session_fn(plugin)` -- a copy-paste of
plugin_begin_session()'s own (correct) guard. Two failure modes: a
plugin registering only end_session_fn silently never has it invoked;
a plugin registering only begin_session_fn crashes via a NULL
function-pointer call. Currently inert in production (no live plugin
hits either shape) but a real latent defect in exactly the
session-boundary code the driver-equivalence invariant governs.
This exact defect had been independently filed twice: bug-1492
(2026-08-04, scheduler-core task 5.2's review) and bug-1682
(2026-08-24, found adjacent to bug-1681's fix) -- neither investigation
cross-referenced the other. Both entries also carried a "fixed" tag
inconsistent with their own "not yet fixed" narrative text, caught by
this audit's independent code+text cross-check.
Fixed the one-line guard; added test_end_session_invokes_end_session_fn_
when_only_it_is_set() (test_plugin_hook_dispatch.c) proving the silent
non-invocation, confirmed RED (84/85) against the buggy guard and GREEN
(85/85) after the fix. Updated both buglog entries to FIXED with
cross-references, and corrected bug-1611's stale narrative to reflect
its narrowing (not elimination) by scheduler-sim-live-allocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Re-ran test_repl_equivalence.c, test_sim_equivalence.c, and
test_sim_contention_parity.c in full: all pass, 85/85, zero assertion
changes anywhere. Confirmed all three still exercise their own
link-satisfying always-PEND sched_job_picker/sched_allocator stub
(added task 4.1), correctly proving the new two-thread jbuf timing
didn't disturb decision ordering -- but not proving real live-path
resource-based dispatch parity, which is already covered separately
by tasks 4.1/4.3/5.1/5.2's real-scheduler_srcs-linked fixtures.
Fixed a stale documentation defect: all three files' header/inline
comments claimed sim mode "structurally CANNOT emit a DISPATCH
decision" via sim_picker_thread_fn(), a function this feature removed
in tasks 3.1/4.1. That claim is now false of production sim mode
(disproved by test_sim_live_allocation.c and others); corrected with
a RECONCILIATION addendum rather than rewriting the original
historical rationale. No production code or assertion values changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds the feature's concrete original acceptance test: a genuine
capnp-encoded .replay (real ReplayHeader/SnapshotEntry/Conf4Sched with
one global and one host-scoped limit), driven end-to-end through the
real, unmodified sched_sim_driver_run(). The dispatched job's usage
registers against both limits via limit_get_configured_limits()/
limit_get_usage_instances() -- the same accessors info limit reads --
proving real, non-zero usage where today's REPL previously showed none.
RED-phase evidence: reverting task 4.3's 4 files to their pre-fix
commit reproduces the exact predicted failure (host-scoped usage
missing, global usage intact); restoring to HEAD reproduces GREEN.
Closes bug-1826 (the bug this whole feature was built to fix) --
buglog entry updated from NOT FIXED to FULLY FIXED, citing this test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds two new fixtures spawning the real sched_job_picker/sched_allocator
thread pair directly against bare normal-mode and sim-mode core state
under genuine resource contention (2.0 capacity, 3 jobs x 1.0 demand),
asserting identical admission decisions and identical accounting state
(job->accounts, w->resInUse), with and without a matching host-scoped
limit. The pre-existing test_sim_contention_parity.c could not be reused
since its white-box stubs hard-code the old always-PEND behavior and
never exercise live election/allocation.
Confirms build_limit_vres_requests()/get_vres_value() (plugins/limit.c)
are #if 0'd dead code with zero live callers -- no limit in this codebase
currently gates/denies admission in either mode, only tracks/displays
usage -- so this proves usage-tracking parity, documented explicitly as
an honest scope boundary rather than a silent narrowing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.
Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.
One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.
TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds a plugin_job_election(&(sim->core), job) call to
sim_reconcile_unaccounted_jobs()'s FOUND branch, before the existing
alloc-reconstruction/state-transition code, so a job whose outcome
comes from the recording still gets its global-limit usage registered
-- the same core-safe dispatcher the live path (task 4.1) already
uses, not a separate mechanism.
The existing job->accounts != NULL trigger guard needs no change: it
already excludes any job the live path has already decided, so this
call is naturally idempotent (match_job_global_limit_'s own
obj->configver guard makes a redundant call a safe no-op) -- narrowing
this function to a genuine timing-gap safety net rather than the
general-case decision source it used to be, per design's Live/Recorded
Authority Rule.
Host-scoped limit matching is deliberately not attempted here (needs a
resolved worker context this function doesn't independently
construct) -- that gap is tracked separately as task 4.3.
TDD: new tests prove a reconciled job's global limit is genuinely
registered via the real limit_get_usage_instances() accessor, and that
a job the live path already decided is never reprocessed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Relocates sched_job_picker()/sched_allocator() out of scheduler.c
(1382 lines, already over this project's 800-line cap) into a new
sched_election.c, retyped to xce_sched_core_t* -- the single
implementation both drivers will spawn (task 4.1 wires simulation
mode onto it). Independently verified bit-for-bit identical to the
pre-relocation bodies modulo the sched->/core-> rename; no statement
reordered, no branch changed.
sched_make_allocation() becomes extern (was static, sole caller moved
translation units); the old commit_decision() thin wrapper is deleted,
its sole caller now calls sched_commit_decision() directly with
identical semantics. sched_init() updated to spawn the relocated pair
against &(sched->core).
Adds an end-to-end fixture test proving a job flows from picked to a
committed decision against bare (non-xce_scheduler_t) core state.
Routes around, rather than fixes, two pre-existing out-of-boundary
gaps this fixture exposed (seqjob/parjob allocator init still reading
plugin->sched; make_sched_job()'s parse path not threading
maxtasks/mintasks) -- both documented in the test's own comments for
future follow-up.
sched_election.c added to the normal build only; sched_sim_isolation_lib
membership deferred to task 4.1, which is where sim mode actually
starts spawning these threads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Migrates match_job_global_limit_()/limit_job_election()/
match_job_host_limit_() (plugins/limit.c) to read plugin->core (added
task 1.2) instead of deriving xce_scheduler_t* from plugin->sched,
which stays NULL for a plugin registered via plugin_register_core()
(simulation mode). Removes the dead ctx.sched write flagged during
task 2.1's review -- verified this ctx never has .sched read anywhere
downstream.
Fixes bug-1828 (found while writing this task's own TDD test for a
previously untested code path): match_job_host_limit_() had the same
key_find()-negative-rc-treated-as-fatal defect bug-1786 already fixed
in its global-limit sibling, but never applied here -- meaning
host-scoped resource limits were never actually enforced, in any
mode, until this fix (not merely a display gap; the limit's vres was
never added to obj->hlimits, so sched_fill_limit_requests() always
saw an empty list for it).
TDD: new tests drive the real hook-registered entry points against a
plugin_register_core()-registered plugin, asserting real usage
instances via limit_get_usage_instances() for both global and
host-scoped limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Retypes sched_make_allocation()/sched_make_primary_allocation()/
sched_make_compound_allocation()/sched_make_alter_allocation()/
sched_commit_decision(), plus 3 private helpers found necessary
during migration (sched_make_primary_alloc_(), sched_make_seqjob_
allocation(), sched_make_parjob_allocation()) from xce_scheduler_t*
to xce_sched_core_t* in scheduler.c.
Wires the one genuine normal-mode-only touch point -- the
unsatisfiable-resreq cache, inside sched_make_primary_alloc_() --
through the task-1.1 NULL-safe callback seam (core->unsatisfiable_
lookup_cb/populate_cb) instead of a direct field read. No other
normal-mode-only state was found in any of the 8 migrated functions,
independently confirmed by both the implementer and reviewer against
xce_scheduler_t's own field list -- no escalation needed.
Minimal &(sched->core) call-site wraps applied inside sched_allocator()/
commit_decision() (task 3.1's home, not yet migrated), matching the
pattern established in tasks 1.2/1.3/2.1.
TDD: two-part RED -- a compile-error RED from the sched_commit_decision
retype, and a behavioral RED using spy callbacks proving the cache
seam was genuinely unconsulted before this fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Retypes xce_sched_ctx_t.sched (sched/types.h) from void* to
xce_sched_core_t* outright, and sched_try_alloc_on_host_() (resource.c)
to take xce_sched_core_t* directly -- completing what tasks 1.2/1.3
left as temporary &(sched->core) call-site wraps. Removes the two
resulting struct-prefix casts entirely (schedobj.c's
sched_ctx_set_primary(), host_iter.c's sched_host_iter_next()): both
now pass core state through honestly, with no cast.
Updates every sched_ctx_init()/sched_try_alloc_on_host_() call site
tree-wide (scheduler.c, parjob_allocator.c, seqjob_allocator.c) with
zero change to host_iter/parjob/seqjob's own host-iteration/
bucket-construction algorithm.
Adds a driver-parity test for sched_try_alloc_on_host_() proving
identical grant/deny results from normal-mode-shaped and
simulation-mode-shaped core state.
Flags (for task 2.3) a confirmed-dead, type-confused write at
plugins/limit.c:319 (ctx.sched = plugin->sched) -- never read today,
left untouched since limit.c is task 2.3's boundary, not this one's.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Re-types sched_meet_host_selection(), sched_check_shared_satisfy()
(+its _ helper), sched_check_host_satisfy() (+its _ helper), and the
private unsatisfy_() helper they share, from xce_scheduler_t* to
xce_sched_core_t*, in resource.c.
Fixes every call site this forces (a C signature migration cannot
keep the build green otherwise): sched_try_alloc_on_host_() (2 sites,
already partially touched by task 1.2), host_iter.c's
sched_host_iter_next() (2 sites), and parjob_allocator.c/
seqjob_allocator.c's my_alloc() (1 site each) -- each fix is a minimal
&(sched->core) argument wrap, no algorithm change, preserving
host_iter/parjob/seqjob's own allocation algorithm exactly as-is.
Adds test_core_safe_allocation_chain_driver_parity, proving all three
functions produce identical grant/deny results whether given
normal-mode-shaped or simulation-mode-shaped core state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Re-types plugin_job_election()/plugin_prepare_allocation()/
plugin_commit_decision() from (void* arg, ...) to (xce_sched_core_t*
core, ...), iterating core->plugins directly instead of deriving it
from a normal-mode-only driver struct -- mirroring
plugin_begin_session()/plugin_end_session()'s already-completed
migration.
Adds a new, honestly-typed xce_sched_plugin_t.core field, set by both
plugin_register() (normal mode) and plugin_register_core() (simulation
mode, closing a gap left since bug-1825 where this path set nothing).
The existing plugin->sched field and its plugins/fcfs.c legacy
consumer are left untouched.
Necessarily updates the 4 existing normal-mode call sites (scheduler.c,
resource.c, schedobj.c) to pass &(...->core) -- a C signature migration
cannot keep the build green without touching every caller in the same
commit; each fix is a minimal argument wrap with no algorithm change,
matching design.md's own Implementation Notes for this component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds a NULL-by-default lookup/populate callback seam on xce_sched_core_t,
mirroring cache_clear_cb's proven shape, so the allocation dispatch
chokepoint can consult the unsatisfiable-resreq cache without a
normal-mode-only field access in shared code. Adapters live in
alloc_cache.c and are registered only during normal-mode startup;
simulation mode leaves both callbacks NULL and always computes fresh.
Adds the scheduler-sim-live-allocation spec (requirements/design/
research/tasks) as the planning artifact this and subsequent tasks
implement against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Four new tests in test_repl_limit_table.c close out Requirement 5:
- test_info_limit_never_mutates_global_limit_config_or_accounting_state /
..._host_scoped_... : snapshot a matched limit's config fields and its
vres capacity/in-use zarray entries (both pointer identity and value)
before and after repeated info limit calls; asserts byte-identical state
and byte-identical repeated stdout output.
- test_info_limit_on_empty_registry_is_repeatable_and_never_registers_plugin:
confirms info limit never causes the plugin to become spuriously
registered as a side effect of being queried.
- test_repl_step_continue_with_info_limit_interleaved_matches_baseline_decision_sequence:
a step/continue sequence run twice (once with info limit interleaved, once
without) produces an identical decision sequence.
Verified via real mutation testing: three targeted fake mutations were
temporarily applied to repl_info.c, confirmed each new test fails exactly as
predicted, then cleanly reverted (repl_info.c shows zero diff from HEAD).
Zero production code changed -- this is a test-only validation task. Full
14-binary regression sweep + 78/78 meson aggregate confirm zero impact on
any pre-existing info subcommand (Requirement 5.3).
This closes out scheduler-sim-diagnostics' first capability (limit
inspection, Requirements 1-5) end to end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_cmd_info() now recognizes "limit" as an info subcommand, routing to the
already-implemented and tested repl_info_limit() with the same ntok<3 usage
guard shape as the queue/worker/resource branches. Usage string and help
text updated to list it. repl_info_limit()'s own body is untouched -- this
is a pure, additive dispatch-wiring change. Two new tests drive it through
the real repl_dispatch() entry point (not a direct call) for both `info
limit <name>` and `info limit all`.
`info limit <name|all>` is now reachable from a live REPL session --
scheduler-sim-diagnostics' first capability is functionally complete.
Remaining: task 5 (determinism/regression validation).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_print_limit_usage_per_host() fans a single host-scoped xce_vres_t out
across every worker with a populated capacity entry at that vres's shared
resource index (w->resvals[idx]/w->resInUse[idx]), rather than constructing
any per-host vres object -- matching design.md's (xce_vres_t, worker) pair
cardinality model exactly. A worker with no populated capacity entry produces
no line at all (not a fabricated zero); a worker with capacity but no
charged usage shows accounted=0. Uses the corrected idx<0 + .caps-only
bounds-check pattern task 3.1 established, avoiding bug-1824's latent
imprecision. This closes out all of Requirement 3 (3.1-3.4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now displays accounted-usage/capacity for GLOBAL/SHARED
limit instances: iterates limit_get_usage_instances(), filters by
vres->limit == matched_limit, reads capacity from core.sharedInst[idx] and
accounted usage from core.sharedInUse[idx]. Existence-of-instance and
zero-usage are kept distinct per design.md: no matching instance (or one
with no populated capacity) prints "no active usage instance yet"; a
matched-but-never-charged instance correctly shows accounted=0 instead.
Self-caught during implementation and independently confirmed by review: a
negative or stale vres->idx (possible per add_vres()'s allocate-then-assign
pattern) is not safely rejected by zarray_get_ex()'s idx>=caps check alone
-- the new code adds an explicit idx<0 guard rather than copying
repl_info_resource_one()'s existing .count-based pre-check (which is itself
imprecise for a sparsely-populated zarray, a pre-existing issue left
out of scope here).
Calling limit_get_usage_instances() for the first time from repl_info.c
reproduced the same ~24-file link-stub ripple task 2.1 established for
limit_get_configured_limits() -- fixed with the identical pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints an explanatory note when a limit is implicitly
host-scoped (is_host_based_limit() > 0 due to a percent-based value on a
non-shared resource) but its own configuration doesn't show an explicit
per-host scope. The real discriminator turned out to be consumers[CONSUMER_
HOST]->per_consumer, not pointer-nullity as design.md's gloss said --
build_consumer() unconditionally allocates a consumer for all four
dimensions, so the struct field is never actually NULL in production. Fixed
using the same field is_host_based_limit() itself already falls back to;
verified via RED-phase reproduction that the literal design.md condition
never fires. design.md corrected to document the real condition.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints a matched limit's configured setting on both the
all and named-lookup success paths: fmt_limit() (existing libxce formatter,
unmodified) into a stack buffer, re-split and printed through a new
repl_print_limit_dump() helper matching this file's existing [sim]-prefix-
per-line convention. Inclusion and exclusion group membership both render
since fmt_group() already handles both, adopted as-is per design.md's
explicit reuse decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() (repl_info.c, not yet wired into repl_cmd_info() -- that's
task 4) resolves "all" vs a specific name against limit_get_configured_limits():
a NULL/empty handle on the all path reports "no limits configured" (Req 1.7);
a NULL handle or key_find() miss on the named path reports "no such limit
'<name>'" (Req 1.3) and never collapses into the all-only message. New
white-box test_repl_limit_table.c exercises both paths directly, including a
dedicated regression guard for the NULL-scoping rule.
Making repl_info.c genuinely call limit_get_configured_limits() broke the link
for ~24 other white-box test files that #include repl_info.c without also
linking plugins/limit.c -- fixed with the same link-satisfying-stub idiom this
file family already uses elsewhere (all return NULL, verified behavior-
preserving since none of these fixtures register the limit plugin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
limit_get_configured_limits()/limit_get_usage_instances() cross the limit
plugin's private-state boundary (plugin->data, previously unreachable outside
limit.c) via key_find on the name-keyed plugin registry, returning NULL when
the plugin is absent/uninitialized and a raw iterable kvmgr handle otherwise.
Pure reads only -- no allocation, mutation, or hook triggering. Foundation for
the upcoming `info limit` REPL command (task 1 of 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_reconcile_host_resource_usage() -- the per-job mem-usage
reconciliation driven by JOB_RUSAGE events -- was missing the same
IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING guard bug-1801 already added to
sched_update_account(), the regular per-round charge/uncharge path.
core_reset_workers() (bug-1751) deliberately freezes a RECONCILING
worker's resInUse each round so a clean rebuild can restore it once the
worker's real state is confirmed; bug-1801 extended that freeze to the
charge path, but this third, independent resInUse-mutating path kept
applying deltas regardless. Combined with core_events.c's JOBS_PUSH
resync-merge RUN branch, which can replace a job's whole account object
(fresh used=0.0 baseline) without uncharging the old one first -- safe
only because the next clean rebuild is supposed to correct it, exactly
the rebuild that stays skipped while RECONCILING -- a rusage tick
landing in that window kept adding a full recharge on top of whatever
was already frozen in the pool, with nothing to ever subtract it back
out until the worker escaped RECONCILING for good or the job finished.
Live-confirmed via a user's own multi-session replay trace: a job's
"mem used" figure never self-corrected until the job finished.
Added the identical guard, mirroring bug-1801 verbatim. RED confirmed
by temporarily disabling it (exactly the new test failed, "Expected 0
Was 5", 76 others green); GREEN confirmed restored (77/77).
bug-1815, follow-up to bug-1814/bug-1801.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.
The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.
test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A resync redelivery (on_scheduler_join()'s full push, fired on every
scheduler reconnect) can race a job's own very-recent dispatch
confirmation and carry a stale, non-terminal state for it. The
duplicate-jobid merge in core_events.c trusted this claim
unconditionally, freeing the job's already-captured allocation and
regressing its state -- live-confirmed via a production scheduler.log
showing a resync WARN landing 16ms after the job's own RUN
confirmation, and matching a live-reported simulator symptom (a RUN
job silently missing from info worker's allocation listing).
Refuse a non-terminal claim when the existing job already has a real
allocation on record, mirroring the existing phantom-RUN guard's own
reasoning in the opposite direction; a genuinely terminal claim
(DONE/EXIT) still applies and releases the allocation correctly, same
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
core_reset_workers already does (bug-1751), closing the gap that let a
worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
a captured job's job->state is set to RUN synchronously at dispatch capture,
mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
live feedback that it over-fires inside busy sessions; replace it with a
prescan-based "next non-dummy session" hint in info summary instead
Full scheduler suite: 77/77 tests pass (asan profile).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
