Task 2.4: outcome capture moves to sched_trace_outcome.c and the
position/host/summary/rollback half of the model to sched_trace_scan.c;
shared helpers go behind sched_trace_internal.h (sti_ prefix). Code is
moved unchanged; every target that compiles the trace sources gets all
four, and the core isolation scan covers the new files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.3: dispatch rows with limit charges listed separately and
resolved through the job's matched limits (raw-number fallback), pend
reason set deep-copied, 50 hosts in detail with the rest aggregated,
and RECONSTRUCTED records holding the dispatch outcome.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_tlog_format_shows_frozen_virtual_time_not_infotime asserted that
"1999" (info->time's year) appears nowhere in the formatted line, but
the line also carries getpid(), so the test failed whenever its PID
contained 1999. It now asserts the exact frozen timestamp field.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.2: each checked row carries name, flags, value type, total,
in-use, reserved, requested and met (same comparison as unsatisfy_());
limit rows resolve through the scope's matched limits and never count as
shortfalls; the cluster-wide check keeps every row, a host only unmet
rows, which set its shortfall/limit verdict.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.1: policy, queue and priority/queue-time ordering keys, each
matched cluster-wide limit as (name, scope), and per-code attempt
failure reasons with the hosts and resources they name, all copied into
the record arena.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.8: the core gains trace_ctl/trace_rec (NULL unless the sim REPL
sets them) and SCHED_TRACE evaluates its call only while a record is
being built.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.7: xce_decision_t gains a trace pointer (NULL from every existing
constructor) and XCE_DECISION_TRACE with make_trace_decision(); the live
driver skips trace-only decisions and frees any attached trace.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.6: the 27 tests that #include a hooked scheduler source now also
include sched_trace.c and sched_trace_capture.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
remove_workdir() now checks snprintf's result and skips a truncated path
instead of unlinking whatever it happens to name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.5: per-requirement plan counts and search-end statements,
decision-wide counters exact past truncation, summary-level position
tallies (per verdict kind and per limit), merged rollback entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_scheduler --sim started with no shared resource pool when the shared
values came only from the recording's SnapshotEntry (license value=(none)),
so jobs needing a shared resource pended instead of dispatching. Two
defects caused it:
- Load side (snapshot_load.c): a resval was treated as shared only when
rv->host == NULL. The capnp round trip (write_snapshot_entry ->
decode_SnapshotEntry) reads a NULL host back as "", so every shared
resval in a .replay file was skipped. A NULL or empty host now means
the shared pool; only a named host marks per-worker data.
- Save side (snapshot.c, snapshot_wire.c): core->sharedInst and each
worker's resvals were walked with i < .count. zarray_t.count is the
number of filled slots, not the highest idx + 1, and both arrays are
sparse by resource idx, so higher-index values were dropped from the
snapshot. Both loops are now bounded by .caps, as repl_info.c already is.
New test test_snapshot_sparse_resvals.c covers the in-memory save/load,
the empty-host load, and the real sched_record_start() -> .replay ->
sched_sim_driver_start() round trip.
Regenerated fixtures/repl_trace_baseline/set_output.replay: its mid-run
snapshot now carries license, limit=cluster_slots and each worker's
limit=host_slots, which the old writer dropped (all other fixtures
unchanged).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.4: requirement path push/pop with balanced begin/end entries
carrying the full nesting path, one entry per traced event kind, and a
line budget that stops storing at the bound and counts omitted lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.3: trace types and constants, per-record arena, job selection
matching, no-decision record finishing; empty capture module; both
sources scanned by the core isolation test with a no-stdio check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.2: translate_units_mode, repl_format_numeric_resource_value,
repl_format_consumable_value_and_used and repl_current_units_mode move
unchanged from repl_info.c and are declared in repl_internal.h so the
trace renderer can reuse them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.1: scripted REPL session over a contention replay (license, cluster
and host limits, parallel and alternative jobs); transcript, recordings and
--sim-verify result committed as fixtures and compared byte for byte.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1904: both scheduler JOB_UPDATE paths parsed "" as-is:
parse_resreq() returns success with a PRIMARY container whose body is
NULL, and apply_job_task_count() wrote through it. Use ingest's default
("type == any") instead.
bug-1906: xcecli jobs list sent local selection-parse errors through
ExtractErrorMessage, which strips brackets and the quoted selection;
report them as written.
Also logs bug-1905 (xcecd_virtual ignores kill of a running job). All
three found by job-array-one-based-index task 10.2's live REST flow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduler expands an array push only when the first ID carries the
array marking and the size fits in its block (size <= 32768 - idx, no
addition); otherwise it logs an ERROR naming the array identity and
expands nothing. Pre-upgrade arrays starting at index 0 still expand.
The INFO log names the array identity (array_jobid=...). This also
closes a crash: a malformed push with a huge size overflowed calloc.
Expansion tests move to tagged block|1 first IDs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
unsatisfy_() compared positional total/inuse/wanted arrays (entry i is
resource wanted->idxs[i]) with uncomitted[i], but the allocation
context's uncommitted vectors are dense and keyed by resource index.
From the second primary of a COMPOUND/ALTER job on, the check ignored
earlier primaries' claim on the same resource (over-admit) or charged
another resource's (wrong pend), with the matching wrong shortfall in
the pend reason.
unsatisfy_() now takes the vector and reads entry wanted->idxs[i]; an
index past its end (sim_loop.c's short all-zero vectors) or a NULL
vector counts as nothing uncommitted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job whose snapshot resreq fails to parse (any replay captured before
the bug-1888 fix) is kept by xce_sched_core_load_snapshot() with no
schedobj, but was still made a picker candidate -- by the loader and, in
simulation mode, by sim_bootstrap_picker()'s picker rebuild. The
allocator thread then dereferenced the NULL schedobj in
limit_job_election() and xce_scheduler --sim crashed.
Both push sites now skip such a job (it stays in the core so a restored
allocation is still charged), and sched_allocator() skips any picked job
without a schedobj as a backstop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comments in scheduler sources and tests that presented the deleted
my_alloc()/parjob claim helpers as current code now name
sched_allocator_run() and its runner helpers; test helpers and tests
named after my_alloc are renamed. No logic or expectation changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seqjob registers sched_allocator_run with seqjob_alloc_iface (no accepts
guard, plus its one-bucket plan iterator); its private my_alloc() is
gone. New fixtures pin the intended changes: a reason for every short
host, the bug-1893 shared reason, the host-stage OOM debit release, and
-EINVAL for missing inputs; seqjob's plan is asserted to be one bucket of
one slot. No other test expectation changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parjob registers sched_allocator_run with parjob_alloc_iface (accepts
maxTasks > 1, bug-1767, plus its plan iterator); its private my_alloc()
and claim helpers are gone. Characterization now pins the intended
changes: a reason for every short host (2/6/5) and the bug-1893 shared
reason. No other test expectation changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_finalize_job_alloc_() turns a satisfied plan's claims into the job
allocation exactly as parjob's commit tail does (host records from the
claim-time snapshots in claim order, shared records, jobid 0, no
duration) and publishes ctx->alloc only on success. Every out-of-memory
step frees what it built and releases the debits of claims not yet
turned into host allocations; each claim snapshot is freed exactly once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The runner tries the plugin's plans in order and claims one distinct host
per bucket, with parjob's rollback and claim snapshots, nslots/nhosts per
candidate, a growable claims buffer and malformed-plan skipping. It
clears alloc_req.host before every candidate so each short host gets its
pend reasons (Req 3.6/3.7). Result assembly follows in task 3.3; until
then a satisfied plan's claims are released and the attempt pends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New alloc_runner.c: sched_allocator_run() validates its inputs (-EINVAL
without touching the pend reason), applies the plugin's optional accepts
guard, creates the pend-reason record, then checks shared resources: one
host-less NO_RESOURCE reason per unmet entry (skip only on vget < 0,
fixing bug-1893) and the met amounts counted once as uncommitted. Plan
search follows in task 3.2. Runner tests use a fake allocator contract;
the pend-reason factory audit now scans the runner.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
15 tests drive the real parjob plugin through job_alloc_fn: success,
short shared, pending on hosts, mid-plan rollback, task counts 0/1,
missing inputs, six OOM steps with their debits, COMPOUND. Values the
runner changes on purpose sit in *_today tests.
Spec: the task showed that parjob records host reasons only for a bucket
search's first rejected candidate (the rejected host's request stays in
alloc_req.host and the host iterator pre-filters later hosts against
it). Decision: the runner clears alloc_req.host before every candidate,
for both allocators, so every short host gets its reasons (Req 3.6/3.7);
publishing reasons only when they change is a follow-up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
12 tests drive the real seqjob plugin through job_alloc_fn and record
today's results, reasons and debits, with switch-stable OOM injection
(sched_make_allocs stub, --wrap of make_host_alloc/make_job_alloc).
Assertions the runner changes on purpose sit in *_today tests.
The characterization showed bug-1893's real effect: vget() returns 0 on
success, so the shared-reason loop (rc <= 0 skip) never records a shared
NO_RESOURCE reason. Bug record and spec wording corrected; the runner
skips only on rc < 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_allocator_iface_t (optional accepts guard + plan_init/next/fini)
and the sched_allocator_run() entry point in sched/alloc_iter.h; the
plugin's const allocator_iface reference, forward-declared in plugin.h
with the picker_iface ownership rule. No plugin uses them yet.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_alloc_plan_iter_t.init takes xce_sched_core_t* instead of
xce_scheduler_t*; both built-in generators, both allocators' call sites
and the contract-test stub follow. The parjob purity test now shows an
identical plan sequence for a NULL and a poisoned core.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
zarray_get_ex() checked only idx >= caps, so a negative index read
array->data[idx] out of bounds -- reachable through a resource whose
registration failed and kept idx == -1 (xcelimit.c add_vres()). It now
returns NULL for idx < 0.
repl_info_resource_one() bounded a shared resource's slot by
sharedInst.count / sharedInUse.count, the number of populated slots, not
the capacity: a populated slot above that count showed as value=(none).
It now relies on zarray_get()'s own bounds check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_sim_driver_run() started the picker and allocator threads on
&sim->core and returned with them still running; only
sched_sim_driver_fini() stopped them, and 24 test files (35 call sites)
run the driver on a stack-local sim without calling it. The leftover
threads kept a pointer into the dead frame, and ASAN occasionally caught
one reading it (stack-use-after-return in cbuf_get) in
sched_sim_verify_test and sched_sim_record_output_test.
sched_sim_driver_run() now wraps the loop and stops both threads on
every return (new sched_sim_driver_stop_threads(), shared with fini). If
the allocator fails to start, the running picker is stopped too. The
stop no longer pushes a jobid 0 to wake the allocator: cbuf_get/put wait
in pthread_cond_wait, a cancellation point, and the fake end-of-session
was visible to the allocator.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The primary_resreq action ignored parse_resreq_primary()'s result, so a
primary whose select (or rusage) clause failed to parse was accepted
without that clause: select[type==] or select[mem>] matched every host.
A malformed clause now fails the whole resreq, and parse_resreq() returns
the clause's own message.
A select naming an unregistered resource is still dropped, not an
error: parse_select() reports it as -ENOENT (new select_context_t
.unknown_resource) and the action lets it pass, because jobs rely on it
-- the scheduler's own "type == any" default among them when no "type"
resource is registered (making it fatal failed 36 scheduler suites).
Test: test_resreq_primary_select_error_fails_parse RED before (all
accepted), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_wire_job() filled the saved job's submit->resreq with
format_resreq()'s human-readable dump ("Primary:\n Tasks: ..."). The
loader's parser accepted that as a primary with neither select nor
rusage, so every job restored from a snapshot (a replay's start snapshot,
REPL snapshots) lost both.
A scheduler job now keeps the text its resreq was parsed from
(xce_sched_job_t.resreq_str, set at ingest, JOB_UPDATE and snapshot load,
freed at teardown) and the snapshot saves that; a job without it restores
as the loader's "type == any" default.
Test: test_snapshot_round_trip_keeps_job_resreq_text RED before (saved
text was the dump), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_init_queues() built a new xce_sched_queue_t for every configured
queue on each config apply; for an existing name the add failed, and the
unregistered queue was handed to the picker (which then tracked it
instead of the registered one) and leaked, while the registered queue
kept its old settings. It now updates the registered queue in place, as
QUEUE_UPDATE's core_upsert_queue() does.
sched_init_config() and the core's CONFIG_PUSH replaced core->config
without freeing the previous one. The core owns its config, and nothing
points into a replaced one (queues, resources, groups and limits built
from it are copies), so it is freed once the new one is applied.
Test: test_config_reapply_updates_queue_in_place_and_frees_old RED before
(priority stayed 1), GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_driver_run_normal_mode_round() drained every decision, handed it
to the recorder and the decision callback, and never freed the struct
make_*_decision() allocates: 64 bytes per decision for the life of the
scheduler. The struct is freed after the callback returns; its allocation
or pend reason belongs to the job, and the callback and recorder copy
what they keep.
Test: test_round_frees_decision_after_callback (ASAN shadow check) RED
before, GREEN after.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_resval() (every resource-value array's item killer) frees
the value, not only a string's buffer: each replaced or torn-down value
leaked its struct, including every WORKERS_PUSH value update.
- put_resvals() shares the value it creates with later entries for the
same index in one call; it created a second one that replaced (and now
freed) the first while still writing through it.
- JOB_UPDATE and the duplicate-JOBS_PUSH path kill_resreq() the old parsed
resreq, and free the rejected job's schedobj (xce_core_free_schedobj(),
exported from core.c), instead of freeing only the containers.
- The previous attempt's (or ALTER alternative's) pend reason is freed when
a new one replaces it.
- kill_resreq_ctx() frees the parser context's error message, and the
primary-resreq PEG action frees an earlier primary's message first.
- write_snapshot_entry() frees the snapshot it builds (config and resource
objects detached, they are the core's); the save side strdup()s its
strings so the generated free_SnapshotData() applies.
Test: test_cli_batch_replay_exits_clean_under_leak_detection
(test_sim_discrete_clock.c) runs the real binary on a two-session fixture
with detect_leaks=1 and requires exit 0: RED before (exit 1), GREEN now.
All nine HA-soak recordings replay with exit 0 and no LeakSanitizer report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failed key_add() runs the table's datakiller on the rejected value, and
callers keep using it (JOBS_PUSH merges a duplicate job into the existing
one; sched_init_queues() re-adds every queue on each config apply), so the
job, queue, worker and resource datakillers must stay shallow. Part 4 made
kill_sched_job() free the whole job; that is reverted. Instead
xce_sched_core_fini() collects each table's entries, destroys the table,
then frees each entry with a full destructor (core.c). The resource table
is released the same way (a reference per resobj).
Also:
- dup_val() tested which == XCE_VAL_STRING (0, the resource value type,
equal to Value_u_ivalue32): real string values were shallow-copied and a
nonzero int32 went to makeString() as a pointer. It now tests
Value_u_svalue and strdup()s, as free_Value() expects.
- The shared/host satisfy checks re-initialize the unmeet vector they are
given; the allocators passed the ctx's live one, dropping its buffers on
every candidate host. The callers finalize it first.
- add_vres() registers a resobj via makeResource(), like every other
resource, instead of a bare calloc(); test_limit.c drops it with
dereference().
- plugin_register_core() rejects a duplicate name before init_fn/key_add,
since a failed add now finalizes (frees) the plugin and callers free it
on failure.
- sharedInUse gets kill_sched_resval, as put_resvals() fills it.
Exit-time leak on the reproduction replay 7,469 -> 928 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kill_sched_job() (the job table's datakiller) frees the whole job:
schedobj, parsed resreq, allocation, accounts and rusages; new
kill_sched_obj() frees the schedobj's names, limit lists, per-primary
wrappers and their pend reasons, and its last allocation/reason.
- The simulator frees each decision struct after draining it (live and
reconciled); the alloc/pend reason it points at stays the job's.
- The builtin_fcfs heap plugin gets a fini_fn that also frees the plugin;
builtin_fcfs_fini() still frees only the picker state (tests run it on
stack plugins).
- xce_sched_core_fini() frees the two pend_updates queues both drivers
calloc().
Exit-time leak on the reproduction replay 11,426 -> 7,469 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_sched_core_load_snapshot() keeps the snapshot's config as core->config
and copies everything else, so the simulator now moves the config into the
core and frees the rest of the decoded SnapshotEntry. xce_sched_core_fini()
frees core->config last, after the plugins and group managers that may
point into it; nothing freed it in either mode. Exit-time leak on the
reproduction replay 25,807 -> 11,426 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sim_process_replay_event_message() now hands the decoded replay event to
the caller, which keeps it beside the translated core event and frees it
after xce_core_push_events() with driver.c's release_translated_payload()
rules: a pushed config is kept by the core, and a rusages report is kept
by the job it found. An untranslatable event is freed at once, and events
after the last session are freed at the end. The payloads were an
"accepted leak" per event; exit-time leak on the reproduction replay
75,495 -> 25,807 bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A batch --sim replay under ASAN exited 1 on 17.2 MB of LeakSanitizer
reports. This removes all but 75 KB:
- sched_sim_driver_fini() (called by main after a clean run) cancels and
joins the picker/allocator threads like sched_fini(), closes the replay,
alternate-events and output files, and finalizes the core. The threads'
ids now live in core.jobpicker/allocator, as in normal mode.
- xce_sched_core_fini() destroys the plugin, job, worker, queue, user and
project registries and the shared-resource arrays it creates.
- The seqjob/parjob sched_make_allocs() stand-ins set up only lock, nextid
and the aliased resmgr. Each ran a full xce_sched_core_init(), which also
replaced the process-wide string and selector tables mid-startup.
- plugin_unregister(), the plugin registry's datakiller, only runs fini_fn;
it key_del()'d from the registry being destroyed and skipped plugins
registered with plugin_register_core().
- The limit plugin's usage-instance table used kill_limit as datakiller on
xce_vres_t records (bad free on destroy); kill_vres now frees the record.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A COMPOUND/ALTER job's primaries share one ctx. The job-level prepare pass
reset only alloc_req.shared, and the allocators append their slots want
into alloc_req.host, so a later primary's host pre-filter (and a global
limit on slots) read the previous primary's last per-host want: a 1-task
primary after a 4-task one was filtered on slots:4 and pended although a
host with its one free slot existed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add test_sim_replay_reproduces_recorded_alloc_from: a recorded DISPATCH at
an epoch-scale beginRound with a sub-second part replays at the default
simulation settings with no --sim-verify mismatch, and the live allocation's
from equals beginRound in whole seconds. Share the dispatch fixture setup
of the existing tests through two helpers.
Live: 2-round HA soak recordings (11 DISPATCH decisions) replayed with
--sim-verify and no --sim-speed report 0 mismatches.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
