limit_seed_config_impl() key_add()ed each pushed limit into the existing
table and returned at the first failure, so on any second push (live
config upgrade, or a simulator CONFIG_PUSH since bug-1964) the first
already-present name failed with -EEXIST: changed limits kept the old
definition and later new limits were never added.
A push now builds the whole new limit set into a fresh table and swaps it
in (all-or-nothing: on any build/register failure the previous set and its
instances stay). Usage instances are carried across: an instance of a kept
limit whose scoping and resource are unchanged is rebound to the new
definition; an instance of a removed limit, or of one whose per-consumer
scoping, resource or percent changed, is dropped and re-created on the next
match. Usage itself lives in the resource manager under the instance's
virtual resource, so counts survive either way. Not part of the snapshot
(a snapshot carries the config and reseeds from it).
The seed moves to plugins/limit_config.c (limit.c is near the 800-line
cap) with the private data type in plugins/limit_internal.h. Two REPL
limit-table fixtures relied on the additive behaviour (two single-limit
pushes) and now configure both limits in one push.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: parjob's allocation now draws candidate hosts from the shared
sched_host_iter_next() (host_iter.c) instead of its own
sched_collect_workers() loop, so skip_bad_worker_ lost its only caller
(the compiler warns it is unused). seqjob's copy was already removed when
it moved onto the shared iterators (b18c5f78).
Fix: remove it. Its only check (a NULL worker) is done by
sched_host_iter_next(), which is also where host filtering now lives
(closed hosts, bug-1939; unavailable and reconciling hosts, bug-1963).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: live applies every pushed config through sched_init_config(),
whose plugin_config_update() runs each plugin's config_hook (the limit
plugin builds its limits there). The simulator seeded the limit plugin
only at bootstrap (sim_bootstrap_limit_plugin()); the core's CONFIG_PUSH
event loaded groups, resources and queues but never told the plugins,
so a limit added during a replay never existed in sim.
Fix: one shared fan-out, plugin_config_update_core(core, config), which
live's plugin_config_update() now delegates to. The core's CONFIG_PUSH
calls a driver-registered xce_core_config_cb_t after loading groups (live
order), keeping the core free of plugin.c; the simulator registers the
fan-out in sim_bootstrap_limit_plugin(). limit_update_config() reads the
honestly-typed plugin->core instead of casting plugin->sched, so the hook
is safe on the sim core; the hand-built plugin fixtures in
test_limit_plugin.c/test_repl_limit_table.c now set plugin->core as
plugin_register() does. test_sim_limit_config_push_parity.c covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cause: sched_host_iter_next(), the one candidate-host source of the
seqjob and parjob allocators on both drivers, skipped only admin-closed
hosts (bug-1939). A host marked XCE_WORKER_UNAVAIL or
XCE_WORKER_RECONCILING was still offered, and its frozen resInUse
(bug-1799/1801) does not reject it, so new jobs were placed on it.
Fix: skip IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING hosts next to the
closed-host check. test_host_unavail_reconciling.c covers serial and
parallel jobs on the normal-mode and sim-mode fixtures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both job-fields update paths, the core JOB_UPDATE event (core_events.c,
used by the live and sim drivers) and sched_update_existing_job()
(schedobj.c), set job->priority to the new queue's priority when a job
changed queue, while ingest gives every job 0. builtin_fcfs orders a
queue's jobs by job priority first, so a moved job overtook every job
already pending in the target queue (and snapshot restore, which sets 0,
disagreed with live for moved jobs).
Moving now only re-points job->queue; queue priority still orders the
queues themselves. test_job_move_priority.c moves a job into a priority-10
queue and checks it keeps priority 0 and does not take the only slot from
the queue's first job; test_job_update.c's in-place update case asserted
the old behaviour and now expects the job's priority to be kept.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only live's sched_init_config() (scheduler.c) registered the config's
named user and host groups into core.ugmgr/hgmgr. The simulator takes its
config from the replay snapshot (snapshot_load.c) and later pushes through
the core CONFIG_PUSH event (core_events.c); neither loaded groups, so a
limit whose users name a group ("ug1") never matched in sim, while the
same limit matched in live.
New xce_core_load_groups() (core.c) is the one loader, called by
sched_init_config(), the CONFIG_PUSH event and snapshot restore, each
before limit consumers are built. snapshot.c's field notes now say the
groups are rebuilt from the snapshot's config.
test_sim_group_limit_parity.c checks a ug1-scoped limit matches the same
job on both drivers and that CONFIG_PUSH alone registers ug1 and hg1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No scheduler code read XCE_WORKER_CLOSED / XCE_WORKER_CLOSED_BY_ADMIN:
sched_host_iter_next() (host_iter.c), the candidate-host source for both
the seqjob and the parjob allocator, filtered only on the select
expression and resource availability, so jobs kept landing on a host an
admin had closed (parjob's skip_bad_worker_ hook is empty and unused).
sched_host_iter_next() now skips a host with either closed bit before
opening its trace scope, which covers every new placement for serial and
parallel jobs on the live and simulation drivers. Accounting of jobs
already running on the host is unchanged. No new pend-reason code: a
closed host is simply not a candidate. test_host_admin_close.c covers a
serial and a parallel job on both fixture shapes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sched_queue_is_ready() in core.c was a `return 1` stub, so the picker
(builtin_fcfs and the legacy fcfs) took jobs from every queue and admin
queue inactivation never reached dispatch.
It now returns false for a queue with QUEUE_STAT_INACT_ADMIN or
QUEUE_STAT_INACT_WINDOW set, so an inactive queue's jobs stay pending.
QUEUE_STAT_CLOSED is deliberately not a gate (LSF semantics: closing only
blocks new submissions; already-pending jobs still dispatch). The status
reaches the core through the same queue add/update paths in both drivers,
so live and sim behave identically. test_queue_admin_control.c covers
open, closed and inactive queues on both fixture shapes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The limit plugin called build_limit() with the host-group and
project-group managers swapped, so a limit's hosts: members were looked up
among the project groups (a host-group name became a literal host name and
the limit never applied) and its projects: members among the host groups.
Pass them in the order build_limit() declares.
New test test_limit_scopes_resolve_against_own_group_kind; the test's
main() now initializes the interned-string table that group names need.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The parjob plan generator split only maxTasks, so a job submitted with a
task range (minTasks < maxTasks) pended until all maxTasks tasks fit,
even when minTasks did (a tasks 2..5 job pended on a host with 2 free
slots).
The runner now handles a task range in sched_alloc_range_(): it probes
minTasks first and pends after that one search when it does not fit;
otherwise it places maxTasks when that fits, or binary-searches for the
largest count that fits and places it. Fitting is monotone in the count
for balanced plans, so the search is exact for today's plan shapes.
ctx->ntasks carries the count; parjob's job-level pass scales per-task
shared wants and global limits to it, and the generator gets the request
with its task count fixed to it. Fixed counts keep the old path.
The decision trace shows each count tried (task count N (minimum) /
(maximum), placing N tasks) and releases a fitting probe's hosts.
Test fixtures that used minTasks = 1 as filler for an N-task job are now
fixed-count.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The session-level unsatisfiable-resreq cache stored only the exhausted
attempt's main reason (an int). The allocators record shortfalls as
per-host/per-resource entries and leave main_reason at 0, so every cache
hit in the live scheduler pended its job with an empty reason. The
simulator registered no cache at all (its table lived on xce_scheduler_t),
so it searched every identical pending job again and its traces differed
from production.
- The cache table moves to xce_sched_core_t; cache_lookup/insert/clear and
the hook adapters take the core, with no cast to a driver struct.
sched_init() builds it, cache_clear() rebuilds it per session,
xce_sched_core_fini() destroys it.
- A cache entry is a deep copy of the attempt's full xce_job_preason_t
plus the job that proved the request unsatisfiable (core->alloc_jobid,
set by sched_allocator()). A hit moves a copy into ctx->reason; a miss
leaves ctx->reason untouched.
- main.c passes the same three adapters to the simulator through new
sched_sim_args_t fields; sched_sim_driver_start() sets them on the sim
core (plain field writes: white-box tests include sim.c alone).
- Decision trace: a hit renders "no plan generated: skipped, same request
as <job> proved unsatisfiable earlier this session" (new
XCE_TRACE_NOPLAN_KNOWN_UNSAT via sched_trace_known_unsat()); the outcome
shows the restored reasons.
- sim_verify keeps its main-only PEND compare (old recordings carry
main-only hits); comments and the scheduler-sim-live-allocation design
note updated.
Tests: full-reason assertions in the cache dispatch tests (RED before:
"a cache hit must restore the original attempt's FULL pend reason"),
test_alloc_cache.c ported to the new API (deep copy, clear frees, -EEXIST
frees), a sim-registered clear test, a renderer test, and a sim e2e
scenario with two identical unsatisfiable jobs (204, 205) where 205 is
skipped with 204's full reasons (RED before: no skip line).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduler appended one NO_RESOURCE/MATCH_FAIL entry per host per
placement plan, so a pending parallel job carried plans x hosts copies
(~93k entries in the 1000-host bounds test). libxce gains
add_job_preason_once() with a per-context hash index keyed on
(host, resource, code) that keeps first-occurrence order; the
scheduler's three builders use it. add_job_preason() is unchanged for
other callers. The parjob equivalence test now expects each short host
once; the bounds test runs in ~2 s instead of ~19 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
repl_u64_str_eq() compared the packed jobid's raw decimal string with
the filter text, so a breakpoint on the dashed jobid the REPL prints
never fired. The filter is now parsed once when the breakpoint is set
(repl_parse_jobsel_arg, moved to repl_fmt.c) and matched with
repl_jobsel_matches: dashed ids, X[k], X[a-b] and raw decimal ids all
work, a bare array-aligned id also matches its elements, and an
unparseable value is rejected instead of creating a dead breakpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bug-1928: repl_count_jobs() and repl_info_jobs() never destroyed their
job-table iterators (16 B per continue round); both now do, and the
baseline and trace wiring tests run with leak detection on.
bug-1929: continue-mode round summaries listed only pending jobs, so a
dispatch-only round printed an empty jobids= and rounds dispatching
different jobs collapsed together; rounds now record and list every
affected job (scheduler-sim-repl-verbosity Req 2.1, 3.2). The baseline
transcript changes only in its two jobids= lines.
bug-1930: trace on job now accepts the raw decimal jobid form that info
trace and info job accept, through a shared parser.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Feature validation round 1: tests for trace off keeping recall (1.5),
no logging from REPL trace sources (8.7), reconstructed records in the
trace file (8.1) and first in continue mode (7.3), output after trace
off matching an untraced run (10.4), and tracing enabled only from the
REPL (11.1). design.md now reflects the REPL-side file split, the added
API, the dropped file tag and the test files; duplicate trace
declarations removed from repl_internal.h.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.4: a 1000-host, 64-task parallel job traced at full level stores
exactly the line bound with exact omitted counts and counters; 10,000
traced decisions keep the trace store within 64 MiB without evicting the
newest. The scheduler guide lists the REPL trace files and test rules
and carries measured file sizes. Full all-components ASAN build passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.3: the contention and extra replays run with tracing off, all
jobs at summary and full level, and a job subset, in step and continue
mode; decisions, full pend reasons, --sim-output recordings, snapshots,
--sim-verify results and per-step accounting dumps are identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.2: injects plan-claims growth, snapshot, finalize, host-scan,
per-host uncommitted lookup, allocation-function and unknown-jobid
faults and asserts each traced outcome. Also fixes two deferred OOM
paths: a failed limit row keeps the host's verdict bits, and an array
listing that cannot be built is reported instead of silently truncated.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pend outcome printed 'main reason: code 0' for every pend the
allocator builds, because 0 (libxce's 'no main reason set') fell through
to the unknown-code fallback. Found by the 6.1 end-to-end review.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 6.1: drives the real simulator REPL with tracing on for all and
selected jobs at both levels, asserting every Requirement 2-6 section,
ordering in step and continue mode and at breakpoints, info trace at a
breakpoint, after finish, for arrays and after a units change, and a
byte-identical trace file copy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
repl_cmd_quit() (quit and EOF) released the trace settings while the
picker and allocator threads could still run; at a mid-session
breakpoint the allocator reads core->trace_ctl unlocked on every job
pop, a possible use-after-free just before exit. quit now cancels and
joins both threads first, as sched_sim_driver_run() does on return.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 5.3: session hooks around each sim session; trace-only decisions
are handled first and skip verification, recording, echo and
breakpoints; traced decisions are emitted after the echo gate and before
the breakpoint check; reconstructed dispatches get reconstructed
records; unclaimed traces are freed at both decision free sites.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 5.2: trace is routed with the raw line and info trace through info;
help gains a Trace section; set output rejects open trace files; quit
and EOF shut tracing down; the continue-mode rollup reports a traced
round on its own; repl.h gains the interactive-only trace gates. The
baseline transcript changes only in the help and info-usage lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 5.1: the 26 white-box tests that include repl.c or sim_loop.c now
include the REPL trace sources (and the core trace sources where no
hooked source brings them); sched_sim_isolation_lib carries all trace
sources and the sim isolation test scans the REPL trace files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.6: session begin makes the next settings current, stamps the
session number and publishes them to the core; session end withdraws and
releases them. Emission marks the round traced, flushes the collapsed
range and reprints the suppressed echo in continue mode, renders to the
console and the trace file with identical bytes, and stores the record.
sim_print_decision is exported unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.5: a named trace file is opened for appending at command time;
an open failure or a path that is the sim output file or the scheduler
log (same device and inode) is rejected with settings unchanged; files
close when their last settings holder releases them; set output gets a
conflict check; shutdown flushes and closes every trace file and frees
the store.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.4: the --logfile path (or the default, now SCHED_DEFAULT_LOG_PATH)
reaches the sim driver as an owned copy so the trace file checks can
reject the scheduler log.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.3: trace on all|job SEL... [level] [lines N] [file PATH], off and
status, parsed from the raw line; invalid arguments are named and leave
the settings unchanged; refcounted next/current settings; status shows
the settings in effect or pending, whole arrays as <base>[*], and stored
trace memory against its bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.2: a new trace replaces a job's earlier one; above 64 MiB the
oldest traces are evicted (never the newest) and marked discarded until
a newer capture; info trace renders a job's or an array's stored traces
in the current units mode, or reports discarded / no trace with the
job's state / unknown job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 4.1: hierarchy, summary-level position summaries, truncation marker
with the fixed totals block, 50-host outcome cap with aggregates and
separate limit charges, no-decision and reconstructed records, amounts
in the REPL units mode, identical bytes to one or two sinks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 3.4: the runner records no-plan reasons, the cluster-wide check,
plans (tried, discarded, not tried) and their ends, positions, claimed
skips and per-host candidate verdicts, rollbacks, commit or finalize
failure, and search end. A declined or generator-failed requirement now
counts once as 'no plan generated'. Runner results are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 3.3: the pre-filter records selection mismatches and internal
errors as host verdicts, a scan stopped by an error marks the position
as a scan error, and unsatisfy_() hands every checked row to the current
record. Iterator and check results are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 3.2: single requirements, top-level compounds, alternatives and
members push their path level and record one begin and one end; member
failures record the members not tried, merge failures and skipped
alternatives are separate entries, the chosen alternative or all-failed
is recorded, and each compound attempt's reasons are captured before the
job's stored reason is replaced. Scheduling results are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 3.1: when the sim publishes trace settings, sched_allocator() counts
job positions, starts a record for wanted jobs with their selection and
cluster-wide limits, pushes no-decision kinds as trace-only decisions,
and attaches the captured outcome to built decisions; every handoff
clears core->trace_rec. Live (trace_ctl NULL) is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.4: outcome capture moves to sched_trace_outcome.c and the
position/host/summary/rollback half of the model to sched_trace_scan.c;
shared helpers go behind sched_trace_internal.h (sti_ prefix). Code is
moved unchanged; every target that compiles the trace sources gets all
four, and the core isolation scan covers the new files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.3: dispatch rows with limit charges listed separately and
resolved through the job's matched limits (raw-number fallback), pend
reason set deep-copied, 50 hosts in detail with the rest aggregated,
and RECONSTRUCTED records holding the dispatch outcome.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_tlog_format_shows_frozen_virtual_time_not_infotime asserted that
"1999" (info->time's year) appears nowhere in the formatted line, but
the line also carries getpid(), so the test failed whenever its PID
contained 1999. It now asserts the exact frozen timestamp field.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.2: each checked row carries name, flags, value type, total,
in-use, reserved, requested and met (same comparison as unsatisfy_());
limit rows resolve through the scope's matched limits and never count as
shortfalls; the cluster-wide check keeps every row, a host only unmet
rows, which set its shortfall/limit verdict.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 2.1: policy, queue and priority/queue-time ordering keys, each
matched cluster-wide limit as (name, scope), and per-code attempt
failure reasons with the hosts and resources they name, all copied into
the record arena.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.8: the core gains trace_ctl/trace_rec (NULL unless the sim REPL
sets them) and SCHED_TRACE evaluates its call only while a record is
being built.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.7: xce_decision_t gains a trace pointer (NULL from every existing
constructor) and XCE_DECISION_TRACE with make_trace_decision(); the live
driver skips trace-only decisions and frees any attached trace.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.6: the 27 tests that #include a hooked scheduler source now also
include sched_trace.c and sched_trace_capture.c.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
remove_workdir() now checks snprintf's result and skips a truncated path
instead of unlinking whatever it happens to name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.5: per-requirement plan counts and search-end statements,
decision-wide counters exact past truncation, summary-level position
tallies (per verdict kind and per limit), merged rollback entries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xce_scheduler --sim started with no shared resource pool when the shared
values came only from the recording's SnapshotEntry (license value=(none)),
so jobs needing a shared resource pended instead of dispatching. Two
defects caused it:
- Load side (snapshot_load.c): a resval was treated as shared only when
rv->host == NULL. The capnp round trip (write_snapshot_entry ->
decode_SnapshotEntry) reads a NULL host back as "", so every shared
resval in a .replay file was skipped. A NULL or empty host now means
the shared pool; only a named host marks per-worker data.
- Save side (snapshot.c, snapshot_wire.c): core->sharedInst and each
worker's resvals were walked with i < .count. zarray_t.count is the
number of filled slots, not the highest idx + 1, and both arrays are
sparse by resource idx, so higher-index values were dropped from the
snapshot. Both loops are now bounded by .caps, as repl_info.c already is.
New test test_snapshot_sparse_resvals.c covers the in-memory save/load,
the empty-host load, and the real sched_record_start() -> .replay ->
sched_sim_driver_start() round trip.
Regenerated fixtures/repl_trace_baseline/set_output.replay: its mid-run
snapshot now carries license, limit=cluster_slots and each worker's
limit=host_slots, which the old writer dropped (all other fixtures
unchanged).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.4: requirement path push/pop with balanced begin/end entries
carrying the full nesting path, one entry per traced event kind, and a
line budget that stops storing at the bound and counts omitted lines.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.3: trace types and constants, per-record arena, job selection
matching, no-decision record finishing; empty capture module; both
sources scanned by the core isolation test with a no-stdio check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.2: translate_units_mode, repl_format_numeric_resource_value,
repl_format_consumable_value_and_used and repl_current_units_mode move
unchanged from repl_info.c and are declared in repl_internal.h so the
trace renderer can reuse them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Task 1.1: scripted REPL session over a contention replay (license, cluster
and host limits, parallel and alternative jobs); transcript, recordings and
--sim-verify result committed as fixtures and compared byte for byte.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
