Switches both fmt_limit() call sites in repl_info_limit() (the "all"
path and the named-lookup path) to
fmt_limit_units(..., translate_units_mode(repl_current_units_mode(sim))),
so a byte-valued limit's configured amount now renders human-readable
under the REPL's real, currently-selected units mode -- the same
convention info worker/info resource already honor.
Adds two tests proving human-readable rendering under the default
mode and consistent switching across set units raw|human|both via
real repl_dispatch(). RED confirmed by reverting the call sites
(2 failures, 28 pre-existing tests in the same file unaffected).
Corrects an inaccurate self-reported regression-safety rationale
(there is no slots-resource fixture in this file; safety instead
comes from those tests never asserting on the formatted value line),
logged in tasks.md's Implementation Notes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds fmt_limit_units(buf, len, limit, units_mode) to src/lib/xcelimit.c,
a units-mode-aware sibling of fmt_limit() following the
fmt_jobid()/fmt_jobid_ex() extended-sibling pattern. Reuses fmt_limit()'s
exact consumer-dump loop untouched; only the final value line branches:
percent limits always render bare "%f%%" regardless of mode, byte-valued
(or "swp"-special-cased) absolute values render via
xce_format_units_value() (task 7), everything else keeps the
unconditional "%f".
fmt_limit() itself is now a one-line XCE_UNITS_RAW wrapper around
fmt_limit_units() -- provably byte-identical to its prior output for
every input shape, so its 5 existing direct unit tests needed zero
changes (the regression proof this refactor relies on).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds Requirement 6 (Human-Readable Units for Byte-Valued Limit Amounts)
to the spec: requirements, design, research, and tasks 7-10.
Task 7 (Foundation): promotes the REPL's byte-unit scaling algorithm
and byte-valued-resource classifier into src/lib/utils.c/utils.h
(xce_format_bytes_human, xce_is_bytes_valued_resource), alongside a
new display-mode-aware composer (xce_format_units_value,
xce_units_mode_t) -- following the fmt_jobid()/fmt_jobid_ex()
precedent. repl_fmt.c's scheduler-only functions become one-line
delegating wrappers. repl_info.c's repl_format_numeric_resource_value()
(shared by info worker/info resource, not just the upcoming info
limit) is retyped to delegate to the new shared composer via a small
translate_units_mode() bridge, so every consumer benefits from one
implementation going forward.
No existing symbol's signature changed; only new symbols were added.
Confirmed behavior-preserving: full scheduler suite 85/85 unchanged,
including info worker/info resource's own regression tests. One
narrow, untested, out-of-requirement-scope corner case found and
documented (swp's RAW/BOTH mode rendering under the new single-value
composer interface) -- logged in tasks.md's Implementation Notes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
core_clock.c joined the mechanically-enforced isolation archive
(sched_core_isolation_lib) and its own doc comment claimed coverage by
the forbidden-headers/forbidden-symbols text scans, but was actually
absent from both -- only the nm(1) undefined-symbol layer covered it,
which cannot catch a forbidden #include resolving to no undefined
symbol (a macro-only header).
Added XCE_CORE_CLOCK_C_PATH (meson.build, mirroring the existing
per-file path macros) and extended both
test_core_never_includes_forbidden_headers() and
test_core_never_calls_forbidden_symbols() to scan core_clock.c. It
remains excluded only from the wall-clock scan, for its one sanctioned
clock_gettime() call.
TDD: confirmed RED by temporarily inserting a forbidden #include into
core_clock.c (84/85, correctly attributed failure), reverted
(byte-identical to HEAD), reconfirmed GREEN (85/85). No live violation
existed -- this closes a coverage gap in the mechanism itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Same pattern as bug-1492/1682/1611: tags claimed "fixed"/
"closed-by-scheduler-sim-live-allocation" while the fix narrative said
"NOT FIXED" -- and correctly so, that feature never touched
test_core_isolation.c. Removed the incorrect tags; entry remains open.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.
The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.
Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
/xce-audit-invariants scheduler (dimension 1, driver state-equivalence,
re-run after scheduler-sim-live-allocation) found plugin_end_session()
still checked `plugin->begin_session_fn != NULL` before invoking
`plugin->end_session_fn(plugin)` -- a copy-paste of
plugin_begin_session()'s own (correct) guard. Two failure modes: a
plugin registering only end_session_fn silently never has it invoked;
a plugin registering only begin_session_fn crashes via a NULL
function-pointer call. Currently inert in production (no live plugin
hits either shape) but a real latent defect in exactly the
session-boundary code the driver-equivalence invariant governs.
This exact defect had been independently filed twice: bug-1492
(2026-08-04, scheduler-core task 5.2's review) and bug-1682
(2026-08-24, found adjacent to bug-1681's fix) -- neither investigation
cross-referenced the other. Both entries also carried a "fixed" tag
inconsistent with their own "not yet fixed" narrative text, caught by
this audit's independent code+text cross-check.
Fixed the one-line guard; added test_end_session_invokes_end_session_fn_
when_only_it_is_set() (test_plugin_hook_dispatch.c) proving the silent
non-invocation, confirmed RED (84/85) against the buggy guard and GREEN
(85/85) after the fix. Updated both buglog entries to FIXED with
cross-references, and corrected bug-1611's stale narrative to reflect
its narrowing (not elimination) by scheduler-sim-live-allocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Deployed a real controller+scheduler+worker cluster and used the real
xceadmin scheduler record start/stop production path (no synthetic
corpus existed in this checkout -- bug-1611/1734's referenced replays
were stale /tmp paths, long gone) to capture two genuinely
live-executed .replay sessions (110 rounds/3 jobs, 28 rounds/5 jobs).
Ran --sim-verify against both: zero MISMATCH in either. Session 1
organically exercised the real deployment's global limit1 (cap=3),
correctly PENDing two 22-task jobs for the whole session -- root-caused
live via resource.c's own DEBUG3 trace and confirmed as pre-existing,
driver-agnostic enforcement (identical in normal mode), not a defect.
Session 2 cycled 15 PEND + 5 DISPATCH decisions under the same limit,
directly exercising the live-decision-authoritative rule under real
multi-round timing.
This closes out all 14 leaf tasks (1.1-5.4) of the
scheduler-sim-live-allocation feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Re-ran test_repl_equivalence.c, test_sim_equivalence.c, and
test_sim_contention_parity.c in full: all pass, 85/85, zero assertion
changes anywhere. Confirmed all three still exercise their own
link-satisfying always-PEND sched_job_picker/sched_allocator stub
(added task 4.1), correctly proving the new two-thread jbuf timing
didn't disturb decision ordering -- but not proving real live-path
resource-based dispatch parity, which is already covered separately
by tasks 4.1/4.3/5.1/5.2's real-scheduler_srcs-linked fixtures.
Fixed a stale documentation defect: all three files' header/inline
comments claimed sim mode "structurally CANNOT emit a DISPATCH
decision" via sim_picker_thread_fn(), a function this feature removed
in tasks 3.1/4.1. That claim is now false of production sim mode
(disproved by test_sim_live_allocation.c and others); corrected with
a RECONCILIATION addendum rather than rewriting the original
historical rationale. No production code or assertion values changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds the feature's concrete original acceptance test: a genuine
capnp-encoded .replay (real ReplayHeader/SnapshotEntry/Conf4Sched with
one global and one host-scoped limit), driven end-to-end through the
real, unmodified sched_sim_driver_run(). The dispatched job's usage
registers against both limits via limit_get_configured_limits()/
limit_get_usage_instances() -- the same accessors info limit reads --
proving real, non-zero usage where today's REPL previously showed none.
RED-phase evidence: reverting task 4.3's 4 files to their pre-fix
commit reproduces the exact predicted failure (host-scoped usage
missing, global usage intact); restoring to HEAD reproduces GREEN.
Closes bug-1826 (the bug this whole feature was built to fix) --
buglog entry updated from NOT FIXED to FULLY FIXED, citing this test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds two new fixtures spawning the real sched_job_picker/sched_allocator
thread pair directly against bare normal-mode and sim-mode core state
under genuine resource contention (2.0 capacity, 3 jobs x 1.0 demand),
asserting identical admission decisions and identical accounting state
(job->accounts, w->resInUse), with and without a matching host-scoped
limit. The pre-existing test_sim_contention_parity.c could not be reused
since its white-box stubs hard-code the old always-PEND behavior and
never exercise live election/allocation.
Confirms build_limit_vres_requests()/get_vres_value() (plugins/limit.c)
are #if 0'd dead code with zero live callers -- no limit in this codebase
currently gates/denies admission in either mode, only tracks/displays
usage -- so this proves usage-tracking parity, documented explicitly as
an honest scope boundary rather than a silent narrowing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.
Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.
One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.
TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds a plugin_job_election(&(sim->core), job) call to
sim_reconcile_unaccounted_jobs()'s FOUND branch, before the existing
alloc-reconstruction/state-transition code, so a job whose outcome
comes from the recording still gets its global-limit usage registered
-- the same core-safe dispatcher the live path (task 4.1) already
uses, not a separate mechanism.
The existing job->accounts != NULL trigger guard needs no change: it
already excludes any job the live path has already decided, so this
call is naturally idempotent (match_job_global_limit_'s own
obj->configver guard makes a redundant call a safe no-op) -- narrowing
this function to a genuine timing-gap safety net rather than the
general-case decision source it used to be, per design's Live/Recorded
Authority Rule.
Host-scoped limit matching is deliberately not attempted here (needs a
resolved worker context this function doesn't independently
construct) -- that gap is tracked separately as task 4.3.
TDD: new tests prove a reconciled job's global limit is genuinely
registered via the real limit_get_usage_instances() accessor, and that
a job the live path already decided is never reprocessed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Relocates sched_job_picker()/sched_allocator() out of scheduler.c
(1382 lines, already over this project's 800-line cap) into a new
sched_election.c, retyped to xce_sched_core_t* -- the single
implementation both drivers will spawn (task 4.1 wires simulation
mode onto it). Independently verified bit-for-bit identical to the
pre-relocation bodies modulo the sched->/core-> rename; no statement
reordered, no branch changed.
sched_make_allocation() becomes extern (was static, sole caller moved
translation units); the old commit_decision() thin wrapper is deleted,
its sole caller now calls sched_commit_decision() directly with
identical semantics. sched_init() updated to spawn the relocated pair
against &(sched->core).
Adds an end-to-end fixture test proving a job flows from picked to a
committed decision against bare (non-xce_scheduler_t) core state.
Routes around, rather than fixes, two pre-existing out-of-boundary
gaps this fixture exposed (seqjob/parjob allocator init still reading
plugin->sched; make_sched_job()'s parse path not threading
maxtasks/mintasks) -- both documented in the test's own comments for
future follow-up.
sched_election.c added to the normal build only; sched_sim_isolation_lib
membership deferred to task 4.1, which is where sim mode actually
starts spawning these threads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Migrates match_job_global_limit_()/limit_job_election()/
match_job_host_limit_() (plugins/limit.c) to read plugin->core (added
task 1.2) instead of deriving xce_scheduler_t* from plugin->sched,
which stays NULL for a plugin registered via plugin_register_core()
(simulation mode). Removes the dead ctx.sched write flagged during
task 2.1's review -- verified this ctx never has .sched read anywhere
downstream.
Fixes bug-1828 (found while writing this task's own TDD test for a
previously untested code path): match_job_host_limit_() had the same
key_find()-negative-rc-treated-as-fatal defect bug-1786 already fixed
in its global-limit sibling, but never applied here -- meaning
host-scoped resource limits were never actually enforced, in any
mode, until this fix (not merely a display gap; the limit's vres was
never added to obj->hlimits, so sched_fill_limit_requests() always
saw an empty list for it).
TDD: new tests drive the real hook-registered entry points against a
plugin_register_core()-registered plugin, asserting real usage
instances via limit_get_usage_instances() for both global and
host-scoped limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Retypes sched_make_allocation()/sched_make_primary_allocation()/
sched_make_compound_allocation()/sched_make_alter_allocation()/
sched_commit_decision(), plus 3 private helpers found necessary
during migration (sched_make_primary_alloc_(), sched_make_seqjob_
allocation(), sched_make_parjob_allocation()) from xce_scheduler_t*
to xce_sched_core_t* in scheduler.c.
Wires the one genuine normal-mode-only touch point -- the
unsatisfiable-resreq cache, inside sched_make_primary_alloc_() --
through the task-1.1 NULL-safe callback seam (core->unsatisfiable_
lookup_cb/populate_cb) instead of a direct field read. No other
normal-mode-only state was found in any of the 8 migrated functions,
independently confirmed by both the implementer and reviewer against
xce_scheduler_t's own field list -- no escalation needed.
Minimal &(sched->core) call-site wraps applied inside sched_allocator()/
commit_decision() (task 3.1's home, not yet migrated), matching the
pattern established in tasks 1.2/1.3/2.1.
TDD: two-part RED -- a compile-error RED from the sched_commit_decision
retype, and a behavioral RED using spy callbacks proving the cache
seam was genuinely unconsulted before this fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Retypes xce_sched_ctx_t.sched (sched/types.h) from void* to
xce_sched_core_t* outright, and sched_try_alloc_on_host_() (resource.c)
to take xce_sched_core_t* directly -- completing what tasks 1.2/1.3
left as temporary &(sched->core) call-site wraps. Removes the two
resulting struct-prefix casts entirely (schedobj.c's
sched_ctx_set_primary(), host_iter.c's sched_host_iter_next()): both
now pass core state through honestly, with no cast.
Updates every sched_ctx_init()/sched_try_alloc_on_host_() call site
tree-wide (scheduler.c, parjob_allocator.c, seqjob_allocator.c) with
zero change to host_iter/parjob/seqjob's own host-iteration/
bucket-construction algorithm.
Adds a driver-parity test for sched_try_alloc_on_host_() proving
identical grant/deny results from normal-mode-shaped and
simulation-mode-shaped core state.
Flags (for task 2.3) a confirmed-dead, type-confused write at
plugins/limit.c:319 (ctx.sched = plugin->sched) -- never read today,
left untouched since limit.c is task 2.3's boundary, not this one's.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Re-types sched_meet_host_selection(), sched_check_shared_satisfy()
(+its _ helper), sched_check_host_satisfy() (+its _ helper), and the
private unsatisfy_() helper they share, from xce_scheduler_t* to
xce_sched_core_t*, in resource.c.
Fixes every call site this forces (a C signature migration cannot
keep the build green otherwise): sched_try_alloc_on_host_() (2 sites,
already partially touched by task 1.2), host_iter.c's
sched_host_iter_next() (2 sites), and parjob_allocator.c/
seqjob_allocator.c's my_alloc() (1 site each) -- each fix is a minimal
&(sched->core) argument wrap, no algorithm change, preserving
host_iter/parjob/seqjob's own allocation algorithm exactly as-is.
Adds test_core_safe_allocation_chain_driver_parity, proving all three
functions produce identical grant/deny results whether given
normal-mode-shaped or simulation-mode-shaped core state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Re-types plugin_job_election()/plugin_prepare_allocation()/
plugin_commit_decision() from (void* arg, ...) to (xce_sched_core_t*
core, ...), iterating core->plugins directly instead of deriving it
from a normal-mode-only driver struct -- mirroring
plugin_begin_session()/plugin_end_session()'s already-completed
migration.
Adds a new, honestly-typed xce_sched_plugin_t.core field, set by both
plugin_register() (normal mode) and plugin_register_core() (simulation
mode, closing a gap left since bug-1825 where this path set nothing).
The existing plugin->sched field and its plugins/fcfs.c legacy
consumer are left untouched.
Necessarily updates the 4 existing normal-mode call sites (scheduler.c,
resource.c, schedobj.c) to pass &(...->core) -- a C signature migration
cannot keep the build green without touching every caller in the same
commit; each fix is a minimal argument wrap with no algorithm change,
matching design.md's own Implementation Notes for this component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds a NULL-by-default lookup/populate callback seam on xce_sched_core_t,
mirroring cache_clear_cb's proven shape, so the allocation dispatch
chokepoint can consult the unsatisfiable-resreq cache without a
normal-mode-only field access in shared code. Adapters live in
alloc_cache.c and are registered only during normal-mode startup;
simulation mode leaves both callbacks NULL and always computes fresh.
Adds the scheduler-sim-live-allocation spec (requirements/design/
research/tasks) as the planning artifact this and subsequent tasks
implement against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Records the investigation and precise design before implementation: the
snapshot format already carries the original limit config (sim->core.config,
borrowed reference set by xce_sched_core_load_snapshot()), so the fix is
registration + a direct config_hook seed, mirroring sim_bootstrap_picker()'s
existing pattern exactly -- not a recording/replay format change. Also
documents the separate, deliberately out-of-scope finding that limit
*matching* during replay needs a further, larger fix (plugin_job_election()
is normal-mode-only today).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Tasks 2 and 3 are container-only major headers (per tasks-generation.md's
hierarchy rule); all their executable sub-tasks (2.1-2.3, 3.1-3.2) are
already complete. Bookkeeping only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified while closing out scheduler-sim-
diagnostics task 5: register_limit() has exactly one call site in the whole
codebase (normal-mode sched_register_plugins()); sim mode's own plugin
bootstrap (sim_bootstrap_picker()) only registers the picker plugin. The
info limit command built across this spec's 5 tasks is correctly
implemented and thoroughly tested but currently inert for a real operator
running xce_scheduler -i/-s <replay> -- it will always report "no limits
configured" regardless of the original recording's actual configuration.
Not fixed -- out of task 5's test-only boundary; recommended as the next
piece of work for this open-ended spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Four new tests in test_repl_limit_table.c close out Requirement 5:
- test_info_limit_never_mutates_global_limit_config_or_accounting_state /
..._host_scoped_... : snapshot a matched limit's config fields and its
vres capacity/in-use zarray entries (both pointer identity and value)
before and after repeated info limit calls; asserts byte-identical state
and byte-identical repeated stdout output.
- test_info_limit_on_empty_registry_is_repeatable_and_never_registers_plugin:
confirms info limit never causes the plugin to become spuriously
registered as a side effect of being queried.
- test_repl_step_continue_with_info_limit_interleaved_matches_baseline_decision_sequence:
a step/continue sequence run twice (once with info limit interleaved, once
without) produces an identical decision sequence.
Verified via real mutation testing: three targeted fake mutations were
temporarily applied to repl_info.c, confirmed each new test fails exactly as
predicted, then cleanly reverted (repl_info.c shows zero diff from HEAD).
Zero production code changed -- this is a test-only validation task. Full
14-binary regression sweep + 78/78 meson aggregate confirm zero impact on
any pre-existing info subcommand (Requirement 5.3).
This closes out scheduler-sim-diagnostics' first capability (limit
inspection, Requirements 1-5) end to end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_cmd_info() now recognizes "limit" as an info subcommand, routing to the
already-implemented and tested repl_info_limit() with the same ntok<3 usage
guard shape as the queue/worker/resource branches. Usage string and help
text updated to list it. repl_info_limit()'s own body is untouched -- this
is a pure, additive dispatch-wiring change. Two new tests drive it through
the real repl_dispatch() entry point (not a direct call) for both `info
limit <name>` and `info limit all`.
`info limit <name|all>` is now reachable from a live REPL session --
scheduler-sim-diagnostics' first capability is functionally complete.
Remaining: task 5 (determinism/regression validation).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_print_limit_usage_per_host() fans a single host-scoped xce_vres_t out
across every worker with a populated capacity entry at that vres's shared
resource index (w->resvals[idx]/w->resInUse[idx]), rather than constructing
any per-host vres object -- matching design.md's (xce_vres_t, worker) pair
cardinality model exactly. A worker with no populated capacity entry produces
no line at all (not a fabricated zero); a worker with capacity but no
charged usage shows accounted=0. Uses the corrected idx<0 + .caps-only
bounds-check pattern task 3.1 established, avoiding bug-1824's latent
imprecision. This closes out all of Requirement 3 (3.1-3.4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified (implementer + reviewer, both reading
src/lib/zarray.c directly) while implementing scheduler-sim-diagnostics task
3.1. Not fixed -- out of that task's scope; its own new code avoids the
mistake rather than touching the pre-existing function.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now displays accounted-usage/capacity for GLOBAL/SHARED
limit instances: iterates limit_get_usage_instances(), filters by
vres->limit == matched_limit, reads capacity from core.sharedInst[idx] and
accounted usage from core.sharedInUse[idx]. Existence-of-instance and
zero-usage are kept distinct per design.md: no matching instance (or one
with no populated capacity) prints "no active usage instance yet"; a
matched-but-never-charged instance correctly shows accounted=0 instead.
Self-caught during implementation and independently confirmed by review: a
negative or stale vres->idx (possible per add_vres()'s allocate-then-assign
pattern) is not safely rejected by zarray_get_ex()'s idx>=caps check alone
-- the new code adds an explicit idx<0 guard rather than copying
repl_info_resource_one()'s existing .count-based pre-check (which is itself
imprecise for a sparsely-populated zarray, a pre-existing issue left
out of scope here).
Calling limit_get_usage_instances() for the first time from repl_info.c
reproduced the same ~24-file link-stub ripple task 2.1 established for
limit_get_configured_limits() -- fixed with the identical pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints an explanatory note when a limit is implicitly
host-scoped (is_host_based_limit() > 0 due to a percent-based value on a
non-shared resource) but its own configuration doesn't show an explicit
per-host scope. The real discriminator turned out to be consumers[CONSUMER_
HOST]->per_consumer, not pointer-nullity as design.md's gloss said --
build_consumer() unconditionally allocates a consumer for all four
dimensions, so the struct field is never actually NULL in production. Fixed
using the same field is_host_based_limit() itself already falls back to;
verified via RED-phase reproduction that the literal design.md condition
never fires. design.md corrected to document the real condition.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() now prints a matched limit's configured setting on both the
all and named-lookup success paths: fmt_limit() (existing libxce formatter,
unmodified) into a stack buffer, re-split and printed through a new
repl_print_limit_dump() helper matching this file's existing [sim]-prefix-
per-line convention. Inclusion and exclusion group membership both render
since fmt_group() already handles both, adopted as-is per design.md's
explicit reuse decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
repl_info_limit() (repl_info.c, not yet wired into repl_cmd_info() -- that's
task 4) resolves "all" vs a specific name against limit_get_configured_limits():
a NULL/empty handle on the all path reports "no limits configured" (Req 1.7);
a NULL handle or key_find() miss on the named path reports "no such limit
'<name>'" (Req 1.3) and never collapses into the all-only message. New
white-box test_repl_limit_table.c exercises both paths directly, including a
dedicated regression guard for the NULL-scoping rule.
Making repl_info.c genuinely call limit_get_configured_limits() broke the link
for ~24 other white-box test files that #include repl_info.c without also
linking plugins/limit.c -- fixed with the same link-satisfying-stub idiom this
file family already uses elsewhere (all return NULL, verified behavior-
preserving since none of these fixtures register the limit plugin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
limit_get_configured_limits()/limit_get_usage_instances() cross the limit
plugin's private-state boundary (plugin->data, previously unreachable outside
limit.c) via key_find on the name-keyed plugin registry, returning NULL when
the plugin is absent/uninitialized and a raw iterable kvmgr handle otherwise.
Pure reads only -- no allocation, mutation, or hook triggering. Foundation for
the upcoming `info limit` REPL command (task 1 of 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Requirement set 1 (info limit <name|all>) of the open-ended scheduler-sim-diagnostics
spec: requirements refined for effective vs. configured scope and exclusion-group
membership, design grounded in the actual limit-plugin/REPL source (accessor boundary,
GLOBAL/SHARED-vs-per-host read split, existence-vs-zero-usage distinction), and an
8-subtask implementation plan with full requirements traceability.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1820's own fix assumed sudo always forks exactly one direct child
that IS the real target, and looked for it via `pgrep -P $PID` (direct
children only). On a real host, sudo commonly interposes an additional
"monitor" process between the invoked sudo and the actual target
(sudo -> monitor -> target, three processes, not two -- standard
sudo 1.8.7+ architecture for signal/exit-status relay) -- a shape this
sandbox's own simpler sudo build doesn't reproduce, which is why
bug-1820 verified clean here but still left the real worker orphaned
on the reporting user's box. `pgrep -P` found and killed the monitor
(a direct child) while the actual worker (the monitor's own child, not
a direct child of the pidfile'd PID) survived untouched.
Replaces the parent/child tree-walking approach entirely with a
depth-independent identity-based re-scan: both generated scripts now
search system-wide (pgrep -f) for any still-running process whose own
cmdline matches the worker's distinguishing identity (-n $NAME plus
this deploy's -l $TOP/logs -- the same two args start-worker.sh's own
"already running" guard checks) and kill every match directly, however
many hops deep it landed. No longer needs to know or guess sudo's
internal process-tree depth.
Live-verified: regression-checked against this sandbox's own real
(2-level) sudo deploy (still clean), and built + killed a synthetic
3-level process tree modeled on the user's exact live report,
confirming the deepest process is found directly and the intermediate
hop then exits on its own once its child is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.
Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.
Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.
TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.
Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.
Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Two independent live 3-node-cluster runs both confirmed the full
Requirement 13 scenario (healthy all-ok, dead-on-kill, unknown-from-
standby-view) with zero bugs found. No source changes -- pure
validation task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds ctrlstatus_peer_status_str (mirrors ctrlstatus_role_str's
convention: 0=unknown/1=ok/2=catching-up/3=dead, safe default fallback
for any other code) and extends cmd_ctrlstatus's per-controller print
line to render it alongside each configured controller's existing
name:address, completing the CtrlPeer.status wire field added in task
8.1 and sourced in task 8.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Populates CtrlPeer.status (added in task 8.1) inside ctrlstatus_fill's
peer-building loop: self always reports ok; a primary responder sources
every other peer from the live replication connection registry
(replicator_t.standbys[], host+port matched, reusing
repl_active_standby_count's ep!=NULL && !in_catch_up predicate) --
dead/catching-up/ok; a non-primary responder reports unknown for every
non-self peer. A self-identity mismatch (pre-existing misconfiguration)
logs one WARN and falls through to the normal non-self sourcing rather
than forcing any peer to ok. Read-only: a bounded lock-protected scan,
no state mutation.
TDD: 3 new tests (primary-sourced-from-registry, standby-unknown-except-
self, self-mismatch-falls-through) plus an extended no-mutation
assertion; RED->GREEN independently reproduced by the reviewer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds CtrlPeer.status (UInt8, next free ordinal @3 after port) plus the
matching xce_ctrlpeer_t.status typemap field, verified field-for-field
against the regenerated codec with a genuine encode/decode round-trip
test. No sourcing/rendering logic yet (task 8.2/8.3) -- field defaults
to 0 (unknown) until wired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Second design-review pass caught a factual error: the Constraints
section claimed ctrlstatus_fill "already takes ha->lock" for the
primary-identity read, citing the ORIGINAL Req 1-9 design text rather
than the as-built code -- after the ade0081 fix, that read takes no
lock at all. Corrected: this delta's replicator_t.lock acquisition is
the first lock ctrlstatus_fill has taken since that fix, not a repeat
of an existing in-function pattern (still safe, per the existing
repl_active_standby_count precedent elsewhere in the codebase).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds requirements, design, and research for showing each configured
controller's connection status (ok/catching-up/dead/unknown) in
ctrlstatus's peer list, with the same off-primary fidelity pattern
already established (primary reports the true picture; a standby
reports unknown for what it cannot itself observe).
Design review found and fixed one gap before implementation: no
defined fallback if the responder's own name doesn't match any
configured peer entry (a pre-existing misconfiguration class, not new
to this delta) -- now an explicit, logged (WARN) degradation instead
of a silently wrong status.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
