core_clock.c joined the mechanically-enforced isolation archive
(sched_core_isolation_lib) and its own doc comment claimed coverage by
the forbidden-headers/forbidden-symbols text scans, but was actually
absent from both -- only the nm(1) undefined-symbol layer covered it,
which cannot catch a forbidden #include resolving to no undefined
symbol (a macro-only header).
Added XCE_CORE_CLOCK_C_PATH (meson.build, mirroring the existing
per-file path macros) and extended both
test_core_never_includes_forbidden_headers() and
test_core_never_calls_forbidden_symbols() to scan core_clock.c. It
remains excluded only from the wall-clock scan, for its one sanctioned
clock_gettime() call.
TDD: confirmed RED by temporarily inserting a forbidden #include into
core_clock.c (84/85, correctly attributed failure), reverted
(byte-identical to HEAD), reconfirmed GREEN (85/85). No live violation
existed -- this closes a coverage gap in the mechanism itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Same pattern as bug-1492/1682/1611: tags claimed "fixed"/
"closed-by-scheduler-sim-live-allocation" while the fix narrative said
"NOT FIXED" -- and correctly so, that feature never touched
test_core_isolation.c. Removed the incorrect tags; entry remains open.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.
The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.
Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
/xce-audit-invariants scheduler (dimension 1, driver state-equivalence,
re-run after scheduler-sim-live-allocation) found plugin_end_session()
still checked `plugin->begin_session_fn != NULL` before invoking
`plugin->end_session_fn(plugin)` -- a copy-paste of
plugin_begin_session()'s own (correct) guard. Two failure modes: a
plugin registering only end_session_fn silently never has it invoked;
a plugin registering only begin_session_fn crashes via a NULL
function-pointer call. Currently inert in production (no live plugin
hits either shape) but a real latent defect in exactly the
session-boundary code the driver-equivalence invariant governs.
This exact defect had been independently filed twice: bug-1492
(2026-08-04, scheduler-core task 5.2's review) and bug-1682
(2026-08-24, found adjacent to bug-1681's fix) -- neither investigation
cross-referenced the other. Both entries also carried a "fixed" tag
inconsistent with their own "not yet fixed" narrative text, caught by
this audit's independent code+text cross-check.
Fixed the one-line guard; added test_end_session_invokes_end_session_fn_
when_only_it_is_set() (test_plugin_hook_dispatch.c) proving the silent
non-invocation, confirmed RED (84/85) against the buggy guard and GREEN
(85/85) after the fix. Updated both buglog entries to FIXED with
cross-references, and corrected bug-1611's stale narrative to reflect
its narrowing (not elimination) by scheduler-sim-live-allocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Adds the feature's concrete original acceptance test: a genuine
capnp-encoded .replay (real ReplayHeader/SnapshotEntry/Conf4Sched with
one global and one host-scoped limit), driven end-to-end through the
real, unmodified sched_sim_driver_run(). The dispatched job's usage
registers against both limits via limit_get_configured_limits()/
limit_get_usage_instances() -- the same accessors info limit reads --
proving real, non-zero usage where today's REPL previously showed none.
RED-phase evidence: reverting task 4.3's 4 files to their pre-fix
commit reproduces the exact predicted failure (host-scoped usage
missing, global usage intact); restoring to HEAD reproduces GREEN.
Closes bug-1826 (the bug this whole feature was built to fix) --
buglog entry updated from NOT FIXED to FULLY FIXED, citing this test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.
Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.
One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.
TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.
Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.
That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.
TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Migrates match_job_global_limit_()/limit_job_election()/
match_job_host_limit_() (plugins/limit.c) to read plugin->core (added
task 1.2) instead of deriving xce_scheduler_t* from plugin->sched,
which stays NULL for a plugin registered via plugin_register_core()
(simulation mode). Removes the dead ctx.sched write flagged during
task 2.1's review -- verified this ctx never has .sched read anywhere
downstream.
Fixes bug-1828 (found while writing this task's own TDD test for a
previously untested code path): match_job_host_limit_() had the same
key_find()-negative-rc-treated-as-fatal defect bug-1786 already fixed
in its global-limit sibling, but never applied here -- meaning
host-scoped resource limits were never actually enforced, in any
mode, until this fix (not merely a display gap; the limit's vres was
never added to obj->hlimits, so sched_fill_limit_requests() always
saw an empty list for it).
TDD: new tests drive the real hook-registered entry points against a
plugin_register_core()-registered plugin, asserting real usage
instances via limit_get_usage_instances() for both global and
host-scoped limits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
info limit was correctly implemented and tested but inert against a real
replay: register_limit()'s only call site was normal-mode sched_register_
plugins(); sim mode's own plugin bootstrap (sim_bootstrap_picker()) never
registered the limit plugin. The snapshot format already carries the
original session's limit config (sim->core.config, a borrowed reference set
by xce_sched_core_load_snapshot()) -- nothing needed to change in the
recording/replay format.
New sim_bootstrap_limit_plugin() (sim_limit.c, mirroring sim_picker.c's
structure) registers the plugin via plugin_register_core() and seeds it via
a new limit_seed_config(plugin, xce_sched_core_t* core, config) entry point,
wired into sim_loop.c alongside the existing sim_bootstrap_picker() call.
limit_new_plugin() extracted from register_limit() as a factory, mirroring
builtin_fcfs_new_plugin()'s established pattern; register_limit() itself is
behavior-preserving.
First attempt set plugin->sched to a disguised xce_sched_sim_driver_t*,
relying on it sharing xce_scheduler_t's first-member layout to reuse the
existing config_hook path -- rejected on review: this repeats the exact
cross-cast pattern that already corrupted memory once in this codebase (see
core.h's xce_core_cache_clear_cb_t history). Fixed by making
limit_seed_config() take xce_sched_core_t* as an honest parameter instead,
with limit_update_config() (normal mode) and the new sim path both
delegating to a shared limit_seed_config_impl() -- no cast, no aliasing risk,
independently verified by two review rounds reading the actual struct
layouts and call chains.
Also added sim_limit.c to sched_sim_isolation_lib so Req 5 AC 17's
controller-I/O-isolation check covers it (an initial gap, closed in the same
remediation round). Two ripple waves of the same established link-stub
pattern applied to the ~24 white-box test files that #include sim_loop.c.
info limit now shows real configured-limit data against a genuine replay.
Live usage will still correctly show "no active usage instance yet" until a
separate, larger fix (logged as bug-1826) wires limit *matching* into the
sim decision path -- deliberately out of this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified while closing out scheduler-sim-
diagnostics task 5: register_limit() has exactly one call site in the whole
codebase (normal-mode sched_register_plugins()); sim mode's own plugin
bootstrap (sim_bootstrap_picker()) only registers the picker plugin. The
info limit command built across this spec's 5 tasks is correctly
implemented and thoroughly tested but currently inert for a real operator
running xce_scheduler -i/-s <replay> -- it will always report "no limits
configured" regardless of the original recording's actual configuration.
Not fixed -- out of task 5's test-only boundary; recommended as the next
piece of work for this open-ended spec.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Discovered and independently verified (implementer + reviewer, both reading
src/lib/zarray.c directly) while implementing scheduler-sim-diagnostics task
3.1. Not fixed -- out of that task's scope; its own new code avoids the
mistake rather than touching the pre-existing function.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
bug-1820's own fix assumed sudo always forks exactly one direct child
that IS the real target, and looked for it via `pgrep -P $PID` (direct
children only). On a real host, sudo commonly interposes an additional
"monitor" process between the invoked sudo and the actual target
(sudo -> monitor -> target, three processes, not two -- standard
sudo 1.8.7+ architecture for signal/exit-status relay) -- a shape this
sandbox's own simpler sudo build doesn't reproduce, which is why
bug-1820 verified clean here but still left the real worker orphaned
on the reporting user's box. `pgrep -P` found and killed the monitor
(a direct child) while the actual worker (the monitor's own child, not
a direct child of the pidfile'd PID) survived untouched.
Replaces the parent/child tree-walking approach entirely with a
depth-independent identity-based re-scan: both generated scripts now
search system-wide (pgrep -f) for any still-running process whose own
cmdline matches the worker's distinguishing identity (-n $NAME plus
this deploy's -l $TOP/logs -- the same two args start-worker.sh's own
"already running" guard checks) and kill every match directly, however
many hops deep it landed. No longer needs to know or guess sudo's
internal process-tree depth.
Live-verified: regression-checked against this sandbox's own real
(2-level) sudo deploy (still clean), and built + killed a synthetic
3-level process tree modeled on the user's exact live report,
confirming the deepest process is found directly and the intermediate
hop then exits on its own once its child is gone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xceadmin's --controller option (config push, and scheduler record
start/stop/status) accepted only a single host:port. Passing every
configured controller as a comma-separated list -- the natural way to
query an HA cluster without knowing which node is alive/primary --
mis-parsed the whole joined string as one bogus hostname and failed
with a DNS-resolution error.
Adds xceadmin.discovery.resolve_controllers() (comma-split candidate
parsing) and follow_redirects_any() (tries each candidate in order,
skipping ones that are unreachable at the network level -- connection
refused, DNS failure, or a discovery timeout -- mirroring xcecli/
xce_test's own client-side master-rotation failover). A candidate that
IS reachable but returns a genuine protocol-level failure is not
retried against a different node. push.py and scheduler.py's own
duplicated single-host _split_controller helpers are removed in favor
of this shared implementation.
Also fixes a related gap bug-1821's own live verification surfaced:
right after a primary is killed, a standby doesn't start a new
election until its own heartbeat timeout elapses (~10s), so it keeps
redirecting every candidate to the now-dead old primary -- exhausting
the whole candidate list and raising NoPrimaryError, which
_send_admin_request()'s existing transient-retry loop (bug-1713/1715)
never caught. That loop now retries on NoPrimaryError too, using its
own (longer) budget sized for the heartbeat-timeout-plus-election
window rather than the RPC-level "already electing" budget the
original retry logic was tuned for.
TDD RED->GREEN, 304/304 tests passing. Live-verified end to end: a
comma-list correctly skips a dead first candidate and reaches a live
one on a healthy cluster, and correctly rides out a real primary kill
and re-election (retry loop visibly cycling ~15s) to reach the newly-
elected primary rather than failing immediately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
start-worker.sh's real-worker branch writes the pidfile via `echo $$`
then `exec sudo "$WORKER_BIN" ...` -- exec replaces the pidfile-writing
shell with sudo in place (same PID), so the pidfile records sudo's own
PID, not the actual xcecd daemon's. sudo forks a separate, root-owned
child to run the real worker; kill -9 on sudo's PID kills only sudo
(SIGKILL can't be forwarded to a child), silently orphaning xcecd. The
existing "still alive after kill -9" warning never caught this either,
since it re-checked the same (already-dead) sudo PID.
Both generated stop-all.sh and cleanup.sh now detect a sudo-wrapped
pidfile entry (its /proc/$PID/cmdline starts with "sudo "), find its
child via pgrep -P, and kill that child with sudo first. The
still-alive check now also tests for the child's continued existence.
Live-verified end-to-end: deployed a 3-node cluster with a real
(XCE_REAL_WORKER=1) worker, confirmed the sudo-parent/root-child PID
pair, ran the fixed stop-all.sh and cleanup.sh and confirmed zero
processes remain after each, with the new "is a sudo wrapper" line
printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Found live during task 8.5's optional catching-up-observation attempt:
a killed-then-reconnected standby can end up occupying a NEW
replicator_t.standbys[] slot while its OLD slot (host/port still
populated, ep == NULL) also matches the same configured peer.
ctrlstatus_fill's primary-side peer-status loop stopped at the first
host:port match, so it permanently reported that peer dead even
minutes after it had genuinely reconnected and fully caught up --
violating Requirement 13.3's "true, currently-known connection status"
guarantee.
Fix: scan all MAX_STANDBYS slots for a given peer and prefer any slot
with ep != NULL over a slot with ep == NULL; fall back to dead only
when no matching slot is live. Read-only, lock scope unchanged from
task 8.2.
TDD RED->GREEN (1216 Tests: 2 Failures -> 0), independently reproduced
by the reviewer (exact RED replay) plus two independent live
kill/backlog/restart/poll cycles on separate fresh clusters, both
confirming the previously-stuck standby now converges to ok.
design.md's Locked Decision 3 corrected to document the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.
The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.
test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
pthread_cond_wait() reacquires its mutex before honoring a cancellation,
but mylock.c installed no cleanup handler to release it again -- a
thread cancelled mid-wait terminated holding the lock forever. vtaskmgr's
pthread_cancel()-based teardown could land there from three angles
(dispatcher cancelled while polling the shared queue, or a vcpu racing
back into it after its own shutdown signal), permanently deadlocking any
other thread that touched the same queue and hanging vtaskmgr_fini()'s
pthread_join() indefinitely.
Reproduced live as xce_test hanging on Ctrl-D exit once a connection
context was entered; gdb confirmed a vcpu thread blocked forever on a
mutex owned by an already-exited thread ID.
Fix: cbuf_put()/cbuf_get()/cbuf_timedget() now unlock via a
pthread_cleanup_push/pop handler around the wait, so a cancelled waiter
always releases the mutex. vcpu_idle()/vcpu_entry() also recognize the
teardown poison pill and exit immediately instead of needlessly
re-announcing into the already-dead shared queue.
New test_vtaskmgr_teardown.c reproduces the race deterministically
(self-pinned to one CPU, since true multi-core parallelism rarely hits
the window) and bounds vtaskmgr_fini() with a timed join so a
regression fails fast instead of hanging the suite. Logged as bug-1810.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce_core_push_events() WARNed on every JOBS_PUSH resync redelivery
(sched_add_job rc=-EEXIST), which is the expected outcome under
on_scheduler_join()'s full resync or the array-push dedup race
(bug-1734/bug-1714) and flooded operational logs. Only that case now
logs via a new DBG() macro (utils.h); any other add_rc (e.g. -ENOMEM)
still WARNs. DBG is named to avoid colliding with the build's own
-DDEBUG command-line define (debug/asan buildtypes).
Also collapsed hash_string()'s duplicated djb2 loop to delegate to
hash_block(), which already implements the same RUN_HASH loop.
Logged as bug-1809 in .wolf/buglog.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Every cmd_* REPL handler in test.c/join_worker_args.c reset flag parsing
with `optind = 1;` only. glibc's getopt_long only fully reinitializes its
internal scan state (nextchar/first_nonopt/last_nonopt) when optind==0, or
on the process's very first getopt_long() call ever -- optind=1 alone
leaves that state stale for every call after the first. Since xce_test's
REPL dispatches many independent cmd_*() calls per process, a command
whose parse ends on a short no-argument flag (`qctrl -a`, `hostctrl -o`)
left state that corrupted the NEXT unrelated command's parse badly enough
to misread its own name as an unrecognized flag. This was the actual
cause of harness_unit_test's 4 deterministic (not flaky -- 20/20 identical
repeat runs) test_test_cli.c failures, each burning its full 5s
confirmation-prompt timeout. Fixed by using optind=0 at all 21 reset
sites; confirmed via 3 isolated pty repros plus 10 consecutive clean
harness_unit_test runs (132/132, was 4 failures/run every time).
Also fixes a resource leak found while investigating: test_test_cli.c's
spawned xce_test sessions were only torn down by inline
clitest_kill_and_reap()/clitest_close() calls at the bottom of each test
function, which Unity's longjmp-based TEST_ASSERT abort skips entirely on
failure, leaking the child process and its pty/pipe fds. Added a static
session mirror (safe across the longjmp, unlike a stack pointer) reaped
from harness_test.c's tearDown(), which Unity guarantees runs after every
test regardless of outcome.
Logged as bug-1807/bug-1808 in .wolf/buglog.json, with the underlying
getopt/longjmp gotchas recorded in .wolf/cerebrum.md. Full ASAN
/xce-build harness also passes 5/5 with no sanitizer findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
core_reset_workers already does (bug-1751), closing the gap that let a
worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
a captured job's job->state is set to RUN synchronously at dispatch capture,
mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
live feedback that it over-fires inside busy sessions; replace it with a
prescan-based "next non-dummy session" hint in info summary instead
Full scheduler suite: 77/77 tests pass (asan profile).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Follow-up audit (requested: check other generated sources, specifically
packcc/PEG output) found the identical bug shape in the four PEG-parser
custom_targets: section_parser, rusage_parser, xce_resreq_parser, and
xce_select_parser each produce a .c + a paired .h (section.h, rusage.h,
xce_resreq.h, xce_select_peg.h), but libxce_src only referenced each
parser's [0] (.c) index. lib/resreq.c -- confirmed via a project-wide
grep to be the only consumer of any of these four headers -- had no
declared build-order dependency on any of the four codegen steps.
Fix: pass each of the four custom_targets unindexed into libxce_src,
same pattern as the capnp fix. No libxce_dep change needed here since
nothing outside src/lib/ consumes these headers.
A project-wide search confirms these, plus the capnp custom_target, are
the only multi-output C/H codegen steps in the whole build -- every other
custom_target() (xceadmin, apiserver, xcecli, autoscaler/providers)
produces a single Go-binary/stamp output with no equivalent hazard.
Verified: a third consecutive full from-scratch `./tools/xce-build.sh all
debug` build (1173/1173 targets) succeeded clean; a fresh `lib debug`
build plus the full lib test suite (313/313) passed; a fresh incremental
controller/ASAN rebuild passed 1331/1331 on a quiet re-run (its first,
concurrent-with-the-full-build run hit an unrelated, confirmed-flaky PTY
harness timing failure in test_smoke_launch_send_echo_capture_stdout_and_exit_code,
reproduced as system-load-induced, not caused by this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
The previous fix (654c6e8c) only closed the missing-dependency-edge race
inside the libxce target itself. A live user report on a second
from-scratch `./tools/xce-build.sh all debug` run showed the same race
still hitting scheduler/src/sim_record.c and core_events.c, which reach
controller.capnp.h directly via a raw -Ilib include path with no
declared dependency on the codegen step at all.
Root cause: libxce_dep = declare_dependency(link_with: libxce) -- the
dependency object controller/worker/scheduler all consume uniformly --
only orders a consumer's LINK step after libxce is built; it never
propagated a build-order dependency on the capnp custom_target to a
consumer's own COMPILE steps.
Fix: add `sources: [gen_srcs]` to libxce_dep's declare_dependency() call.
This kwarg exists specifically for generated files that must be built
before any consumer of a dependency compiles, and since every component
already lists libxce_dep in its own dependencies, this one change
propagates the missing edge everywhere with no per-component changes.
Verified: two consecutive full from-scratch `./tools/xce-build.sh all
debug` builds (1173/1173 targets, the user's own exact repro command)
both succeeded with no retry; a subsequent incremental controller/ASAN
rebuild stayed clean (1331/1331 tests, 0 ASAN diagnostics).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
libxce_src listed only gen_srcs[0] (controller.capnp.c) as a source of
the static_library('xce', ...) target, never controller.capnp.h -- which
several library files (utils.c and others) reach only via #include.
Ninja had no build edge forcing the capnp custom_target to finish before
those files compiled, so a from-scratch build could schedule them first,
intermittently failing with "controller.capnp.h: No such file or
directory". This is the same symptom recorded across 5+ prior recurrences
in buglog.json since 2026-07-16, previously worked around by retrying the
build rather than root-caused.
Fix: pass the whole gen_srcs custom_target (not gen_srcs[0]) into
libxce_src -- Meson still compiles only its .c output, but now orders the
entire target's build after both outputs are generated. Verified via two
consecutive from-scratch `xce-build.sh lib debug` builds (no retry
needed), the full lib test suite (313/313), and an incremental controller
asan rebuild (1331/1331, 0 ASAN diagnostics) to confirm no regression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Live-cluster validation, no source changes: independently reproduced
(implementer + reviewer, separate deploys) that a destructive command
inside a sourced file, run within an otherwise-interactive session,
still prompts for confirmation with wording identical to a directly-typed
invocation, is skipped once the session toggle is set, and that decline
does not trip set -e's abort-on-failure (Req 13.7). Structurally confirmed
the gate has no code path capable of distinguishing a sourced line from
a typed one. Bumps bug-1797's occurrence count (recurred via the source
call path, confirmed pre-existing, not new).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Live-cluster validation, no source changes: independently reproduced
(implementer + reviewer, separate deploys) that all five destructive
commands prompt by default, treat decline and real-SIGINT identically as
"not run" (not "failed", the deliberate Req 9.9 exception to Req 1.3),
skip the prompt under --yes/non-interactive/session-toggle, and match
Req 9.5's exact enumeration including its reversal-action exclusions.
Logs bug-1796 (misleading perror() text from stale errno in
cmd_qctrl/hostctrl/jobkill) and bug-1797 (cmd_qctrl queue-name leak on
the RPC-error path) as pre-existing, deliberately-deferred findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Routes all 26 blocking xcecli_* call sites across ~24 commands through
xcetest_run_interruptible(), wires xcetest_set_active_connection() at
execute_line()'s dispatch point (active_ctx->ctx, read fresh per
command), and fixes main()'s REPL loop to distinguish a real EOF from
an EINTR-interrupted readline() so Ctrl-C at an empty prompt discards
the line instead of exiting the session.
Live-discovered and fixed bug-1794: fill_job_alloc() never initializes
xce_job_alloc_t's nres_shared/allocs_shared fields -- harmless as stack
garbage before this task, genuine heap garbage once moved into a
heap-allocated ctx block (a necessary consequence of the adapter's
heap-only precondition). Fixed by switching all ~26 call-context
allocations from malloc to calloc uniformly. Also fixed an argv-aliasing
hazard: cmd_hello/cmd_jobckpt's -m values aliased optarg, which
execute_line() frees immediately after return including on the
abandoned path -- now strdup'd into the heap block.
Also closed a safety-critical gap this task's own design introduced
(bug-1795): before this task the REPL was fully synchronous, so
leave_ctx() (called by up/quit) could never run while an RPC was
outstanding. An abandoned worker can now persist past command
completion, so leave_ctx() tearing down the connection could
use-after-free it. Fixed via xcetest_wait_connection_idle(), which
blocks on the same per-connection semaphore the adapter already
tracks, then releases it, proving no worker is active before teardown.
Proven with a deterministic white-box RED->GREEN test, since the race
itself is too timing-dependent for a black-box test to reliably force.
1361/1361 across the full controller suite (+2), 0 regressions.
Task 3.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.2, 1.3, 1.4, 1.5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Migrates 17 commands (cmd_hello, cmd_jobinfo, cmd_jobrun, cmd_jobsub,
cmd_resval, cmd_jobkill, cmd_jobckpt, cmd_qctrl, cmd_hostctrl,
cmd_runtasks, cmd_setpend, cmd_jobstart, cmd_jobfinish, cmd_jobfail,
cmd_resupdate, cmd_walcompare, join_worker_parse_attrs) from raw
getopt()/hand-rolled parsing to the shared flagspec_t + getopt_long
mechanism, so every one now uniformly rejects an unrecognized or
malformed flag with a usage summary and, where a close match exists,
a suggestion. cmd_walcompare's hand-rolled parser needed a leading '+'
in its optstring to preserve its pre-migration flags-only-before-
positionals behavior. Also fixes bug-1704: cmd_qctrl's uninitialized
action variable, now sentineled and rejected when unset, mirroring
cmd_hostctrl's existing pattern -- this also makes the destructive-
command gate's -i/-c detection (task 3.3) safe to rely on. Every
existing short flag's letter, value format, and default is unchanged
for a valid invocation. 1357/1357 across the full controller suite
(+4), 0 regressions.
Task 3.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4, 10.1, 10.2, 10.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
cmd_join_worker's simulated-worker maxmem default was 64 bytes -- a
leftover from when the field was MB-scale, unfixed when maxmem became
bytes-canonical elsewhere. Now 64 * 1024 * 1024. Extracted the
argument-parsing/defaulting logic into join_worker_parse_attrs()
(join_worker_args.c), since it runs entirely before the function's
network call and is a genuine testable seam -- same precedent as
task 2.5's joblist_display.c. Verified live against a real deployed
cluster (workerlist reports MaxMem: 64.00 MB) plus a permanent 5-test
regression suite. maxswap/maxtmp have the identical stale-default bug
(logged as bug-1793) but are deliberately left unfixed here since
USAGE_JOIN_WORKER's help text currently documents their stale values
accurately -- fixing only the code would create a new docs mismatch;
a locked-in sibling-defaults test documents this for the follow-up.
1353/1353 across the full controller suite (+5), 0 regressions.
Task 2.6 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 12.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Registers libedit's GNU-readline-compatible rl_attempted_completion_function
under g_interactive only. The candidate generator reads active_ctx->commands[]
live on every call, so completion correctly tracks context switches rather
than freezing at registration time. Single/multi-match UI (auto-complete vs.
double-Tab candidate listing) is entirely libedit's stock behavior once given
a correct context-scoped generator. Proven via a black-box pty test verified
sound against a false-positive risk: xce_test has no fuzzy command-name
matching, so an uncompleted partial token cannot coincidentally trigger the
same effect a real completion would. Also fixes a harness bug (bug-1792):
disabling kernel ECHO for output capture was silently suppressing libedit's
own completion feedback too; added clitest_spawn_ex() with disable_echo as
an explicit parameter, clitest_spawn() unchanged for existing callers.
1307/1307 controller suite passing (+1), re-verified 8/8 for flakiness.
Task 2.3 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 7.1, 7.2, 7.3, 7.4
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
A registry of per-connection locks, acquired (interruptibly) before
spawning a worker and released by whichever side finishes last, so an
abandoned call's worker is never running concurrently with a new call
on the same connection. Uses a POSIX semaphore (sem_t), not a mutex:
review round 1 found the initial pthread_mutex_t implementation relied
on behavior POSIX classifies as undefined for a non-owning-thread
unlock (not just error-checking mutex types) -- this design's worker
thread legitimately releases a lock the caller thread acquired, which
sem_t supports with no ownership concept at all. bug-1791. Design.md
synced to reflect the semaphore choice. 1285/1285 controller suite
passing.
Task 1.5 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Captured simulator REPL session findings from bug-1790 verification:
incorrect maxswap resource value display and a suggestion to show
per-worker slot allocation counts in `info worker` output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Adds ug1_global_mem to limits.yaml: a permanent, real GLOBAL limit on
"mem" (a per-host resource, not "slots"), capping ug1's total
cluster-wide mem usage at 5GB. This is exactly the scenario bug-1790
fixed but only unit-tested until now -- no limit in the shipped example
config exercised a global cap on a non-slots per-host resource.
Live-verified via a fresh single-node deploy: 3 jobs submitted as user1
(ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue
normal. Scheduler trace confirms ug1_global_mem correctly matches and,
for the first time, computes a real nonzero job-wide-total wanted value:
unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000
uncomit 0.000000 wanted 2000000000.000000
unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall
1000000000.000000
The 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) correctly PENDed
until an earlier job's mem freed up. Deploy config validated with zero
new warnings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
bug-1743 (gitea#46's original tracking entry) still said Part 1
(dispatch-stops-at-8) was "UNRESOLVED/NOT ACTIONABLE, pending reporter
info" -- stale since the 6-bug limit-plugin cascade (bug-1784 through
bug-1790) fully resolved it without needing that info, and gitea#46 was
just validated live and closed. Updates the entry with the full
resolution chain and the closing live-verification evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Closes the gap bug-1789 explicitly carved out: a GLOBAL limit built on a
per-host resource other than "slots" (e.g. a future absolute global mem
cap) needs ctx->alloc_req.host to carry that resource's job-wide total
want during the global phase, but only "slots" ever got that treatment
(via seqjob/parjob_prepare_allocation()'s own unconditional store).
Adds calc_wanted_host_resval_job_totals() (resreq.c), sharing its loop
body with calc_wanted_global_resvals() via a new static helper
parameterized only by which XCE_RESF_GLOBAL direction to match -- it
computes the SAME job-wide-total formulas (PER_TASK: value*nslots, PER_JOB:
value, PER_HOST: value*nhosts) for non-global resources instead of global
ones. seqjob/parjob_prepare_allocation()'s global-phase branch now calls
it into ctx->alloc_req.host alongside the existing calc_wanted_global_
resvals() call into .shared.
Safety was traced, not assumed: plugin_prepare_allocation() only resets
.host at the top of a per-host call, never during the global phase, and
sched_fill_limit_requests() for glimits reads .host later in the SAME
global-phase invocation -- transferring the value into the limit's
permanent .shared record before any later per-host call ever resets
.host. An earlier candidate fix (reusing calc_wanted_host_resvals()
directly) was rejected during design: its PER_HOST formula and, for
parjob, its per-bucket nslots don't give a job-wide total.
New regression tests in test_seqjob_allocator.c/test_parjob_
combinatorial.c prove a non-"slots" per-host resource ("mem") now gets
correctly populated; parjob's test uses maxTasks=3 specifically to prove
the formula uses the job's whole nslots, not a bucket's share.
Full suite (xce-build.sh all asan --incremental): 92/94, both failures
pre-existing and unrelated (bug-1762, bug-1733's already-documented
xcealloc.c leak). Not live-verified -- no limits.yaml entry exercises
this path yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
meson/ninja/capnp/Go only exist inside the hermetic nix dev shell, but
xce-build.sh never entered it itself -- the /xce-build skill documented
wrapping every invocation in `nix develop ./src --command ...` by hand,
so a bare `bash src/tools/xce-build.sh ...` (run directly instead of
through the skill) failed opaquely with "meson: command not found". This
was a recurring, previously-logged gap that resurfaced again today mid
bug-1789 verification.
The script now checks IN_NIX_SHELL at the top and re-execs itself under
`nix develop` if unset, so every caller -- the skill, a bare invocation,
CI, a human at a terminal -- gets it automatically. A nested call (already
inside the shell) is a no-op past that check. README.md's daily-develop
section updated to match; the old "or use nix develop --command" example
is now redundant and removed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
sched_fill_limit_requests() used to read a limit's underlying resource's
want from the same vector its own virtual-resource entry gets written to.
That's wrong whenever the two differ in scope -- e.g. a GLOBAL limit built
on top of a per-host resource like "slots" (limits.yaml's
ug1_normal_slots): the limit's own vres entry belongs in alloc_req->shared,
but "slots"'s want is only ever staged into alloc_req->host by
seqjob/parjob_prepare_allocation(), so the old code always read a zero and
silently never enforced the cap.
Now both the read source and write destination are derived independently
from each resource's own XCE_RESF_GLOBAL flag (vres->res for the source,
vres->vres for the destination) -- the same flag
calc_wanted_global_resvals()/calc_wanted_host_resvals() (resreq.c) already
dispatch on, so this is the existing rule applied consistently rather than
a new one. Also drops sched_fill_limit_requests()'s unused xce_scheduler_t*
parameter.
Full scheduler suite: 76/77 (the one failure, test_job_rusage_reconciles_
mem_reservation_against_usage, is bug-1762, pre-existing and unrelated).
Live-verified against a fresh single-node deploy.
Filed bug-1790 to track a known, deliberately out-of-scope residual gap:
a future global limit on a per-host resource other than "slots" would need
calc_wanted_host_resvals() to also run during the global phase for that
resource, which nothing currently does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
While live-verifying gitea#46, found and fixed a cascade of 5 previously
undetected bugs that together made every usergroup-scoped limit in
limits.yaml silently non-functional:
- Named usergroups/hostgroups (users.yaml/hosts.yaml) were never
registered into the scheduler's group managers (groupmgr_add() had zero
production callers), so any limit referencing one by name could never
resolve real membership.
- An unconfigured limit scope (e.g. no "projects:") arrived as an
empty-but-present capnp group, not NULL, so build_consumer() built a
real empty group (matching no one) instead of falling back to "all".
- Three separate call sites conflated key_find()'s negative "not found"
return with a real error, short-circuiting before ever reaching the
"instantiate on first use" logic right below them -- the same defect
class already fixed once before in this codebase (bug-1495).
- Two call sites passed &(sched->core.resmgr) where a plain handle was
expected, one level of indirection too many, causing RTTI's key_find()
to reject it as an invalid handle.
All five are live-verified: fresh cluster deploys now correctly match and
instantiate usage tracking for all 7 limits in limits.yaml on every job
submission, reproduced across multiple independent from-scratch redeploys.
A 6th, architecturally distinct issue remains open (bug-1789, logged, not
fixed): sched_fill_limit_requests() computes a job's consumption of a
slots-scoped global limit by reading the wrong request vector, so the
numeric cap itself still never binds. This needs a design decision on
intended cross-scope semantics, not a mechanical fix, so it's left for a
follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
ControlJob() set xce_jobctrl_t.action to the job_ctrl_type_t action code
(kill/stop/resume/checkpoint), but that field is actually the capnp
JobControl_u_which union discriminant -- a different enum entirely. This
made kill/resume/checkpoint hit request.c's -EINVAL default case, and made
stop collide with the checkpoint discriminant and read through the wrong
union member. Live-verified against a real deployed cluster: kill/stop/
resume/checkpoint now behave correctly with no apiserver crash.
Also additively accept a human-readable job ID ("{cluster}-{apiserver}-
{timestamp}-{seq}") on the job control endpoint, alongside the existing
numeric form xcecli already translates to -- lets a caller hit the REST
API directly without reimplementing xcecli's ID packing. Live-verified via
a direct curl call with the human-readable ID.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
