collect_job_rusages() (xcecd_cgroups.c) reads CPUUsageNSec/MemoryCurrent/
MemorySwapCurrent/TasksCurrent via D-Bus and only checked whether the
round trip itself succeeded (r >= 0). systemd's own convention -- not
previously accounted for here -- is to report UINT64_MAX for a counter
that isn't actually being tracked yet, not an error and not zero.
For MemoryCurrent specifically, an unguarded sentinel read produced
(float)UINT64_MAX / (1024*1024) == 17592186044416.0 (2^44) as a single
job's "memory usage", which the scheduler's rusage reconciliation then
summed across every RUN job that hit the same condition -- a live replay
session showed exactly 19 * 2^44 = 334251534843904.0 ("used=304.00 TB")
for 19 affected jobs, matching to the bit.
Added sd_resource_counter_is_available() and gated all four property
reads on it. Applied uniformly (not just MemoryCurrent) since systemd
uses the identical convention for the other three counters. Left the
existing memory_rss MB wire unit alone (bug-1735's established
contract, also relied on by the controller's own display pipeline) --
changing it would need auditing that separate consumer, out of scope
here.
bug-1814.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce-build.sh previously left --jobs unset with no fallback -- a bare/
direct invocation (CI, a manual `bash xce-build.sh`, anything bypassing
the /xce-build skill) got meson/ninja/go's own unthrottled
auto-parallelism (~nproc+2), risking starvation on this sandbox's
routinely-concurrent build/test sessions.
The script now computes the same live default the /xce-build skill has
always computed by hand -- nproc, available memory, and load average
read fresh at invocation, with a per-profile memory budget (1536 MB/job
asan, 1024 MB/job debug/release) and process-factor discount (0.5 asan/
debug, 0.7 release) for the compiler driver's forked cc1/as children --
whenever --jobs isn't passed explicitly. An explicit --jobs N still
always wins; skill-driven builds are unaffected since the skill already
passes its own computed value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.
The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.
test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
A resync redelivery (on_scheduler_join()'s full push, fired on every
scheduler reconnect) can race a job's own very-recent dispatch
confirmation and carry a stale, non-terminal state for it. The
duplicate-jobid merge in core_events.c trusted this claim
unconditionally, freeing the job's already-captured allocation and
regressing its state -- live-confirmed via a production scheduler.log
showing a resync WARN landing 16ms after the job's own RUN
confirmation, and matching a live-reported simulator symptom (a RUN
job silently missing from info worker's allocation listing).
Refuse a non-terminal claim when the existing job already has a real
allocation on record, mirroring the existing phantom-RUN guard's own
reasoning in the opposite direction; a genuinely terminal claim
(DONE/EXIT) still applies and releases the allocation correctly, same
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
pthread_cond_wait() reacquires its mutex before honoring a cancellation,
but mylock.c installed no cleanup handler to release it again -- a
thread cancelled mid-wait terminated holding the lock forever. vtaskmgr's
pthread_cancel()-based teardown could land there from three angles
(dispatcher cancelled while polling the shared queue, or a vcpu racing
back into it after its own shutdown signal), permanently deadlocking any
other thread that touched the same queue and hanging vtaskmgr_fini()'s
pthread_join() indefinitely.
Reproduced live as xce_test hanging on Ctrl-D exit once a connection
context was entered; gdb confirmed a vcpu thread blocked forever on a
mutex owned by an already-exited thread ID.
Fix: cbuf_put()/cbuf_get()/cbuf_timedget() now unlock via a
pthread_cleanup_push/pop handler around the wait, so a cancelled waiter
always releases the mutex. vcpu_idle()/vcpu_entry() also recognize the
teardown poison pill and exit immediately instead of needlessly
re-announcing into the already-dead shared queue.
New test_vtaskmgr_teardown.c reproduces the race deterministically
(self-pinned to one CPU, since true multi-core parallelism rarely hits
the window) and bounds vtaskmgr_fini() with a timed join so a
regression fails fast instead of hanging the suite. Logged as bug-1810.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
xce_core_push_events() WARNed on every JOBS_PUSH resync redelivery
(sched_add_job rc=-EEXIST), which is the expected outcome under
on_scheduler_join()'s full resync or the array-push dedup race
(bug-1734/bug-1714) and flooded operational logs. Only that case now
logs via a new DBG() macro (utils.h); any other add_rc (e.g. -ENOMEM)
still WARNs. DBG is named to avoid colliding with the build's own
-DDEBUG command-line define (debug/asan buildtypes).
Also collapsed hash_string()'s duplicated djb2 loop to delegate to
hash_block(), which already implements the same RUN_HASH loop.
Logged as bug-1809 in .wolf/buglog.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Every cmd_* REPL handler in test.c/join_worker_args.c reset flag parsing
with `optind = 1;` only. glibc's getopt_long only fully reinitializes its
internal scan state (nextchar/first_nonopt/last_nonopt) when optind==0, or
on the process's very first getopt_long() call ever -- optind=1 alone
leaves that state stale for every call after the first. Since xce_test's
REPL dispatches many independent cmd_*() calls per process, a command
whose parse ends on a short no-argument flag (`qctrl -a`, `hostctrl -o`)
left state that corrupted the NEXT unrelated command's parse badly enough
to misread its own name as an unrecognized flag. This was the actual
cause of harness_unit_test's 4 deterministic (not flaky -- 20/20 identical
repeat runs) test_test_cli.c failures, each burning its full 5s
confirmation-prompt timeout. Fixed by using optind=0 at all 21 reset
sites; confirmed via 3 isolated pty repros plus 10 consecutive clean
harness_unit_test runs (132/132, was 4 failures/run every time).
Also fixes a resource leak found while investigating: test_test_cli.c's
spawned xce_test sessions were only torn down by inline
clitest_kill_and_reap()/clitest_close() calls at the bottom of each test
function, which Unity's longjmp-based TEST_ASSERT abort skips entirely on
failure, leaking the child process and its pty/pipe fds. Added a static
session mirror (safe across the longjmp, unlike a stack pointer) reaped
from harness_test.c's tearDown(), which Unity guarantees runs after every
test regardless of outcome.
Logged as bug-1807/bug-1808 in .wolf/buglog.json, with the underlying
getopt/longjmp gotchas recorded in .wolf/cerebrum.md. Full ASAN
/xce-build harness also passes 5/5 with no sanitizer findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Follow-up to the xce-harness-component spec, which relocated xce_test
from src/controller/testsuite/ into a standalone src/harness/
component (commits cf55d691..cf380750). These four files were outside
that spec's own file boundary (owned by other specs / general tooling)
so were deliberately left alone at the time, but their hardcoded
default path and error-message advice were now stale and would have
misled anyone who hit them:
- .claude/skills/xce-build/SKILL.md: added `harness` to both component
lists (frontmatter description + args section), and replaced the
"controller always builds xce_test too" claim with an accurate
description of the harness component, its own build command, and
the accepted Meson test-runner quirk (bug-1806) that can still
transitively rebuild xce_test as a side effect of another
component's default-profile test run.
- src/controller/testsuite/test_rcfile_integration.sh (owned by the
xce-test-rc-file spec): XCE_TEST_BIN default and failure message.
- src/controller/testsuite/clean_failover_preservation.sh (owned by
ha-term-safety task 5.2): same.
- src/tools/test-commit-durability.sh: same, plus its header comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Feature-level validation (/kiro-validate-impl) found src/harness/CLAUDE.md
(written by task 1.3, before tasks 2.1/2.2/3.1 landed) still claimed no
/xce-build harness entry point and no unit-test binaries existed -- both
false by the time all 16 tasks completed. Updated to describe the real,
final state: /xce-build harness as the build command, and all 5 test
binaries (harness_unit_test + 4 standalone) under the harness suite.
Also logs one more out-of-boundary stale-path finding
(src/tools/test-commit-durability.sh:97) for a future owner.
This closes out feature-level validation for xce-harness-component --
mechanical (173+1213 tests, 0 failures), design/boundary alignment
(zero controller-private dependency, both sanctioned exceptions exact
match), and requirements coverage (22/22 mapped) all passed cleanly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add a harness) case to src/tools/xce-build.sh following the same
three-part template (setup_build / meson compile / conditional meson
test) already used by scheduler)/worker), building all 6 harness
targets (xce_test plus its 5 test binaries) and running suite harness.
Remove controller/xce_test from the controller) case's compile target
list -- it no longer exists as a controller target. Add harness to the
script's usage/component-list text.
Verified both directions: `/xce-build harness` alone produces xce_test
and all 5 test binaries (4/20/5/12/132 tests, 0 failures); `/xce-build
controller` alone no longer produces controller/xce_test while
controller_unit_test stays green at 1199 tests.
Task 3.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move test_resval_typeresolve.c, test_joblist_display.c,
test_join_worker_args.c, and test_listrow_json.c into
src/harness/testsuite/, and re-create their executable()/test() blocks
verbatim in src/harness/meson.build under suite 'harness' (task 1.1 had
already fully deleted them from src/controller/meson.build, not just
moved them, so these had been orphaned on disk since that commit).
Fix test_joblist_display.c's ctrlstatus_fmt.h include path to the
xce/ location promoted in task 1.2.
Verified against the pre-task-1.1 baseline (git show cf55d691^): all
four blocks' source lists, c_args, and env are faithfully reproduced,
only paths and the suite tag changed.
harness suite: 5 binaries, all green (resval_typeresolve 4, joblist_display
20, join_worker_args 5, listrow_json 12, harness_unit_test 132 -- 173 total,
0 failures). controller suite unchanged: 1199 Tests 0 Failures 0 Ignored.
This completes Major Task 2 (Core) -- the harness component now owns all
5 of its test binaries independently of the controller build.
Task 2.2 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Relocate the 13 test files that lived inside controller_unit_test into
src/harness/testsuite/, add a new Unity runner (harness_test.c, modeled
on controller_test.c) carrying their extern/RUN_TEST entries, and
register the new harness_unit_test executable + suite in
src/harness/meson.build with an explicit build-order dependency on
xce_test (two of its tests spawn that binary over a pty).
Finish clearing the controller side: remove test_test_cli.c/
test_xcetest_process_entry.c from controller_unit_test_src and their
extern/RUN_TEST entries from controller_test.c, and remove the
temporary xce_test_controller_link symlink bridge task 1.1 added (no
longer needed now that both binaries build as siblings under harness/).
Verified: harness suite 132 Tests 0 Failures 0 Ignored; controller suite
1199 Tests 0 Failures 0 Ignored. 1199+132=1331 matches the pre-move
controller_unit_test baseline exactly (cross-checked against buglog
bug-1798/bug-1805).
Task 2.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
New src/harness/CLAUDE.md documenting the component's purpose, current
build command, and file layout -- every claim grounded in the live repo
state (verified: no /xce-build harness entry point yet, no
harness_unit_test binary yet, both land in later tasks of this spec).
Task 1.3 of xce-harness-component -- completes the Foundation phase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move the one genuinely shared header (header-only, static inline, zero
link dependency) from src/controller/testsuite/ to src/include/xce/,
alongside the existing cross-daemon shared headers (cluster.h, types.h,
errnox.h). Repoint its three current consumers (harness's test.c and
joblist_display.c, controller's test_ctrlstatus.c) to the new path.
test_joblist_display.c is intentionally left on the old path -- it
relocates in task 2.2.
Also removes src/harness/meson.build's temporary '../controller/testsuite'
incdir shim, which task 1.1 added and explicitly marked for removal once
this task landed.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (unchanged from
task 1.1's baseline, confirming a header-path-only change).
Task 1.2 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Move xce_test's 15 production sources + headers from src/controller/testsuite/
into a new src/harness/ component with its own meson.build, and strip every
now-dangling controller-side reference (11 controller_unit_test_src entries,
4 standalone test-binary blocks, 11 controller_test.c extern/RUN_TEST lines)
so the controller build is never left broken.
Fixes a broken relative-symlink bridge (xce_test_controller_link) added to
keep two controller-embedded tests (test_test_cli.c, test_xcetest_process_entry.c)
finding the relocated xce_test binary until task 2.1 moves them too.
Discovered and documented in design.md: four controller-domain regression
tests call functions directly from four of xce_test's own files, requiring
controller_unit_test to permanently recompile them from src/harness/ — a
real dependency neither the original discovery nor design audit caught.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (verified via the
Unity binary's own printed summary line, not just meson's aggregate count).
Task 1.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
core_reset_workers already does (bug-1751), closing the gap that let a
worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
a captured job's job->state is set to RUN synchronously at dispatch capture,
mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
live feedback that it over-fires inside busy sessions; replace it with a
prescan-based "next non-dummy session" hint in info summary instead
Full scheduler suite: 77/77 tests pass (asan profile).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
The hand-written per-flag description explaining -n's <min>[,<max>]
format was dropped when task 3.5 moved the flag list to the generated
flagspec block (which only shows short/long/required/default, with no
room for a flag's value-format semantics). Restore the explanation as
prose in USAGE_JOBSUB_HEAD, following the same pattern already used by
resupdate's HEAD (-n/-v pairing, -t's conditional meaning).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Follow-up audit (requested: check other generated sources, specifically
packcc/PEG output) found the identical bug shape in the four PEG-parser
custom_targets: section_parser, rusage_parser, xce_resreq_parser, and
xce_select_parser each produce a .c + a paired .h (section.h, rusage.h,
xce_resreq.h, xce_select_peg.h), but libxce_src only referenced each
parser's [0] (.c) index. lib/resreq.c -- confirmed via a project-wide
grep to be the only consumer of any of these four headers -- had no
declared build-order dependency on any of the four codegen steps.
Fix: pass each of the four custom_targets unindexed into libxce_src,
same pattern as the capnp fix. No libxce_dep change needed here since
nothing outside src/lib/ consumes these headers.
A project-wide search confirms these, plus the capnp custom_target, are
the only multi-output C/H codegen steps in the whole build -- every other
custom_target() (xceadmin, apiserver, xcecli, autoscaler/providers)
produces a single Go-binary/stamp output with no equivalent hazard.
Verified: a third consecutive full from-scratch `./tools/xce-build.sh all
debug` build (1173/1173 targets) succeeded clean; a fresh `lib debug`
build plus the full lib test suite (313/313) passed; a fresh incremental
controller/ASAN rebuild passed 1331/1331 on a quiet re-run (its first,
concurrent-with-the-full-build run hit an unrelated, confirmed-flaky PTY
harness timing failure in test_smoke_launch_send_echo_capture_stdout_and_exit_code,
reproduced as system-load-induced, not caused by this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
The previous fix (654c6e8c) only closed the missing-dependency-edge race
inside the libxce target itself. A live user report on a second
from-scratch `./tools/xce-build.sh all debug` run showed the same race
still hitting scheduler/src/sim_record.c and core_events.c, which reach
controller.capnp.h directly via a raw -Ilib include path with no
declared dependency on the codegen step at all.
Root cause: libxce_dep = declare_dependency(link_with: libxce) -- the
dependency object controller/worker/scheduler all consume uniformly --
only orders a consumer's LINK step after libxce is built; it never
propagated a build-order dependency on the capnp custom_target to a
consumer's own COMPILE steps.
Fix: add `sources: [gen_srcs]` to libxce_dep's declare_dependency() call.
This kwarg exists specifically for generated files that must be built
before any consumer of a dependency compiles, and since every component
already lists libxce_dep in its own dependencies, this one change
propagates the missing edge everywhere with no per-component changes.
Verified: two consecutive full from-scratch `./tools/xce-build.sh all
debug` builds (1173/1173 targets, the user's own exact repro command)
both succeeded with no retry; a subsequent incremental controller/ASAN
rebuild stayed clean (1331/1331 tests, 0 ASAN diagnostics).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
libxce_src listed only gen_srcs[0] (controller.capnp.c) as a source of
the static_library('xce', ...) target, never controller.capnp.h -- which
several library files (utils.c and others) reach only via #include.
Ninja had no build edge forcing the capnp custom_target to finish before
those files compiled, so a from-scratch build could schedule them first,
intermittently failing with "controller.capnp.h: No such file or
directory". This is the same symptom recorded across 5+ prior recurrences
in buglog.json since 2026-07-16, previously worked around by retrying the
build rather than root-caused.
Fix: pass the whole gen_srcs custom_target (not gen_srcs[0]) into
libxce_src -- Meson still compiles only its .c output, but now orders the
entire target's build after both outputs are generated. Verified via two
consecutive from-scratch `xce-build.sh lib debug` builds (no retry
needed), the full lib test suite (313/313), and an incremental controller
asan rebuild (1331/1331, 0 ASAN diagnostics) to confirm no regression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Add one black-box regression test proving process-level help, in-session
help content, and context-scoped tab completion work correctly when
exercised together in one continuous session (context switches +
completion + help lookups), not just individually as each was already
covered by tasks 1.6/2.3/3.5. This closes the final task of the
xce-test-cli-ux implementation plan (23/23).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Audit every printf() call site in test.c not already covered by the
flagspec-rejection or perror() paths; convert 33 genuine error/warning
messages (expect-jobs, expect-job, wait-primary, help's not-found path,
jobrun/jobsub/jobstart/jobfinish/jobfail validation, resupdate, set) from
stdout to stderr, so a command's successful result data and its
diagnostics never share a stream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Wire flagspec_render_help() into cmd_help via new command_t fields
(flagspec/flagspec_count/usage_tail), so every flag-accepting command's
help composes description -> generated flag block -> example/related
notes, with hand-written per-flag lines removed as a duplicate source of
truth. Add missing purpose descriptions, examples, and confirmation-gate
related notes across ~20 commands; fix bug-1711's "matchs" grammar
(Req 11.2); confirm join's corrected maxmem default (bug-1557, Req 12.2)
and the --json flag (Req 5.4) now render correctly end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Bring workerlist/joblist/reslist onto the shared flagspec_t library with an
identically-spelled --json flag, wired to the task-2.2 JSON list formatter
via a new pure field-mapping module (listrow_json.c/.h). Default table
output is unchanged; JSON rows carry raw numeric values (flags, timestamps,
memory sizes) rather than their humanized display strings, per Requirement
5.1's "same fields" contract.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Gates jobkill and shutdown unconditionally, qctrl only when action is
QUEUE_CLOSE/QUEUE_INACTIVE (not ACTIVE/OPEN), hostctrl only when
HOST_CLOSE (not OPEN), and upgradeconfig unconditionally -- exactly
Requirement 9.5's enumeration, with jstart/jfinish/jfail (worker
context) correctly untouched. Each gains a -y/--yes bypass flag via
flagspec_t; cmd_set gains -y/+y calling confirm_set_skip(). A decline
maps to a plain `return 0` (success-coded, distinct from a negative
failure return) per confirm.h's own documented contract -- this means
a declined destructive command inside a source'd file under `set -e`
won't trip abort-on-failure, a real but narrow interaction only
reachable when a human is actively declining; left for task 4.3 to
test explicitly. 1321/1321 across the full controller suite (+3),
0 regressions.
Task 3.3 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Routes all 26 blocking xcecli_* call sites across ~24 commands through
xcetest_run_interruptible(), wires xcetest_set_active_connection() at
execute_line()'s dispatch point (active_ctx->ctx, read fresh per
command), and fixes main()'s REPL loop to distinguish a real EOF from
an EINTR-interrupted readline() so Ctrl-C at an empty prompt discards
the line instead of exiting the session.
Live-discovered and fixed bug-1794: fill_job_alloc() never initializes
xce_job_alloc_t's nres_shared/allocs_shared fields -- harmless as stack
garbage before this task, genuine heap garbage once moved into a
heap-allocated ctx block (a necessary consequence of the adapter's
heap-only precondition). Fixed by switching all ~26 call-context
allocations from malloc to calloc uniformly. Also fixed an argv-aliasing
hazard: cmd_hello/cmd_jobckpt's -m values aliased optarg, which
execute_line() frees immediately after return including on the
abandoned path -- now strdup'd into the heap block.
Also closed a safety-critical gap this task's own design introduced
(bug-1795): before this task the REPL was fully synchronous, so
leave_ctx() (called by up/quit) could never run while an RPC was
outstanding. An abandoned worker can now persist past command
completion, so leave_ctx() tearing down the connection could
use-after-free it. Fixed via xcetest_wait_connection_idle(), which
blocks on the same per-connection semaphore the adapter already
tracks, then releases it, proving no worker is active before teardown.
Proven with a deterministic white-box RED->GREEN test, since the race
itself is too timing-dependent for a black-box test to reliably force.
1361/1361 across the full controller suite (+2), 0 regressions.
Task 3.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.2, 1.3, 1.4, 1.5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Migrates 17 commands (cmd_hello, cmd_jobinfo, cmd_jobrun, cmd_jobsub,
cmd_resval, cmd_jobkill, cmd_jobckpt, cmd_qctrl, cmd_hostctrl,
cmd_runtasks, cmd_setpend, cmd_jobstart, cmd_jobfinish, cmd_jobfail,
cmd_resupdate, cmd_walcompare, join_worker_parse_attrs) from raw
getopt()/hand-rolled parsing to the shared flagspec_t + getopt_long
mechanism, so every one now uniformly rejects an unrecognized or
malformed flag with a usage summary and, where a close match exists,
a suggestion. cmd_walcompare's hand-rolled parser needed a leading '+'
in its optstring to preserve its pre-migration flags-only-before-
positionals behavior. Also fixes bug-1704: cmd_qctrl's uninitialized
action variable, now sentineled and rejected when unset, mirroring
cmd_hostctrl's existing pattern -- this also makes the destructive-
command gate's -i/-c detection (task 3.3) safe to rely on. Every
existing short flag's letter, value format, and default is unchanged
for a valid invocation. 1357/1357 across the full controller suite
(+4), 0 regressions.
Task 3.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 2.1, 2.2, 2.3, 2.4, 6.1, 6.2, 6.3, 6.4, 10.1, 10.2, 10.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
cmd_join_worker's simulated-worker maxmem default was 64 bytes -- a
leftover from when the field was MB-scale, unfixed when maxmem became
bytes-canonical elsewhere. Now 64 * 1024 * 1024. Extracted the
argument-parsing/defaulting logic into join_worker_parse_attrs()
(join_worker_args.c), since it runs entirely before the function's
network call and is a genuine testable seam -- same precedent as
task 2.5's joblist_display.c. Verified live against a real deployed
cluster (workerlist reports MaxMem: 64.00 MB) plus a permanent 5-test
regression suite. maxswap/maxtmp have the identical stale-default bug
(logged as bug-1793) but are deliberately left unfixed here since
USAGE_JOIN_WORKER's help text currently documents their stale values
accurately -- fixing only the code would create a new docs mismatch;
a locked-in sibling-defaults test documents this for the follow-up.
1353/1353 across the full controller suite (+5), 0 regressions.
Task 2.6 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 12.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Eight display/text defects in test.c: worker_status_str's "UNKOWN"
typo, cmd_runtasks's "sequentail" typo (display-only, verified inert
against the real launch_type consumer), cmd_reslist's right-justified
columns and unlabeled hex Flags (now matching workerlist/state_raw's
conventions), joblist's raw-epoch timestamps (now via a format_job_time
helper delegating to the same formatter ctrlstatus already uses) and
combined NTasks column split into MinTasks/MaxTasks, job_state_str's
missing raw-state fallback for ambiguous RUN+EXIT (mirroring
print_jobinfo's existing pattern), and getconf's no-op group/config
list stubs (now genuinely implemented from already-fetched data,
matching the three sibling print functions' convention).
worker_status_str/job_state_str/format_job_time are pure functions with
no networking/REPL/global-state dependency, extracted into a new
joblist_display.c/.h module with a permanent 20-test standalone suite --
following this codebase's own resval_typeresolve.c/misc.c precedent for
testing functions that would otherwise be stranded inside untестable
test.c. cmd_reslist/joblist_query/cmd_getconf's table-printing bodies
inline fetch+print and remain untestable without a live cluster.
1348/1348 across the full controller suite (+20), 0 regressions.
Task 2.5 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 11.1, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
cmd_source reuses rcfile_load() unmodified for line execution, adding
an explicit missing-file check (bypassing rcfile_load's silent-missing
policy, which is correct for its one existing startup-rc-file caller
but wrong for an explicit user command) and a realpath()-canonicalized
in-flight path stack that rejects direct or transitive cycles instead
of hanging. Registered in all four command contexts. Translates
rcfile_load's raw positive "aborted" return to -1 so an enclosing
source line correctly sees a transitively-aborted nested source as
failure, not success (preprocess_and_run only treats negative returns
as failure). Proven via real fixture files driven through the black-box
pty harness: a genuine two-file mutual cycle, a missing path, and
ordered execution across a non-cyclic nested source. 1310/1310
controller suite passing (+3).
Task 2.4 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 13.1, 13.2, 13.3, 13.4, 13.5, 13.6
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Registers libedit's GNU-readline-compatible rl_attempted_completion_function
under g_interactive only. The candidate generator reads active_ctx->commands[]
live on every call, so completion correctly tracks context switches rather
than freezing at registration time. Single/multi-match UI (auto-complete vs.
double-Tab candidate listing) is entirely libedit's stock behavior once given
a correct context-scoped generator. Proven via a black-box pty test verified
sound against a false-positive risk: xce_test has no fuzzy command-name
matching, so an uncompleted partial token cannot coincidentally trigger the
same effect a real completion would. Also fixes a harness bug (bug-1792):
disabling kernel ECHO for output capture was silently suppressing libedit's
own completion feedback too; added clitest_spawn_ex() with disable_echo as
an explicit parameter, clitest_spawn() unchanged for existing callers.
1307/1307 controller suite passing (+1), re-verified 8/8 for flakiness.
Task 2.3 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 7.1, 7.2, 7.3, 7.4
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
listfmt_json_begin/new_row/add_string/add_number/finish wrap cJSON
(already linked via struct2json_dep) into a minimal array/object
lifecycle library with no generic "record" abstraction -- each command's
own row struct maps its fields directly when wired in later (task 3.4).
Proven via a real stdout-capture-then-reparse round trip, including a
733-character field with no truncation, and stderr staying empty.
1306/1306 controller suite passing (+9).
Task 2.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 5.1, 5.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
confirm_destructive() prompts on stderr and blocking-reads a y/n answer
via a single, non-looping fgets() call (not libedit) -- the mechanism
that makes a real Ctrl-C during the prompt return EINTR and be treated
as a decline (Requirement 9.9), same as an explicit "n" or EOF.
confirm_set_skip()/confirm_is_skip_set() add a session-scoped toggle
mirroring cmd_set's existing -e/+e style. Auto-skips when input is not
interactive, via a new xcetest_is_interactive() accessor over the
existing g_interactive global. Destructive-command enumeration and
per-command wiring (Requirement 9.5) is task 3.3, not this task.
Proven with a real fork+SIGINT test. 1297/1297 controller suite
passing (+7).
Task 2.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 9.1, 9.2, 9.3, 9.4, 9.6, 9.7, 9.8, 9.9
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
clitest_spawn/send_line/send_partial/send_tab/send_signal/wait_for_text/
wait_for_exit/kill_and_reap launch the real compiled xce_test behind a
genuine pty (so isatty/g_interactive/libedit behave as a real terminal),
with stderr routed to a separate pipe for real stream-separation testing.
All waits are poll()-bounded with an explicit timeout so a future
regression can't hang the whole test suite. This is the shared
infrastructure tasks 2.3, 3.5, and all of Validation phase (4.1-4.4)
will build their black-box scenarios on top of. One smoke test proves
launch/send/capture/exit-code/stream-separation end to end.
1290/1290 controller suite passing (+1).
Task 1.7 of .kiro/specs/xce-test-cli-ux/tasks.md.
Foundation phase (1.1-1.7) now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
argp already provided --help/-?/--version with exit-without-REPL
behavior, but had no -h alias (its default short help flag is -? per
GNU argp convention) despite Requirement 4.1 naming -h explicitly.
Added a hidden -h option triggering the same help path, and extended
the doc string to describe the in-session help command. Proven against
the real compiled binary via subprocess tests, not just argp's own
in-process behavior. 1289/1289 controller suite passing (+4).
Task 1.6 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 4.1, 4.2, 4.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
A registry of per-connection locks, acquired (interruptibly) before
spawning a worker and released by whichever side finishes last, so an
abandoned call's worker is never running concurrently with a new call
on the same connection. Uses a POSIX semaphore (sem_t), not a mutex:
review round 1 found the initial pthread_mutex_t implementation relied
on behavior POSIX classifies as undefined for a non-owning-thread
unlock (not just error-checking mutex types) -- this design's worker
thread legitimately releases a lock the caller thread acquired, which
sem_t supports with no ownership concept at all. bug-1791. Design.md
synced to reflect the semaphore choice. 1285/1285 controller suite
passing.
Task 1.5 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
xcetest_run_interruptible() runs a blocking call on a detached worker
thread while the caller polls xcetest_interrupted() every 50ms. A single
atomic CAS (RUNNING -> DONE_UNCLAIMED | ABANDONED) decides which side
frees the heap-allocated call block, so an abandoned call's worker
thread never double-frees or leaks once it eventually finishes. Proven
under active ASAN with a real detached thread and a real SIGINT-forced
abandonment. 1282/1282 controller suite passing (+2).
Task 1.4 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.2, 1.3, 1.4
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Replaces the bare signal(SIGINT, sigbreak) install with sigaction() and
SA_RESTART cleared, so a blocking read reliably returns EINTR instead of
being silently restarted by glibc's BSD signal() semantics. Adds
xcetest_interrupted() as a read-only accessor over the existing break_out
flag for the interruptible-call adapter (tasks 1.4/1.5) to consume.
Proven by a real fork+SIGINT+blocked-read test that first demonstrates
the old bug, then the fix. 1280/1280 controller suite passing (+2).
Task 1.3 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 9.9
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
flagspec_reject() now names a likely-intended flag when an unrecognized
flag closely matches exactly one entry in the descriptor table (genuine
Levenshtein DP, silent on zero or ambiguous matches). flagspec_render_help()
generates the short/long/required-optional/default flag-listing block from
a descriptor table. 1278/1278 controller suite passing (+5).
Task 1.2 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 6.1, 6.2, 6.3, 6.4, 8.2, 10.1, 10.2, 10.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Adds flagspec_t descriptor table, getopt_long option-table builder, and
the shared stderr-routed rejection path that every flag-accepting
command will migrate onto in later tasks. TDD'd via test_flagspec.c
(1273/1273 controller suite passing).
Task 1.1 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 2.1, 2.2, 2.3, 2.4, 3.1
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
The previous fix for the "slots" regression (5b18f14a) worked but was an
unneeded special case: strcmp(vres->res->name, "slots") to fall back to
nslots directly. alloc_req->host already holds exactly that value during
the global phase -- both allocators' own unconditional "add slots" step
stages it there regardless of phase, and plugin_prepare_allocation()
never resets .host during the global phase, so by the time sched_fill_
limit_requests() runs (right after, same invocation), .host contains
exactly "slots"'s job-wide total and nothing else.
Read alloc_req->host FIRST for this combination and fall back to the
rusages-based on-demand computation only when nothing is staged there
(get_resval_by_id()'s "not found" == 0.0, safe here since "slots" is
never legitimately 0 -- seqjob only runs for maxTasks==1, parjob only
for maxTasks>1). Data-driven instead of name-based, matching bug-1789's
own earlier flag-based-dispatch philosophy, and generalizes for free to
any future resource an allocator plugin similarly pre-stages outside
its normal rusages loop.
test_fill_limit_requests_global_limit_on_slots_reads_host now pre-stages
alloc_req->host exactly as the allocators do, rusages left empty,
matching production exactly.
Full xce-build.sh all asan --incremental: 94/94 Ok. Live re-verified
against both the original gitea#46 scenario and the ug1_global_mem
scenario: byte-identical scheduler traces to every prior verification
round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
The limit-plugin-ownership refactor (fa34b038) broke gitea#46's exact
original scenario. calc_wanted_job_total_resval() scans the job's
rusages by name, but "slots" is never a declared rusage entry --
seqjob_prepare_allocation()/parjob_prepare_allocation() derive its want
directly from the job's own task count via their own unconditional
key_find("slots", ...) + store, entirely outside the rusages loop. The
refactor's on-demand lookup therefore always missed for "slots",
silently leaving the computed want at 0 -- reintroducing bug-1789's
exact original symptom (matching worked, but caps never bound).
Live-confirmed against a fresh deploy of gitea#46's own repro (9 plain
sleep jobs, user1, queue normal, no -R flag): limit1/ug1_slots/
ug1_normal_slots all matched correctly but zero unsatisfy_ lines
appeared and all 9 jobs dispatched uncapped. The refactor's own new
unit test had masked this -- it artificially added a "slots" rusages
entry that doesn't reflect production reality.
Fix: sched_fill_limit_requests() now falls back to `nslots` directly
(the same value both allocators already store as "slots"'s own
per-host want) whenever the rusages lookup misses and the resource is
named "slots" -- the one documented exception to the rusages-lookup
rule, matching this codebase's existing "slots" special-casing
convention.
New test test_fill_limit_requests_global_limit_on_slots_uses_nslots
proves the fallback with no "slots" entry in rusages at all. The
original test reworked to use "mem" instead, so it now genuinely
exercises the general on-demand path.
Full xce-build.sh all asan --incremental: 94/94 Ok. Live re-verified
against both the original gitea#46 scenario (byte-identical to the
pre-refactor trace) and the ug1_global_mem scenario -- both correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
bug-1790's fix worked but put limit-plugin-specific logic in the wrong
place: seqjob_prepare_allocation()/parjob_prepare_allocation() (the
allocator plugins) unconditionally pre-computed a job-wide total for
every per-host resource in the job's rusages, "just in case" some
global limit needed one. That's the allocator plugins' concern leaking
into limit evaluation -- duplicated across both allocators (a third
would need it too), and wasteful (computed for resources no limit
actually references).
Moved ownership to sched_fill_limit_requests() (resource.c), called
once centrally from plugin_prepare_allocation() regardless of which
allocator ran:
- Removed calc_wanted_host_resval_job_totals() and its calls from both
allocator plugins entirely.
- Added calc_wanted_job_total_resval() (resreq.c): an on-demand,
single-resource lookup by name (no resmgr/key_find needed -- the
caller already has the resolved xce_resource_t), computing the same
job-wide-total formula only for the one (scope, underlying-resource)
combination that needs it -- a GLOBAL limit on a per-host resource.
The other three combinations are unchanged, still reading straight
out of alloc_req.
- sched_fill_limit_requests() gains rusages/nhosts/nslots parameters
back (a genuine need this time, not a re-added unused one).
plugin.c's two call sites pass obj->u.primary->resreq->rusages
(NULL for COMPOUND/ALTER, matching this codebase's existing scope
convention for those tags) and ctx->nhosts/ctx->nslots.
All 23 testsuite link-satisfying stubs mechanically updated. Removed
the two allocator-level regression tests from the first pass (the
behavior they proved no longer exists by design); rewrote
test_fill_limit_requests_global_limit_on_host_resource_reads_host to
exercise the new on-demand path directly, nslots=3 to prove the job's
WHOLE total is used.
Full xce-build.sh all asan --incremental: 94/94 Ok, zero regressions.
Live re-verified against the same ug1_global_mem deploy/scenario:
byte-identical scheduler trace to the pre-refactor result, confirming
behavior-preserving.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
test_job_rusage_reconciles_mem_reservation_against_usage asserted
w->resInUse was already charged 4.0 for a JOBS_PUSH-dispatched RUN job,
immediately after the push with no round boundary simulated. That
assumption was wrong, not the scheduler: resInUse/sharedInUse is never
charged incrementally at JOBS_PUSH/dispatch time. It's fully re-derived
once per round by sched_account_jobs() (scheduler.c), which iterates
every currently-RUN job and calls sched_charge_account() on each one's
accounts -- called by both the real normal-mode driver (driver.c) and
the sim driver (sim_loop.c), always before anything else in that round
reads resInUse. Confirmed by reading the full JOBS_PUSH handler
(core_events.c) and builtin_fcfs_on_job_push(): neither ever charges or
uncharges resInUse, by design -- core_session.c's own doc comment on
core_begin_session_internal() already documents this "recompute from
source of truth each round" architecture.
Fix: the test now calls sched_charge_account(&core, tracked->accounts)
right after JOBS_PUSH, simulating the one round-boundary reconciliation
step sched_account_jobs() would perform for its single RUN job (that
function itself lives in scheduler.c, not white-box #included here, so
its single-job effect is reproduced directly via the same primitive it
calls internally). No production code changed.
Full scheduler suite: 77/77 Ok (was 76/77). Full xce-build.sh all asan
--incremental: 94/94 Ok, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
free_host_account() (src/lib/misc.c) freed acct->accounts[] and acct
itself but never acct->host, the strdup'd string make_host_account()
(xcealloc.c) allocates -- a real leak on every free_job_account() call,
including the live production dispatch path (scheduler.c, core_events.c),
not just test fixtures. Already root-caused and the exact fix specified
in bug-1621 (logged, never applied); applied here: FREEUP(acct->host)
before FREEUP(acct), mirroring free_res_account()'s existing
FREEUP(acct->resname) pattern.
Also closes bug-1723: test_xcealloc.c's two multi-host overflow
regression tests (added by bug-1722's fix) never freed their own
locally-built fixture trees. Added free_JobAlloc_ptr(&jalloc) to
test_make_job_account_multi_host_no_overflow() and
free_HostAlloc_list(2, hallocs) to
test_make_host_allocs_multi_host_no_overflow(), exactly as bug-1723's
own root-cause note specified.
Together these take xcelib_test from a deterministic 456-byte/29-
allocation LeakSanitizer SIGABRT (masking every other libxce test result
behind it) to 1/1 Ok. Full xce-build.sh all asan --incremental: 94/94 Ok,
zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Adds ug1_global_mem to limits.yaml: a permanent, real GLOBAL limit on
"mem" (a per-host resource, not "slots"), capping ug1's total
cluster-wide mem usage at 5GB. This is exactly the scenario bug-1790
fixed but only unit-tested until now -- no limit in the shipped example
config exercised a global cap on a non-slots per-host resource.
Live-verified via a fresh single-node deploy: 3 jobs submitted as user1
(ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue
normal. Scheduler trace confirms ug1_global_mem correctly matches and,
for the first time, computes a real nonzero job-wide-total wanted value:
unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000
uncomit 0.000000 wanted 2000000000.000000
unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall
1000000000.000000
The 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) correctly PENDed
until an earlier job's mem freed up. Deploy config validated with zero
new warnings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Closes the gap bug-1789 explicitly carved out: a GLOBAL limit built on a
per-host resource other than "slots" (e.g. a future absolute global mem
cap) needs ctx->alloc_req.host to carry that resource's job-wide total
want during the global phase, but only "slots" ever got that treatment
(via seqjob/parjob_prepare_allocation()'s own unconditional store).
Adds calc_wanted_host_resval_job_totals() (resreq.c), sharing its loop
body with calc_wanted_global_resvals() via a new static helper
parameterized only by which XCE_RESF_GLOBAL direction to match -- it
computes the SAME job-wide-total formulas (PER_TASK: value*nslots, PER_JOB:
value, PER_HOST: value*nhosts) for non-global resources instead of global
ones. seqjob/parjob_prepare_allocation()'s global-phase branch now calls
it into ctx->alloc_req.host alongside the existing calc_wanted_global_
resvals() call into .shared.
Safety was traced, not assumed: plugin_prepare_allocation() only resets
.host at the top of a per-host call, never during the global phase, and
sched_fill_limit_requests() for glimits reads .host later in the SAME
global-phase invocation -- transferring the value into the limit's
permanent .shared record before any later per-host call ever resets
.host. An earlier candidate fix (reusing calc_wanted_host_resvals()
directly) was rejected during design: its PER_HOST formula and, for
parjob, its per-bucket nslots don't give a job-wide total.
New regression tests in test_seqjob_allocator.c/test_parjob_
combinatorial.c prove a non-"slots" per-host resource ("mem") now gets
correctly populated; parjob's test uses maxTasks=3 specifically to prove
the formula uses the job's whole nslots, not a bucket's share.
Full suite (xce-build.sh all asan --incremental): 92/94, both failures
pre-existing and unrelated (bug-1762, bug-1733's already-documented
xcealloc.c leak). Not live-verified -- no limits.yaml entry exercises
this path yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
meson/ninja/capnp/Go only exist inside the hermetic nix dev shell, but
xce-build.sh never entered it itself -- the /xce-build skill documented
wrapping every invocation in `nix develop ./src --command ...` by hand,
so a bare `bash src/tools/xce-build.sh ...` (run directly instead of
through the skill) failed opaquely with "meson: command not found". This
was a recurring, previously-logged gap that resurfaced again today mid
bug-1789 verification.
The script now checks IN_NIX_SHELL at the top and re-execs itself under
`nix develop` if unset, so every caller -- the skill, a bare invocation,
CI, a human at a terminal -- gets it automatically. A nested call (already
inside the shell) is a no-op past that check. README.md's daily-develop
section updated to match; the old "or use nix develop --command" example
is now redundant and removed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
