A registry of per-connection locks, acquired (interruptibly) before
spawning a worker and released by whichever side finishes last, so an
abandoned call's worker is never running concurrently with a new call
on the same connection. Uses a POSIX semaphore (sem_t), not a mutex:
review round 1 found the initial pthread_mutex_t implementation relied
on behavior POSIX classifies as undefined for a non-owning-thread
unlock (not just error-checking mutex types) -- this design's worker
thread legitimately releases a lock the caller thread acquired, which
sem_t supports with no ownership concept at all. bug-1791. Design.md
synced to reflect the semaphore choice. 1285/1285 controller suite
passing.
Task 1.5 of .kiro/specs/xce-test-cli-ux/tasks.md.
Requirements: 1.1, 1.5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Captured simulator REPL session findings from bug-1790 verification:
incorrect maxswap resource value display and a suggestion to show
per-worker slot allocation counts in `info worker` output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019jaMf9QEwpugH2W9KAuc7f
Adds ug1_global_mem to limits.yaml: a permanent, real GLOBAL limit on
"mem" (a per-host resource, not "slots"), capping ug1's total
cluster-wide mem usage at 5GB. This is exactly the scenario bug-1790
fixed but only unit-tested until now -- no limit in the shipped example
config exercised a global cap on a non-slots per-host resource.
Live-verified via a fresh single-node deploy: 3 jobs submitted as user1
(ug1 member) each requesting rusage[mem=2000000000] (2GB) on queue
normal. Scheduler trace confirms ug1_global_mem correctly matches and,
for the first time, computes a real nonzero job-wide-total wanted value:
unsatisfy_ resource 36 total 5000000000.000000 inuse 4000000000.000000
uncomit 0.000000 wanted 2000000000.000000
unsatisfy_ unmet resource 36 (limit=ug1_global_mem) shortfall
1000000000.000000
The 3rd job (4GB inuse + 2GB wanted = 6GB > 5GB cap) correctly PENDed
until an earlier job's mem freed up. Deploy config validated with zero
new warnings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
bug-1743 (gitea#46's original tracking entry) still said Part 1
(dispatch-stops-at-8) was "UNRESOLVED/NOT ACTIONABLE, pending reporter
info" -- stale since the 6-bug limit-plugin cascade (bug-1784 through
bug-1790) fully resolved it without needing that info, and gitea#46 was
just validated live and closed. Updates the entry with the full
resolution chain and the closing live-verification evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
Closes the gap bug-1789 explicitly carved out: a GLOBAL limit built on a
per-host resource other than "slots" (e.g. a future absolute global mem
cap) needs ctx->alloc_req.host to carry that resource's job-wide total
want during the global phase, but only "slots" ever got that treatment
(via seqjob/parjob_prepare_allocation()'s own unconditional store).
Adds calc_wanted_host_resval_job_totals() (resreq.c), sharing its loop
body with calc_wanted_global_resvals() via a new static helper
parameterized only by which XCE_RESF_GLOBAL direction to match -- it
computes the SAME job-wide-total formulas (PER_TASK: value*nslots, PER_JOB:
value, PER_HOST: value*nhosts) for non-global resources instead of global
ones. seqjob/parjob_prepare_allocation()'s global-phase branch now calls
it into ctx->alloc_req.host alongside the existing calc_wanted_global_
resvals() call into .shared.
Safety was traced, not assumed: plugin_prepare_allocation() only resets
.host at the top of a per-host call, never during the global phase, and
sched_fill_limit_requests() for glimits reads .host later in the SAME
global-phase invocation -- transferring the value into the limit's
permanent .shared record before any later per-host call ever resets
.host. An earlier candidate fix (reusing calc_wanted_host_resvals()
directly) was rejected during design: its PER_HOST formula and, for
parjob, its per-bucket nslots don't give a job-wide total.
New regression tests in test_seqjob_allocator.c/test_parjob_
combinatorial.c prove a non-"slots" per-host resource ("mem") now gets
correctly populated; parjob's test uses maxTasks=3 specifically to prove
the formula uses the job's whole nslots, not a bucket's share.
Full suite (xce-build.sh all asan --incremental): 92/94, both failures
pre-existing and unrelated (bug-1762, bug-1733's already-documented
xcealloc.c leak). Not live-verified -- no limits.yaml entry exercises
this path yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
meson/ninja/capnp/Go only exist inside the hermetic nix dev shell, but
xce-build.sh never entered it itself -- the /xce-build skill documented
wrapping every invocation in `nix develop ./src --command ...` by hand,
so a bare `bash src/tools/xce-build.sh ...` (run directly instead of
through the skill) failed opaquely with "meson: command not found". This
was a recurring, previously-logged gap that resurfaced again today mid
bug-1789 verification.
The script now checks IN_NIX_SHELL at the top and re-execs itself under
`nix develop` if unset, so every caller -- the skill, a bare invocation,
CI, a human at a terminal -- gets it automatically. A nested call (already
inside the shell) is a no-op past that check. README.md's daily-develop
section updated to match; the old "or use nix develop --command" example
is now redundant and removed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
sched_fill_limit_requests() used to read a limit's underlying resource's
want from the same vector its own virtual-resource entry gets written to.
That's wrong whenever the two differ in scope -- e.g. a GLOBAL limit built
on top of a per-host resource like "slots" (limits.yaml's
ug1_normal_slots): the limit's own vres entry belongs in alloc_req->shared,
but "slots"'s want is only ever staged into alloc_req->host by
seqjob/parjob_prepare_allocation(), so the old code always read a zero and
silently never enforced the cap.
Now both the read source and write destination are derived independently
from each resource's own XCE_RESF_GLOBAL flag (vres->res for the source,
vres->vres for the destination) -- the same flag
calc_wanted_global_resvals()/calc_wanted_host_resvals() (resreq.c) already
dispatch on, so this is the existing rule applied consistently rather than
a new one. Also drops sched_fill_limit_requests()'s unused xce_scheduler_t*
parameter.
Full scheduler suite: 76/77 (the one failure, test_job_rusage_reconciles_
mem_reservation_against_usage, is bug-1762, pre-existing and unrelated).
Live-verified against a fresh single-node deploy.
Filed bug-1790 to track a known, deliberately out-of-scope residual gap:
a future global limit on a per-host resource other than "slots" would need
calc_wanted_host_resvals() to also run during the global phase for that
resource, which nothing currently does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
While live-verifying gitea#46, found and fixed a cascade of 5 previously
undetected bugs that together made every usergroup-scoped limit in
limits.yaml silently non-functional:
- Named usergroups/hostgroups (users.yaml/hosts.yaml) were never
registered into the scheduler's group managers (groupmgr_add() had zero
production callers), so any limit referencing one by name could never
resolve real membership.
- An unconfigured limit scope (e.g. no "projects:") arrived as an
empty-but-present capnp group, not NULL, so build_consumer() built a
real empty group (matching no one) instead of falling back to "all".
- Three separate call sites conflated key_find()'s negative "not found"
return with a real error, short-circuiting before ever reaching the
"instantiate on first use" logic right below them -- the same defect
class already fixed once before in this codebase (bug-1495).
- Two call sites passed &(sched->core.resmgr) where a plain handle was
expected, one level of indirection too many, causing RTTI's key_find()
to reject it as an invalid handle.
All five are live-verified: fresh cluster deploys now correctly match and
instantiate usage tracking for all 7 limits in limits.yaml on every job
submission, reproduced across multiple independent from-scratch redeploys.
A 6th, architecturally distinct issue remains open (bug-1789, logged, not
fixed): sched_fill_limit_requests() computes a job's consumption of a
slots-scoped global limit by reading the wrong request vector, so the
numeric cap itself still never binds. This needs a design decision on
intended cross-scope semantics, not a mechanical fix, so it's left for a
follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
ControlJob() set xce_jobctrl_t.action to the job_ctrl_type_t action code
(kill/stop/resume/checkpoint), but that field is actually the capnp
JobControl_u_which union discriminant -- a different enum entirely. This
made kill/resume/checkpoint hit request.c's -EINVAL default case, and made
stop collide with the checkpoint discriminant and read through the wrong
union member. Live-verified against a real deployed cluster: kill/stop/
resume/checkpoint now behave correctly with no apiserver crash.
Also additively accept a human-readable job ID ("{cluster}-{apiserver}-
{timestamp}-{seq}") on the job control endpoint, alongside the existing
numeric form xcecli already translates to -- lets a caller hit the REST
API directly without reimplementing xcecli's ID packing. Live-verified via
a direct curl call with the human-readable ID.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
modelJobToSubmit() never set job_startinfo_t.execuser when converting
a submitted job to the C struct sent to the controller/worker, so
every job ran as the worker's 'nobody' default and failed to exec
(exit 126). Default execuser to the submitting user via a new
resolveExecUser() pure helper, and free the newly allocated C string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
bug-1554: init_host_attrs() replaces hardcoded placeholder stubs with real
detection -- maxswap from sysinfo()'s totalswap (reusing the same syscall
snapshot already taken for maxmem), maxtmp from a fresh statvfs("/tmp")
total-capacity measurement. No synthetic fallback on detection failure, so
a system with swap genuinely disabled correctly reports 0.
bug-1776: the scheduler simulator's "info worker" assigned-jobs listing now
annotates a job consuming more than one "slots" unit (e.g. "jobid*2"),
using per-host allocation data that was already being walked but discarded.
Falls back to the pre-existing per-job listing when jobs on the same worker
have differing slot counts, since a single suffix on a compressed range
would be ambiguous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
/kiro-validate-impl returned GO: 21/21 requirement ACs covered, zero
boundary violations (parjob_allocator/seqjob_allocator/alloc_cache/
scheduler.c untouched, no revalidation trigger fired), zero blocked
tasks, cross-task integration chain (job_submit_data_t ->
apply_job_task_count() -> xce_primary_resreq_t -> scheduler dispatch
-> controller/worker launch) confirmed real end to end.
Closes two minor documentation gaps the audit found:
- tasks.md Implementation Notes now records tasks 2.1/2.2's colocated
unit tests (in pre-existing test_core_events.c/test_snapshot_save_
load.c) as a deliberate, reviewed File Structure Plan deviation.
- bug-1770's buglog.json entry updated from OPEN/NOT FIXED to FIXED,
referencing this feature's commits and end-to-end verification.
spec.json phase set to implementation-complete.
This closes out all 12 sub-tasks of the resreq-maxtasks-wiring
feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extends xce-test-multihost with a second scenario: a 2-element array
job (jobsub -a 2) where each element independently requests the same
nproc+1 task count as the plain-job scenario. Drives element [0]
through the identical RUN-then-DONE, both-hosts-started proof, and
confirms its jobinfo Slots: field matches the plain job's exactly
(Requirement 5.5). Traced the array-expansion path
(handler.c's sched_on_job_array()/clone_job_submit_data()) to confirm
no array-specific shortcut bypasses apply_job_task_count() -- each
element flows through the same core_build_job() ingest as a
standalone job.
Fixes a real bug (bug-1775) found while building this: an array
element's bracketed jobid (e.g. "<base>[0]") was misparsed by
grep -E's bracket-expression syntax, causing a false negative in the
per-host verification. Added ere_escape(), a no-op for plain
(non-bracketed) jobids.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Drive a real multi-slot parallel job through the actual production
dispatch path -- real sched_init(), real queue/worker/job ingest via
xce_core_push_events(), a real scheduling session (exercising task 5.2's
cache-clear wiring), the real sched_allocator() thread, cache lookup,
parjob_allocator's real my_alloc(), host_iter, and commit -- and confirm
the resulting job allocation genuinely spans 2 hosts with correct
per-host slot counts (Req 4.1, 4.2, 4.3).
Discovered and logged bug-1770: parse_resreq_primary() unconditionally
hardcodes primary->minTasks/maxTasks to 1 regardless of what a job
actually requests, and nothing in the real ingest pipeline ever copies
the job's own submitted task count (correctly threaded from the
apiserver's num_processors/maxnum_processors all the way to
xce_sched_job_t.mintasks/maxtasks) onto the nested xce_primary_resreq_t
fields the allocator dispatch actually reads. This predates the feature
by ~7 months and is out of this spec's scope to fix, but it means
parjob_allocator has zero live production path until a dedicated
follow-up lands. The test compensates with one clearly-labeled patch of
the real, already-ingested resreq's task-count fields immediately after
real parsing; everything else in the pipeline is unmodified production
code.
Also logged bug-1771 (xce_sched_fcfs's fcfs_del_job() dereferences
job->queue->name with no NULL guard) discovered while building this
task's fixture; worked around by registering the job's queue before
push, mirroring the apiserver's own pre-submission queue validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Wrap sched_make_primary_alloc_() -- the single choke point all three
primary-resreq shapes (standalone, compound-nested, alter-nested via
compound delegation) already funnel through -- with a cache lookup
before allocation and a cache insert on exhaustion. On a hit, pend the
job immediately via the same make_pendreason()/add_job_preason() path
every exhaustion case already uses, without invoking either allocator
(Req 7.3, 7.4, 7.5).
A nested primary has no real per-primary schedobj, so
sched_primary_cache_obj_() synthesizes one on the stack per dispatch for
cache-key purposes; verified memory-safe since alloc_cache.c's
build_key_() copies the fingerprint/identity by value synchronously
before the stack frame unwinds.
Log bug-1768 (compound/alter-nested primaries never get real slot/rusage
availability checks -- seqjob/parjob's prepare_allocation hooks no-op
whenever ctx->schedobj->tag != XCE_RESREQ_PRIMARY) and bug-1769
(sched_make_alter_allocation() type-confuses xce_compound_schedobj_t*
as xce_sched_obj_t*, a ~112-byte heap over-read, currently untested).
Both are pre-existing and out of this feature's scope; discovered and
confirmed while building this task's dispatch tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extensive requirements review found and fixed a real correctness bug
(host_iter shared across plan buckets instead of re-scanned fresh per
bucket, which would silently drop solvable multi-host allocations),
a self-contradiction around parjob_prepare_allocation's scope, a
plan/host-iterator init calling-convention mismatch, and an inverted
kvtab .cmp polarity that would have made the unsatisfiable-resreq
cache permanently non-functional. brief.md brought back into sync
with requirements.md's corrected architecture.
design.md and tasks.md generated and validated (kiro-validate-design
GO after two fixes; independent task-graph sanity review PASS).
Two pre-existing, out-of-boundary gaps found along the way logged as
bug-1766/bug-1767.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
on_scheduler_join()'s full job resync and sched_job_new_fun()'s
incremental per-commit push both hold c->sched.lock for their entire
body, making concurrent delivery of the same jobid structurally
impossible -- confirmed by direct code reading, no live repro needed.
This also fully explains the previously-unexplained "redelivery
storm" (up to 18x per jobid): every scheduler reconnect triggers a
full resync of the whole job set, and the scheduler's connection
already flaps repeatedly under heavy load (found earlier this
session), so N flaps produce N redundant resyncs with no race
involved. All scheduler-side and controller-side push-path candidates
are now exhausted; the actual stale-reader for j->submit remains
unfound after 9 sessions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Static tracing confirms the resreq/select-expression parser fully
copies/interns every string (no aliasing into the short-lived
job_submit_data_t), and the pend_updates dataq's enqueue/dequeue path
cannot return the same item twice. Neither explains the UAF. Updates
the running tally of ruled-out candidates so future sessions don't
re-chase them; the controller-side HOOK_DETAIL_JOB_NEW trace remains
the sharpest open lead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
An idle scheduler (nothing queued to publish) never noticed a dead
controller connection: sched_needs_rejoin was only ever set as a side
effect of an active client-library call, and an idle round makes none.
After any controller failover with nothing already in flight, jobs
submitted to the new primary stayed PEND indefinitely.
Adds xcecli_is_connection_broken() (libxce) -- a pure, socket-free read
of the transport's already-async-maintained dead/alive flag, set for
free by the existing RDHUP disconnect handler with zero application
calls. sched_check_pending_rejoin() now retries xcecli_join() on either
the existing reactive flag or this new passive signal, so a healthy
idle scheduler still makes zero extra network calls while a genuinely
dead connection self-heals within one ~1s round -- live-verified
against a 3-node HA cluster (rejoin fired ~300ms after new-primary
election with nothing submitted in between).
Also closes out bug-1735 (job memory/CPU usage reporting): confirmed
already fixed on this branch since 2026-08-30, just never marked in
the bug log.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Pure verification, no code changes. Deployed a live single-node
cluster and ran tests/regression/run.sh against a Meson-built xcecli
binary, then again via the still-present Makefile path -- both
produced identical 28 passed/1 failed/1 skipped results. Reviewer
independently reproduced this three times and traced the one
pre-existing failure (jobs submit's job-ID format regex) through
source and git history to a 2026-05-07 commit, four months before
this spec existed -- logged as bug-1765 (not fixed, out of scope).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
OpenWolf hook-maintained project bookkeeping, reflecting this
session's work across three shipped specs
(scheduler-sim-repl-verbosity, scheduler-sim-step-verbosity,
scheduler-sim-reconcile-terminal-guard), the go_build_improve merge,
and the xce-build skill fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c)
now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state),
alongside the existing job->accounts != NULL skip. A terminal job is
never searched for a matching recorded DISPATCH entry, so a leftover
or duplicate DISPATCH decision from a later replay round can no
longer resurrect allocs/accounts on a job that has already completed
- the exact mechanism behind info worker's assigned-jobs listing
showing a finished job as still consuming a worker's resources.
The exclusion mirrors job_state_regresses_terminal()'s existing
"terminal is permanent" invariant (core_events.c) rather than
inventing a new rule. The function's doc comment gains a paragraph
clarifying the terminal exclusion is complementary to, not a reversal
of, the existing PEND/RUN state-independence rationale that motivated
this trigger's original widening.
Both RED-phase tests (task 1.1) pass; the pre-existing PEND
eager-capture regression test and core_events.c's JOBS_PUSH
resync-merge terminal-state tests are unaffected. The analogous
resync-merge branch was investigated and confirmed already correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
