A resync redelivery (on_scheduler_join()'s full push, fired on every
scheduler reconnect) can race a job's own very-recent dispatch
confirmation and carry a stale, non-terminal state for it. The
duplicate-jobid merge in core_events.c trusted this claim
unconditionally, freeing the job's already-captured allocation and
regressing its state -- live-confirmed via a production scheduler.log
showing a resync WARN landing 16ms after the job's own RUN
confirmation, and matching a live-reported simulator symptom (a RUN
job silently missing from info worker's allocation listing).
Refuse a non-terminal claim when the existing job already has a real
allocation on record, mirroring the existing phantom-RUN guard's own
reasoning in the opposite direction; a genuinely terminal claim
(DONE/EXIT) still applies and releases the allocation correctly, same
as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
test_job_rusage_reconciles_mem_reservation_against_usage asserted
w->resInUse was already charged 4.0 for a JOBS_PUSH-dispatched RUN job,
immediately after the push with no round boundary simulated. That
assumption was wrong, not the scheduler: resInUse/sharedInUse is never
charged incrementally at JOBS_PUSH/dispatch time. It's fully re-derived
once per round by sched_account_jobs() (scheduler.c), which iterates
every currently-RUN job and calls sched_charge_account() on each one's
accounts -- called by both the real normal-mode driver (driver.c) and
the sim driver (sim_loop.c), always before anything else in that round
reads resInUse. Confirmed by reading the full JOBS_PUSH handler
(core_events.c) and builtin_fcfs_on_job_push(): neither ever charges or
uncharges resInUse, by design -- core_session.c's own doc comment on
core_begin_session_internal() already documents this "recompute from
source of truth each round" architecture.
Fix: the test now calls sched_charge_account(&core, tracked->accounts)
right after JOBS_PUSH, simulating the one round-boundary reconciliation
step sched_account_jobs() would perform for its single RUN job (that
function itself lives in scheduler.c, not white-box #included here, so
its single-job effect is reproduced directly via the same primitive it
calls internally). No production code changed.
Full scheduler suite: 77/77 Ok (was 76/77). Full xce-build.sh all asan
--incremental: 94/94 Ok, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLfB5uWRdmx2WiZSFC7Fm7
core_build_job() now calls apply_job_task_count() right after its
resreq parse succeeds and before core_build_schedobj() reads the
resreq's task count for resreq_fp — so a freshly-submitted multi-task
job's dispatched primary resreq reflects its real requested min/max
task count instead of the previous always-one-task hardcode.
Tracked follow-up (not blocking, see tasks.md Implementation Notes):
test_repl_breakpoints_inspection.c:735 hardcodes an incidental
"Tasks: min = 0, max = 0" byproduct of a fixture's parse_select()
failure; now correctly reports (1,1) per Requirement 1.2. Fix folded
into task 5.1's full-suite regression pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
bug-1738: a scheduler-side notify could clobber an already-terminal
job state. The controller's own terminal-transition guard
(job_state_is_terminal(), job.c) treats DONE/EXIT as permanent once
set -- apply_job_fail_transition()'s POST_EXEC/Q_POST_EXEC branch only
ever MERGES a JOB_STAT_PERR flag onto whatever DONE/EXIT already
holds, never re-deciding DONE vs EXIT after the fact. The scheduler
had no equivalent guard: HOOK_DETAIL_JOB_FAIL's schedFailJob notify
(sched_job_fail_fun, sched.c) unconditionally requests JOB_STAT_EXIT
with no regard for the job's current state, so a job that finished
successfully (DONE) and then hit an unrelated post-exec script failure
would have the scheduler's view forcibly clobbered back to EXIT.
Fixed with one shared helper, job_state_regresses_terminal(), applied
at both XCE_CORE_EVENT_JOB_STATE_CHANGE and the JOBS_PUSH duplicate-
jobid resync merge (a full resync can just as easily redeliver a stale
snapshot as a genuinely newer one) -- once a job is DONE/EXIT, neither
path can move it to a different terminal state.
bug-1739 (Symptom 1, "phantom RUN"): the bug-1734 fix's own JOBS_PUSH
merge adopted a redelivery's RUN state without transferring the real
allocation data core_build_job() had already attached to the
redelivered clone -- leaving existing->state==RUN with
existing->allocs==NULL, a job the scheduler's own resource/slot
bookkeeping has no record of. Live-reproduced: `info summary` showed
30 elements of an array marked RUN, but `info worker <name>` only
listed 22 real slot assignments for a 22-core worker. Fixed by
transferring job->accounts/job->allocs onto the existing job on a RUN
merge (freeing whatever it previously held), and refusing the merge
entirely (WARN-logged, existing state kept) if the redelivery carries
no allocation data -- treating an unbacked RUN claim as untrustworthy
rather than manufacturing a phantom.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid handling (sched_add_job()
EEXIST) assumed a redelivery is always a harmless no-op -- true for
bug-1714's array-push TOCTOU race (both copies start identically
PEND), but not for a scheduler reconnect's full resync
(on_scheduler_join() -> xcesched_push_jobs(), sched.c), which
re-sends every currently-committed job through this same path
regardless of how far its state has moved on since this scheduler
last saw it.
Live-reproduced on a real 3-node HA cluster + worker: the scheduler's
outbound xcecli connection repeatedly failed over under a 50-element
array-job load; each reconnect's resync redelivered jobs that had
already finished (PEND/RUN -> DONE) while disconnected -- confirmed
via ctrl-1.log's existing bug-1731 "c->sched.ep is NULL, notify
SKIPPED" diagnostic firing for the same jobids during the gap, making
the resync that job's only remaining chance to reach the scheduler.
Discarding it unconditionally permanently froze the tracked job at
its stale state: controller shows DONE, scheduler stuck at RUN/PEND
forever.
Fix: on EEXIST, look up the existing tracked job and merge in the
redelivery's state if different (same terminal-transition
accounts/allocs cleanup XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE
already do), instead of silently dropping it. Only state is merged;
broader field reconciliation is out of scope, same as the existing
allocs/accounts scope note this code path already carries. Bundled a
requested logging fix at the same site (raw jobid -> fmt_jobid()).
Verified live end to end: rebuilt scheduler, reran the same load
against the same cluster (reconnect storm still occurred, proving the
fix doesn't depend on suppressing it), both test arrays now show
consistent DONE state matching the controller exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Found while chasing bug-1714's crash (root cause still open -- see that
entry): two related, independently-confirmed defects, not proven to be
bug-1714's own cause but real bugs on their own merits.
xce_core_push_events()'s JOBS_PUSH case never checked sched_add_job()'s
return value -- if key_add() failed (duplicate jobid), the freshly-built
xce_sched_job_t was handed to picker_iface->on_job_push() anyway, even
though it was never registered in core->jobs. An orphaned job the
picker knows about but sched_get_job() can never find again. Now checks
the return, frees the orphan (at core_build_job()'s own established
cleanup granularity -- xce_sched_job_t has no complete destructor
anywhere in this codebase, kill_sched_job() only frees ->rusages, so
this intentionally doesn't invent a more thorough one), and skips the
picker call.
This makes it safe to fix (not the other way around): sched_on_job_array()'s
per-element dedup check (handler.c) has a TOCTOU race -- a job is only
added to core->jobs later, by the driver thread, so a redelivered array
push arriving on the request thread before the driver catches up isn't
caught. Documented as a best-effort optimization, not a correctness
boundary -- the actual authoritative uniqueness check is sched_add_job()'s
key_add(), which the JOBS_PUSH fix above now handles correctly.
TDD: new test reproduces the duplicate-jobid path (RED: on_job_push
fired twice, Expected 1 Was 2), GREEN after the fix. Full scheduler
regression: 54/55, only the known bug-1561-adjacent flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOB_UPDATE (core_events.c) is the only currently-live path
a controller job-finish notification reaches, but unlike its sibling
XCE_CORE_EVENT_JOB_STATE_CHANGE case, it never released job->accounts/
job->allocs on a terminal transition, leaking them for the scheduler
process's lifetime. Mirrors the existing JOB_STATE_CHANGE cleanup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
info worker all/<name> in the interactive/sim REPL initially printed
full per-resource values correctly, then lost them after a later
WORKERS_PUSH event replayed. core_apply_worker_resvals()'s per-resval
loop only guarded against nv==NULL/nv->name==NULL; a resval entry with
a NULL nv->val (a legitimate "no update this round" marker some
producers send) fell through and overwrote the worker's existing good
value with nothing, instead of being skipped -- same defensive pattern
already used elsewhere in this function, just missing here. A related
count-doubling gap in the same event path is fixed by switching an
iteration bound from `.count` to `.caps`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Bugs fixed, discovered across a day of live-operator testing against a
real deployed cluster:
- bug-1584: clock anchors never initialized to real wall-clock values,
timestamps showed system uptime instead of Unix epoch.
- bug-1585: heap-use-after-free in the normal-mode driver's periodic
resval update path -- core_events.c's RESVALS_FULL/_DELTA handlers do a
shallow zarray_put() of the wire-decoded xce_val_t*, but driver.c's
release_translated_payload() unconditionally freed the same decoded
array right after, so the next periodic report for the same
worker+resource triggered a UAF read via kill_sched_resval(). New
regression test: test_resval_ownership.c.
- bug-1587/1588/1591: raw-ns/raw-uint64 jobid/timestamp prints across the
REPL/SIM family converted to the same human-readable formatting already
used elsewhere (fmt_jobid(), timestamp formatting).
- bug-1590: repl_info.c's XCE_VAL_NUMERIC checks corrected to
Value_u_fvalue, fixing `info worker`/`info resource` falling back to
0.00 for real numeric resvals.
task 12.5 (Req 5.22 amendment) -- five REPL gaps surfaced by a live
operator session, each with its own bug id:
- bug-1594: `continue`/`step` reaching EOF with no breakpoint hit now
re-enters the REPL via xce_repl_on_replay_complete() instead of exiting
the process; step/continue issued after replay is already complete
re-prompt instead of silently exiting.
- bug-1595: `step`/`continue` now echo each input event (not just
decisions) via repl_print_event_summary(), in real event/decision
order.
- bug-1593: `info resource <name>|all` now reports resource definition
metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, read
from core->sharedInst, instead of aggregating host-based resources
across every worker.
- bug-1596: repl_split_tokens() strips one matching pair of surrounding
quote characters from each token.
See requirements.md's Req 5 AC 22 "Amendment (2026-08-19)" for the full
rationale on each of the five task 12.5 gaps.
Full scheduler suite green (41/42; the one remaining failure,
test_recordstart_symlink_escape_replies_error, is the pre-existing
unrelated bug-1560).
xce_scheduler -i -s <replay>: "info worker <name>" showed
total=0.00 available=0.00 for every resource, even though the live
cluster (xce_test resval/workerlist) reported real non-zero values
for the same worker.
Two independent root causes, both live-traced end-to-end against a
real deployed cluster (value-level TRACE at every hop: controller
push, scheduler apply, snapshot save, snapshot load/decode):
1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH
called sched_add_worker() -- a plain key_add(), add-only --
unconditionally on every push. A worker connecting triggers
multiple WORKERS_PUSH events in quick succession (an early push
before the controller finishes writing real capacity values,
carrying stale/zero data, followed moments later by a push
carrying the real values). The FIRST push always won the race;
every push after it for the same worker name silently failed
-EEXIST and its resource values were discarded -- not just at
join, for the rest of the process's lifetime (unlike jobs, there
was no "update an existing worker" event path at all). Fixed via
a new core_upsert_worker() (mirrors the existing
core_upsert_queue() add-or-update precedent): looks up the
worker by name first, updates status+resvals in place if found,
else builds+adds a new one as before.
2. src/scheduler/src/repl_info.c: repl_info_worker_one()/
repl_info_resource_one() checked `xce_val_t.which ==
XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1)
is a resource-DEFINITION value-type constant
(xce_resource_t.valtype's own enum), a completely different enum
from xce_val_t.which's real domain, Value_u_which
(controller.capnp.h, Value_u_fvalue=5). Every producer in this
codebase sets xce_val_t.which to Value_u_fvalue, never
XCE_VAL_NUMERIC, so the comparison was always false -- every
genuinely-numeric resval displayed as 0.00 unconditionally,
independent of fix (1). Corrected all 4 occurrences to
Value_u_fvalue.
Both fixed via TDD (RED test before each fix, GREEN after):
test_core_events.c's test_workers_push_twice_updates_existing_
worker_in_place (two pushes, same name, different values -- both
must apply) and test_repl_breakpoints_inspection.c's
test_info_worker_reports_resource_values (a real numeric resval
must display, not fall back to 0.00).
Live-validated by reproducing the user's exact original scenario
end-to-end (deploy a real cluster, record, `xce_scheduler -i -s
<replay>`, `info worker <name>`) -- now shows slots=22.00,
maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching
the user's own live xce_test resval/workerlist numbers exactly.
Full scheduler suite green (41/41, excluding pre-existing unrelated
bug-1560); controller suite green (1102/1104, the same pre-existing
bug-1561 pair, confirmed unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
