sim_reconcile_unaccounted_jobs() (the mechanism that replays historical
DISPATCH decisions in --sim mode) reconstructed a job's allocation but
never called the picker's on_dispatch() hook, unlike the live allocator
path which always does. The job therefore stayed a live candidate in the
picker's own pendJobs bookkeeping, and sim_picker_thread_fn() (which
unconditionally PENDs whatever it is still offered) kept re-offering and
re-PENDing it every subsequent round -- live-observed as an already-
dispatched job receiving a bogus PEND decision (bug-1759: 22/50 jobs in
one batch; bug-1779: 5/10 array-job elements).
This directly contradicts scheduler-sim-decision-parity's own design.md,
which already documented "the picker never re-decides an already-resolved
job" as the correct assumption -- the implementation had drifted from it.
Fixed by calling on_dispatch() immediately after reconstructing the job's
allocation, mirroring sched_allocator()'s own live-dispatch ordering.
test_sim_verify_no_false_mismatch_across_round_separated_dispatch's own
fixture had baked in the stale-PEND symptom as "expected" data; updated to
match the corrected behavior (dispatch-only round 1, empty filler rounds).
Both design docs updated in place with correction notes.
info worker's assigned-jobs listing gated only on job->allocs != NULL,
so a job already at a terminal state (DONE/EXIT) could still appear as
"assigned" if its allocs pointer was ever left stale -- live-observed via
a real captured replay, three DONE jobs kept showing as assigned to a
worker despite the worker's own slot-usage accounting correctly excluding
them. Root cause of the stale allocs itself is not yet pinned (suspected
snapshot save/load inconsistency); this adds the same defensive "terminal
is permanent" exclusion job_state_regresses_terminal()/sim_reconcile_
unaccounted_jobs() already apply elsewhere, at the display layer.
bug-1554: init_host_attrs() replaces hardcoded placeholder stubs with real
detection -- maxswap from sysinfo()'s totalswap (reusing the same syscall
snapshot already taken for maxmem), maxtmp from a fresh statvfs("/tmp")
total-capacity measurement. No synthetic fallback on detection failure, so
a system with swap genuinely disabled correctly reports 0.
bug-1776: the scheduler simulator's "info worker" assigned-jobs listing now
annotates a job consuming more than one "slots" unit (e.g. "jobid*2"),
using per-host allocation data that was already being walked but discarded.
Falls back to the pre-existing per-job listing when jobs on the same worker
have differing slot counts, since a single suffix on a compressed range
would be ambiguous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
test_info_job_reports_state_queue_priority now expects "Tasks: min =
1, max = 1" instead of the pre-fix "min = 0, max = 0" incidental
byproduct (a fixture's parse_select() failing on an unregistered
"type" resource used to leave the primary's task-count fields at
memset-zero; apply_job_task_count() now correctly overwrites them to
the job's real submitted default of 1/1). This was the one deferred
item tracked in tasks.md's Implementation Notes since task 2.1.
Full-suite regression confirmed clean: scheduler/controller/worker
suites pass except the pre-existing, unrelated bug-1758 (predates
this feature, attributed to a separate spec). A full /xce-build all
also surfaces bug-1723, a pre-existing lib-suite fixture-teardown
leak dating to before this feature and outside Requirement 5.1's
named scope (scheduler/controller/worker) -- confirmed untouched by
any of this feature's commits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
test_parjob_e2e_multislot.c's standing "bug-1770" assertion now
expects the correct, real primary->minTasks/maxTasks (1/JOB_MAXTASKS)
instead of the buggy always-1 value it used to document. The
PATCH_MAXTASKS_FOR_BUG_1770 block that manually overwrote the
already-ingested resreq's task-count fields is removed entirely --
the full multi-host allocation scenario (2-host commit, per-host
slots) now runs end to end through unmodified real ingest. Header
comment updated to record that bug-1770 is fixed by this feature
rather than describing a live gap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Extends test_maxtasks_wiring_ingest.c with two tests: (1) a resource-
request update never changes a job's already-dispatched task count,
because core_events.c's live JOB_UPDATE handler never rebuilds
job->schedobj (Requirement 1.4); (2) two otherwise-identical jobs
differing only in requested task count don't collide in the
unsatisfiable-resource-request cache on the fresh-ingest path
(Requirement 5.4).
Adds a new file, test_maxtasks_wiring_sim_driver_parity.c, proving the
simulation driver's real static translator chain
(sim_process_replay_event_message()/replay_event_to_core_event())
applies this feature's wiring identically to the real driver -- a
genuine white-box test of the real sim_loop.c code (following
test_sim_event_loop.c's established #include technique), not a mirror
of its logic. This was split into its own file because
test_maxtasks_wiring_ingest.c already links the full scheduler_srcs
object list for its own task-3.1 test, which collides at link time
with white-box #include-ing core_events.c/sim_loop.c in one
translation unit; see tasks.md's Implementation Notes.
No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds test_maxtasks_wiring_ingest.c proving: (1) fresh ingest's
dispatched primary resreq reflects a real multi-task job's requested
count via the real core_build_job()/JOBS_PUSH path; (2) fresh ingest
and snapshot-restore ingest derive the identical task count for an
equivalent job, asserted both against expected constants and via
direct cross-comparison (Requirement 5.6); (3) a single-task job
reaches identical real host-selection/allocation behavior through the
live allocator dispatch chain, proving the feature is a true no-op
for maxTasks==1 (Requirements 3.1/3.2). No production code changed --
tasks 2.1/2.2 already wired the fix; this adds cross-call-site and
behavioral proof neither of those isolated tests attempted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
snapshot_load.c's job-restore path now calls apply_job_task_count()
with the job's already-restored mintasks/maxtasks, right after its
resreq parse succeeds and before snapshot_build_schedobj() reads the
resreq's task count for its fingerprint. This is the simulation/replay
driver's only live call site (xce_sched_core_load_snapshot(), reached
via --sim), independent of and parallel to task 2.1's fresh-ingest
wiring in core_events.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
core_build_job() now calls apply_job_task_count() right after its
resreq parse succeeds and before core_build_schedobj() reads the
resreq's task count for resreq_fp — so a freshly-submitted multi-task
job's dispatched primary resreq reflects its real requested min/max
task count instead of the previous always-one-task hardcode.
Tracked follow-up (not blocking, see tasks.md Implementation Notes):
test_repl_breakpoints_inspection.c:735 hardcodes an incidental
"Tasks: min = 0, max = 0" byproduct of a fixture's parse_select()
failure; now correctly reports (1,1) per Requirement 1.2. Fix folded
into task 5.1's full-suite regression pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Drive a real multi-slot parallel job through the actual production
dispatch path -- real sched_init(), real queue/worker/job ingest via
xce_core_push_events(), a real scheduling session (exercising task 5.2's
cache-clear wiring), the real sched_allocator() thread, cache lookup,
parjob_allocator's real my_alloc(), host_iter, and commit -- and confirm
the resulting job allocation genuinely spans 2 hosts with correct
per-host slot counts (Req 4.1, 4.2, 4.3).
Discovered and logged bug-1770: parse_resreq_primary() unconditionally
hardcodes primary->minTasks/maxTasks to 1 regardless of what a job
actually requests, and nothing in the real ingest pipeline ever copies
the job's own submitted task count (correctly threaded from the
apiserver's num_processors/maxnum_processors all the way to
xce_sched_job_t.mintasks/maxtasks) onto the nested xce_primary_resreq_t
fields the allocator dispatch actually reads. This predates the feature
by ~7 months and is out of this spec's scope to fix, but it means
parjob_allocator has zero live production path until a dedicated
follow-up lands. The test compensates with one clearly-labeled patch of
the real, already-ingested resreq's task-count fields immediately after
real parsing; everything else in the pipeline is unmodified production
code.
Also logged bug-1771 (xce_sched_fcfs's fcfs_del_job() dereferences
job->queue->name with no NULL guard) discovered while building this
task's fixture; worked around by registering the job's queue before
push, mirroring the apiserver's own pre-submission queue validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Clear the cache inside core_begin_session_internal() so a job cached as
unsatisfiable in one session is re-evaluated fresh in the next (Req 7.2).
An initial weak-symbol cast-based approach was rejected: it corrupted
xce_sched_sim_driver_t.alt_events_file, since core_begin_session_internal()
is shared between driver.c (real xce_scheduler_t) and sim_loop.c (a
structurally different, deliberately isolated sim driver struct with no
cache field at the same byte offset) -- the strong cache_clear() symbol
won unconditionally for both, invisible to ASAN and untested by any
test_sim_*.c file.
Fixed via a driver-registered callback: xce_sched_core_t gains a
cache_clear_cb hook (NULL by default, per xce_sched_core_init()'s
full-struct memset), set only in main.c's driver-mode branch --
mirroring the existing on_decision/xce_core_set_decision_cb() pattern.
--sim mode returns before that registration is ever reached, so its hook
stays NULL and core_begin_session_internal()'s guarded call is a no-op.
The real xce_scheduler_t-casting adapter lives in alloc_cache.c rather
than driver.c, since ~11 tests white-box-include driver.c without
linking alloc_cache.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Wrap sched_make_primary_alloc_() -- the single choke point all three
primary-resreq shapes (standalone, compound-nested, alter-nested via
compound delegation) already funnel through -- with a cache lookup
before allocation and a cache insert on exhaustion. On a hit, pend the
job immediately via the same make_pendreason()/add_job_preason() path
every exhaustion case already uses, without invoking either allocator
(Req 7.3, 7.4, 7.5).
A nested primary has no real per-primary schedobj, so
sched_primary_cache_obj_() synthesizes one on the stack per dispatch for
cache-key purposes; verified memory-safe since alloc_cache.c's
build_key_() copies the fingerprint/identity by value synchronously
before the stack frame unwinds.
Log bug-1768 (compound/alter-nested primaries never get real slot/rusage
availability checks -- seqjob/parjob's prepare_allocation hooks no-op
whenever ctx->schedobj->tag != XCE_RESREQ_PRIMARY) and bug-1769
(sched_make_alter_allocation() type-confuses xce_compound_schedobj_t*
as xce_sched_obj_t*, a ~112-byte heap over-read, currently untested).
Both are pre-existing and out of this feature's scope; discovered and
confirmed while building this task's dispatch tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Task 4.3's multi-host merge itself was already implemented and verified
by task 4.2. The one unproven clause -- job-level shared-resource
allocation computed once regardless of host count -- is now covered by
test_shared_resource_debited_once_regardless_of_host_count, using a
PER_JOB (not PER_TASK) global resource so a per-bucket-repeated
computation can't coincidentally sum to the correct total. Mutation
testing (a deliberately injected duplicate per-host shared debit)
confirms the test fails exactly as expected; production code required
no change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replace parjob's return-0 stub with a real multi-host allocator: for each
candidate plan, walk buckets in the given (descending) order, claiming a
distinct host per bucket via a fresh host_iter excluding hosts already
claimed elsewhere in the same plan attempt. On a bucket failure, reverse
every earlier successful claim's provisional debit via
sched_ctx_uncommit_host() before advancing to the next plan (Req 4.2, 4.4,
4.5, 4.6, 4.7).
Fix parjob_prepare_allocation() to compute each bucket's want from that
bucket's own slot count (ctx->nslots) instead of a fixed value, and reset
ctx->alloc_req.host at the top of each bucket's scan so host_iter's
pre-filter can't inherit a stale want from a differently-sized prior
bucket (safe: sched_try_alloc_on_host_() always repopulates it from the
real per-bucket want before its own authoritative check runs).
Add a maxTasks<=1 guard matching Req 4 AC1's precondition, needed because
a snapshot-restored job with unpopulated maxTasks==0 (pre-existing,
tracked separately as bug-1767) would otherwise reach parjob's now-real
allocator instead of being silently ignored as before.
On full success, merges every claimed bucket into one xce_job_alloc_t via
make_job_alloc()'s existing runtime-sized host array, satisfying task
4.3's multi-host commit scope as well.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add the real xce_alloc_plan_iter_t for parjob: for host-count k=1..maxTasks
(ascending, most-preferred-first), generate one balanced partition of
maxTasks into k buckets (floor/ceil split, remainder to the first rem
buckets), which is descending by construction. Pure function of maxTasks
alone -- never dereferences sched -- with a real pthread_testcancel()
cancellation point and stable post-exhaustion NULL (Req 2.1-2.6, 4.1).
my_alloc()/parjob_prepare_allocation() remain untouched stubs; wiring the
iterator into real allocation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add a dedicated equivalence suite (test_seqjob_rewrap_equivalence.c)
covering multi-host fixtures the trivial single-host test doesn't reach:
select-filter-only rejection, resource-only rejection, a combined
select-then-resource-then-success ordering, and per-host debit
correctness. The iteration-order test is grounded in the deleted
pre-rewrap sched_collect_workers() code path (verified via git show
against the parent commit) and the unmodified xlist.c primitive, not
merely in host_iter.c's own traversal, so it proves equivalence to the
old behavior rather than internal self-consistency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Restructure seqjob's my_alloc() as a trivial single-bucket case over the
shared xce_alloc_plan_iter_t/xce_host_iter_t contracts, replacing its own
inline worker-collection loop. Guard seqjob_prepare_allocation() to
maxTasks == 1, closing the fan-out gap for the parallel allocator's later
fix (Req 9.1, 4.6).
Export sched_try_alloc_on_host_() from resource.c (drop static/va_list,
plain (worker, sched, ctx) signature) so seqjob calls the real per-host
try-alloc leaf instead of a private duplicate; delete the now-dead
sched_try_alloc_on_hosts() variadic wrapper. Also wires host_iter.c into
the production scheduler_srcs list (task 2.1 only added it to testsuite).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds alloc_cache.c: xce_cache_key_t, cache_key_ops (the corrected
three-way, 0-means-equal .cmp -- an inverted-boolean version was a
real bug found during design review), and cache_lookup/insert/clear
against a real kvtab_t (constructed matching xce_select.c's
sel_node_table_init() precedent). Cache values are a boxed reason-
category int, not a bare hit/miss sentinel, satisfying Req 7 AC5's
requirement that a hit reproduce the original pend-reason category.
unsatisfiable_cache lives on xce_scheduler_t per Req 7 AC1.
Task 2.2 (scheduler-allocation spec). Requirements 7.1, 7.3, 7.4, 7.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds host_iter.c: the one shared xce_host_iter_t implementation,
lazily filtering sched->core.workers by select expression
(sched_meet_host_selection, unchanged) and resource availability
(sched_check_host_satisfy, unchanged), with a real cancellation
point per candidate visited. xce_alloc_plan_t gains a ctx field
(xce_sched_ctx_t*) so the iterator can reach the scheduler/select
expression/staged resource-want -- additive to task 1.1's approved
interface, NULL-guarded at init(). Test proves the core design
property this spec exists to establish: a host too small for one
search remains a valid candidate for a later, smaller search via a
fresh iterator instance over the same worker set.
Task 2.1 (scheduler-allocation spec). Requirements 3.1-3.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds sched_ctx_uncommit_host() (resource.c), the structural inverse
of sched_try_alloc_on_host_()'s existing XVEC_ADD_FLOAT commit call,
scoped to ctx->uncomitted only per requirements.md's Rollback scope
discussion. Test exercises the real sched_ctx_get_uncomit_host()
(schedobj.c) via white-box #include rather than a private
reimplementation, matching test_picker_config_fields.c's established
precedent.
Task 1.4 (scheduler-allocation spec). Requirement 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds xce_alloc_mode_t (XCE_ALLOC_NORMAL=0, XCE_ALLOC_PREEMPT, with
commented-out RESERVE/BACKFILL placeholders) and alloc_mode on
xce_sched_ctx_t, zero-defaulted by sched_ctx_init()'s existing
memset. Purely additive: nothing branches on the field yet -- this
spec only declares the extension point scheduler-preemption needs.
Task 1.3 (scheduler-allocation spec). Requirement 9.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Defines xce_alloc_plan_t (partition of a job's task count into
per-host slot buckets), xce_alloc_plan_iter_t, and xce_host_iter_t
(init/next/fini vtables, stack-allocate-then-init-in-place convention
matching sched_ctx_init) as reusable abstractions any allocator can
adopt. Proven usable by a trivial no-op reference implementation and
a fresh-per-bucket host_iter composition test, no allocator wired in
yet.
Task 1.1 (scheduler-allocation spec). Requirements 2.1, 3.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
An idle scheduler (nothing queued to publish) never noticed a dead
controller connection: sched_needs_rejoin was only ever set as a side
effect of an active client-library call, and an idle round makes none.
After any controller failover with nothing already in flight, jobs
submitted to the new primary stayed PEND indefinitely.
Adds xcecli_is_connection_broken() (libxce) -- a pure, socket-free read
of the transport's already-async-maintained dead/alive flag, set for
free by the existing RDHUP disconnect handler with zero application
calls. sched_check_pending_rejoin() now retries xcecli_join() on either
the existing reactive flag or this new passive signal, so a healthy
idle scheduler still makes zero extra network calls while a genuinely
dead connection self-heals within one ~1s round -- live-verified
against a 3-node HA cluster (rejoin fired ~300ms after new-primary
election with nothing submitted in between).
Also closes out bug-1735 (job memory/CPU usage reporting): confirmed
already fixed on this branch since 2026-08-30, just never marked in
the bug log.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch,
mirroring bug-1764's actual live-observed shape: a worker with one job
driven to a terminal state and two jobs genuinely still RUN, all real
allocations to the SAME worker. After reconciling a stray DISPATCH
entry for the terminal job's id, info worker excludes it while still
listing the two running jobs, and info summary's running count agrees.
Round 1 review rejected an initial version for using a mismatched host
string on the stray entry ("workerarr-stray" instead of "workerarr"),
which made the info worker/info summary assertions pass vacuously
regardless of whether the fix existed, since repl_info_worker_one()
filters assigned jobs by an exact host-string match. Round 2 fixed the
host string and confirmed via an independently-reproduced mutation
test (fix reverted, the test's own accounting-level pre-check
bypassed) that the REPL-level assertions now genuinely, independently
detect the bug.
design.md's Testing Strategy is corrected: it originally said no new
test was needed, reasoning the existing display-path suites already
covered this; tasks.md's own task 2.2 correctly required a
purpose-built test for this exact scenario instead.
Full scheduler test suite: 63/64 OK, sole failure the pre-existing,
unrelated bug-1758. This is the final task of the
scheduler-sim-reconcile-terminal-guard spec; all implementation tasks
(1.1-2.2) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds two failing tests formalizing bug-1764's reproduction: a job
driven to JOB_STAT_DONE (and, separately, JOB_STAT_EXIT) via a real
xce_core_push_events() JOB_STATE_CHANGE call correctly has its
allocs/accounts nulled by the existing end-of-life cleanup, but
sim_reconcile_unaccounted_jobs() then wrongly resurrects both fields
when presented with a stray DISPATCH entry for that same jobid, since
its trigger checks only job->accounts == NULL with no state check.
The pre-existing eager-capture regression test,
test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
already covers Requirement 1.2's "PEND job's early DISPATCH still
captured" case explicitly and needed no duplicate. No production code
touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 6 failing tests for the Event Echo Preamble Deduplicator
(Requirements 1.1-1.4, 2.2): a second echoed event sharing its
predecessor's rendered timestamp+session omits its preamble; two
events across a session boundary both print their own preamble; the
critical raw-vs-rendered nuance (Requirement 1.3) - two events whose
raw nanosecond timestamps genuinely differ but whose rendered,
second-granularity timestamp and session are identical still dedupe,
with a fixture-premise self-check proving the two raw values really
do render identically; the first echoed event in a fresh session
always prints its preamble; a structural event under continue mode
after suppressed routine events follows the same rule; event content
and ordering are unaffected regardless of the preamble decision.
Lands in a new test_repl_step_verbosity.c (1010 lines, 6 tests) rather
than extending test_repl_event_echo.c (705 lines) - proactive split
per this file family's established practice, since appending would
have pushed that file well past the project's ~1000-line trigger.
Registered in testsuite/meson.build. No production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds failing tests for Requirement 7 (Discoverable Help and Breakpoint
Event Types): help's four labeled sections in correct order, one
consistent column width across every listed command, info config's
placement alongside its info siblings, set units's placement alongside
set speed/output/verify, the break line's complete BP_KIND_TABLE-derived
event-type listing, and break's own unrecognized-event-type error
listing the same complete set. Each new test hardcodes BP_KIND_TABLE's
current 12 tokens rather than referencing the not-yet-implemented
repl_bp_kind_names() accessor, so RED comes from genuine assertion
failures against today's behavior rather than a link error.
New tests for Requirements 7.1-7.4 land in a new test_repl_help.c
rather than test_repl_execution_control.c (949 lines going in, within
reach of this spec's own ~1000-line split trigger already flagged at
task 2.1) - proactive split per this spec's established practice,
documented in design.md. break's Requirement 7.5 error-message test
lands in test_repl_breakpoints_inspection.c exactly as originally
planned, alongside the pre-existing test_break_unrecognized_event_type_rejected
(left unmodified).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Rewrites info worker's/info resource's resource-listing loops from a
flat, unconditional fprintf into a 3-pass, section-grouped table
(Consumable/Capacity, Load Metrics, Static Hardware Attributes),
units_mode-aware value rendering (raw/human/both, including swp's
MB->bytes conversion), whole-number formatting with no trailing .00
outside the consumable value=/used= pair, and jobid-range-compression
adoption for info worker's assigned-job listing (Requirements 6.4-6.6,
4.5). All 10 of task 8.1's tests pass. Preserves every named prior-bug
invariant (bug-1727's avail_v-as-total, bug-1613's consumable/
non-consumable split, bug-1593's GLOBAL/SHARED filter, bug-1688's
.caps-bound iteration).
Fixes one pre-existing test (bug-1688, predating this spec) whose
"value=5.00" assertion for a whole-number non-consumable resource
directly conflicts with Requirement 6.5's intentional no-trailing-.00
change - same class of fix as task 4.2's pre-existing-test updates.
repl_info.c grew from 985 to 1267 lines (58% over the 800-line cap,
worse than any other file this spec touched) - but unlike repl.c/
repl_break.c, it was already 23% over cap before this spec began.
Review empirically confirmed a narrow helper-only extraction doesn't
achieve compliance (still 41% over), and a compliance-achieving split
would mean restructuring ~690 lines of pre-existing display logic
this feature doesn't otherwise touch - a larger scheduler-core-level
decision, accepted as a residual rather than folded into this task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 10 tests in a new test_repl_resource_table.c covering info
worker's/info resource's resource-listing table/grouping layout,
units_mode-aware rendering (raw/human/both, including swp's MB->bytes
conversion and a no-mutation check on the underlying stored value),
whole-number-vs-fractional formatting, and info worker's assigned-job
listing adopting the real repl_fmt_jobid_range() (Requirements
6.4-6.6, 4.5). All 10 fail as expected against today's flat,
unconditional per-resource fprintf loop and uncompressed listing.
Found and fixed a genuine ASAN stack-use-after-return in the test's
own fixture code (not production): WORKERS_PUSH's handler moves a raw
pointer into a job's resvals rather than deep-copying, so the fixture
must heap-allocate rather than stack-allocate the structures it feeds
through that path - independently reproduced by review.
New file rather than extending test_repl_breakpoints_inspection.c:
landing this content there first pushed it to 3215 lines, 3.2x this
file family's own ~1000-line split trigger already used to justify
splitting test_repl_round_rollup.c/test_repl_event_echo.c for
structurally identical reasons.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements repl_is_bytes_valued_resource()/repl_format_bytes_human()
(mirroring xce_test's confirmed is_bytes_valued_resource()/
format_bytes_human() precedent exactly) and the units_mode setting
(explicitly defaulted to both, not left to zero-init) wired into
set units raw|human|both alongside the existing set speed/output/verify
branches (Requirements 6.1-6.3). All 8 of task 7.1's tests pass.
Split repl_format_ts_human(), the Jobid Range Compressor (task 1.2),
and these two new helpers out of repl.c into a new repl_fmt.c: adding
them in place pushed repl.c to 929 lines (16.1% over the 800-line
cap), and unlike task 6.2's single-function split attempt (which
didn't even achieve compliance), bundling these three pure-formatter
groups together does - repl.c settles at 587 lines, repl_fmt.c at 402,
both comfortably under cap. Required a matching #include line in the
20 test files that white-box #include repl.c, plus extending
test_sim_isolation.c's forbidden-include/symbol scan to cover the new
file, following the repl_rollup.c split precedent from task 2.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 5 pure-function tests in a new test_repl_resource_formatting.c
for repl_is_bytes_valued_resource()/repl_format_bytes_human()
(classification, adaptive B/KB/MB/GB/TB/PB scaling including exact
boundary values, swp's MB->bytes conversion), and 3 tests in
test_repl_snapshot_params.c for set units raw|human|both parsing
(Requirements 6.1-6.3). All fail as expected: link failure for the
not-yet-existing helpers, compile failure for the not-yet-existing
units_mode field/enum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 2 tests in a new test_repl_prompt_context.c covering the dynamic
prompt's plain-before-first-session baseline and its session/timestamp
content in both step and continue modes (Requirement 5.5).
Required genuinely new test infrastructure: libedit's readline() only
writes its prompt text when both stdin and stdout are real ptys, which
no existing REPL test in this codebase's freopen()-from-file harness
can observe. Built a PTY-based capture technique (openpty(), ECHO
disabled via termios) instead - independently reproduced via a
standalone probe and stress-tested 15x with zero flakiness during
review.
New file rather than extending test_repl_execution_control.c (already
at 948 lines, over the 800-line cap), per this spec's established
split precedent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Gates sim_print_decision() so the bare per-decision line no longer
prints under continue mode, making the Round Rollup's summary the
sole source of decision output (Requirement 2.3, closing bug-1610).
step mode and non-interactive/headless runs are completely
unaffected (Requirements 5.1, 5.4). Implemented as a file-local
inline wrapper in sim_loop.c rather than repl.h, since the gated
function is static to that file and repl.h can't see repl_state's
internals without leaking them to every other repl.h consumer.
Fixes 3 pre-existing tests (predating this spec, from scheduler-core)
that pinned the now-intentionally-changed bare-line behavior under
continue mode - each fixed differently based on its actual purpose
rather than uniformly: two kept continue mode with their anchor
switched to the Round Rollup's summary (since they specifically test
continue-mode breakpoint interruption, which step mode would
trivialize), one switched to step mode (since its actual invariant -
print-before-breakpoint-check call order - is mode-independent and
continue mode no longer has any observable anchor for it).
Review caught a second, unguarded sim_print_decision() call site in
sim_reconcile_unaccounted_jobs() (added by an earlier, separately-
merged spec) that design.md's boundary never accounted for - per the
review, this is actually the dominant real-world path for DISPATCH
decisions in replay mode, so leaving it unguarded would have defeated
Requirement 2.3's "sole source of decision output" claim for the
common case. Gated it through the same wrapper; added 2 new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 3 tests in a new test_sim_print_decision_gate.c covering the old
unconditional per-decision echo (sim_loop.c's sim_print_decision())
no longer duplicating the Round Rollup's per-round summary under
`continue` mode (Requirement 2.3), while `step` mode and headless
(non-interactive) batch runs stay completely unaffected (Requirements
5.1, 5.4). This is exactly the duplication bug-1610 reported.
1 of 3 tests fails as expected RED evidence (continue-mode bare
"[sim] PEND jobid=..." line still appears alongside the Round Rollup's
own summary today); the other 2 (step-mode and headless regression
checks) correctly pass since they pin already-correct, unaffected
behavior.
New file rather than extending either sibling: test_repl_round_rollup.c
is already 1022 lines (222 over the 800-line cap), and
test_repl_event_echo.c (704 lines) deliberately keeps its fixtures
job-free (bug-1595 discipline), which this task's decision-producing
fixtures cannot follow.
No production code touched (sim_loop.c/repl.c/repl_rollup.c/repl_break.c
untouched) - tests only, per this task's RED-phase boundary. The actual
gate implementation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 4 tests in a new test_repl_event_echo.c covering mode-aware event
echo suppression and ending an active collapsed range on structural
events/breakpoint hits (Requirements 1.1-1.4, 3.5, 3.6). Job-free
fixtures (QUEUE_UPDATE/WORKERS_PUSH only) sidestep bug-1595's
jobid-reuse hazard entirely rather than merely avoiding it. 3 of 4
tests fail as expected RED evidence; the 4th (step-mode regression
check) correctly passes since it exercises unchanged behavior.
New file rather than extending test_repl_execution_control.c (already
over the 800-line cap at 937 lines) - applying the same file-size
lesson from tasks 2.1/2.2 proactively, decided before implementation
rather than after a review round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements the round-boundary decision summary, collapsing of repetitive
rounds, and the shared repl_round_collapse_flush() entry point
(Requirements 2.1, 2.2, 2.4, 3.1-3.4, 3.6). Fingerprint composition
matches design.md exactly: dispatch/preempt counts from the existing
per-round tally, pend-jobid set state-derived from core->jobs (not
decision events), so Requirement 2.4 holds independent of bug-1611.
All 7 of task 2.1's RED-phase tests now pass.
Went through three review rounds:
- Round 1 approved the logic but rejected repl.c's size (1160 lines,
45% over the 800-line cap). Remediated by splitting the Round Rollup
functions into a new repl_rollup.c, following this codebase's own
established repl.c/repl_break.c/repl_info.c/repl_params.c domain-split
precedent.
- Round 2 approved the split but found the new file wasn't added to the
"simulation isolation" regression guard (sched_sim_isolation_lib +
test_sim_isolation.c's forbidden-include/symbol scans), plus two stale
repl.c-location doc comments left over from the move. Both fixed.
- Round 3 found repl.c still 17 lines over cap post-split (808 after
trimming, from 817) and rejected on that basis. Verified empirically
that closing the remaining gap isn't achievable via comment trimming
alone (even deleting both comments entirely only reaches 801) without
another disproportionate file split for single-digit lines. Accepted
808 lines (1% over) as the final state, consistent with round 1's own
explicit judgment that a small residual is not a blocker once the
original 45%-over violation is resolved - documented in design.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 7 tests in a new test_repl_round_rollup.c covering per-round
decision summaries, collapsing of repetitive rounds, and the shared
repl_round_collapse_flush() entry point (Requirements 2.1-2.4, 3.1-3.4,
3.6). repl_round_collapse_flush() itself is not yet implemented (task
2.2) - build fails at link time, confirming RED.
Split into its own file rather than the originally-planned
test_repl_execution_control.c: that file was already at the project's
800-line cap before this task, and landing these tests there would
have pushed it to 1563 lines. design.md's File Structure Plan is
corrected accordingly (mirrors the existing test_repl_snapshot_params.c
/test_repl_breakpoints_inspection.c split in this same file family).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
delete <id> used atoi(), which silently coerces a non-numeric or
trailing-garbage token to 0 instead of reporting an error, e.g.
"delete abc" printed "no breakpoint with id 0" rather than rejecting
the input. Switched to strtol()+endptr validation, mirroring the same
"fail loud" discipline repl_cmd_step() already established in this
file family. Added test_delete_non_numeric_id_rejected covering
non-numeric, zero, negative, and trailing-garbage ids.
Logged as bug-1761 (buglog.json entry already committed alongside an
unrelated task's changes earlier in this branch's history).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements repl_fmt_jobid_range() in repl.c (per design.md's "no new
production .c file" commitment), grouping job-array identifiers by base
and compressing contiguous index runs into [lo-hi] ranges. All 5 of task
1.1's RED-phase tests now pass.
Went through one review round: an initial GREEN implementation placed
the function in a new file (repl_jobid_range.c) to sidestep a linker
issue in the lean test target from task 1.1. Review found and verified
a cleaner fix: link the test against the full scheduler_srcs object list
(mirroring the existing sched_scheduler_connect_test precedent), which
resolves the same symbols without a new file. Remediated accordingly;
bug-1762 in buglog.json documents the investigation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds failing unit tests for repl_fmt_jobid_range() covering contiguous
ranges, non-contiguous sub-ranges, multiple array bases, mixed array/
non-array ids, and a too-small output buffer (Requirement 4, AC 4.1-4.5).
The function itself is not yet implemented (task 1.2) - build fails at
link time with undefined reference to repl_fmt_jobid_range, confirming
the RED phase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Extend the IS_WORKER_UNAVAIL-style candidate-host filter to also skip
a host with XCE_WORKER_RECONCILING set (Req 5.1), naturally
re-including it once the bit clears with no separate mechanism (Req
5.2). Leaves the scheduler's decision-publish code and its
symbol-guard regression test untouched.
Fixes bug-1751: design.md's cited filter site (scheduler.c's
sched_reset_worker_(), via sched_begin_session()) is dead code on both
live drivers - driver.c/sim_loop.c actually call
xce_core_begin_session() -> core_session.c's core_reset_workers(), an
independently-maintained mirror not listed in design.md's File
Structure Plan. Applied the identical disjunct to both sites so the
task's own observable-completion criterion holds in production, not
just on the literally-cited dead path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
A job with no rusage[mem=...] request (reserved=0) was being SKIPPED
entirely during reservation-vs-usage reconciliation, on the mistaken
assumption that "no reservation" meant "nothing needs charging". It
doesn't: this codebase's admission check has no live OS read at
decision time (sched_check_host_satisfy_(), resource.c) -- it compares
the worker's last-reported "mem" value against this scheduler's OWN
tracked resInUse tally, built entirely from summed xce_res_account_t
entries. Skipping an unreserved job left its real, live memory
consumption permanently invisible to that tally -- an actual
over-subscription risk (the scheduler could admit more onto a host
than it can support, believing memory was free that a live process was
genuinely using), not merely a display gap.
Fix: charged = max(reserved, actual) applies to EVERY job, reserved or
not. ensure_host_account_by_resname() (resource.c) now creates a fresh
reserved=0 account entry (resolving residx via key_find() against
core->resmgr, same pattern sched_update_res_accounts() already uses
for a fresh account) instead of skipping when a job has no existing
entry for a resource. Returns NULL (a genuine no-op) only when the
resource isn't registered in the cluster's config at all.
Rewrote resource-accounting.md to correct the mechanism it documented:
"available" is NOT a live OS read (mem vs the similarly-named but
static maxmem was worth naming explicitly too) -- it's worker's
last-reported mem minus this scheduler's own resInUse tally, and that
tally must include every job's real usage regardless of reservation.
Also documented a known, pre-existing property surfaced while
correcting this: since mem's own worker report already reflects real
consumption and the scheduler's charge is a separate layer that
doesn't get cleared on a fresh report, tracking can become
conservative over time (under-utilization risk, never over-
subscription) -- the same direction slots already accepts, and a
separate, larger question than this fix's scope.
Replaced test_reconcile_job_resource_usage_skips_when_no_mem_request
(asserted the now-wrong skip behavior) with two tests: one proving an
unreserved job's real usage is created and charged, one proving a
genuine no-op when "mem" isn't a registered resource at all. Full
scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
repl_info_worker_one()/repl_info_resource_one() (repl_info.c) printed
value=avail+used for a CONSUMABLE resource, on the assumption that
avail_v holds a live "remaining capacity" figure that shrinks as jobs
run. That assumption was false: w->resvals[idx] (and
core->sharedInst[idx] for shared resources) is the resource's STATIC
TOTAL capacity, self-reported by the worker at join/heartbeat time and
never decremented as jobs consume it -- confirmed against the actual
admission-control formula this same field feeds (resource.c's
unsatisfy_(): `total[i] < inuse[i] + uncomitted[i] + wanted[i]` treats
it as a total, not a pre-reduced remaining figure). Printing avail+used
therefore double-counted `used` on top of a value that already
included it.
Live-reproduced against a real 3-node cluster + worker: a fully-packed
22-slot worker showed "value=44.00 used=22.00" instead of the correct
"value=22.00 used=22.00" (22 never-decremented total + 22 used = 44,
matching this bug's mechanism exactly). This was previously
investigated as bug-1727 ("recorded for future investigation only") --
confirmed its hypothesis (a) and closed it here rather than opening a
new entry.
Fixed by printing avail_v as-is (no addition) at both call sites.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
bug-1738: a scheduler-side notify could clobber an already-terminal
job state. The controller's own terminal-transition guard
(job_state_is_terminal(), job.c) treats DONE/EXIT as permanent once
set -- apply_job_fail_transition()'s POST_EXEC/Q_POST_EXEC branch only
ever MERGES a JOB_STAT_PERR flag onto whatever DONE/EXIT already
holds, never re-deciding DONE vs EXIT after the fact. The scheduler
had no equivalent guard: HOOK_DETAIL_JOB_FAIL's schedFailJob notify
(sched_job_fail_fun, sched.c) unconditionally requests JOB_STAT_EXIT
with no regard for the job's current state, so a job that finished
successfully (DONE) and then hit an unrelated post-exec script failure
would have the scheduler's view forcibly clobbered back to EXIT.
Fixed with one shared helper, job_state_regresses_terminal(), applied
at both XCE_CORE_EVENT_JOB_STATE_CHANGE and the JOBS_PUSH duplicate-
jobid resync merge (a full resync can just as easily redeliver a stale
snapshot as a genuinely newer one) -- once a job is DONE/EXIT, neither
path can move it to a different terminal state.
bug-1739 (Symptom 1, "phantom RUN"): the bug-1734 fix's own JOBS_PUSH
merge adopted a redelivery's RUN state without transferring the real
allocation data core_build_job() had already attached to the
redelivered clone -- leaving existing->state==RUN with
existing->allocs==NULL, a job the scheduler's own resource/slot
bookkeeping has no record of. Live-reproduced: `info summary` showed
30 elements of an array marked RUN, but `info worker <name>` only
listed 22 real slot assignments for a 22-core worker. Fixed by
transferring job->accounts/job->allocs onto the existing job on a RUN
merge (freeing whatever it previously held), and refusing the merge
entirely (WARN-logged, existing state kept) if the redelivery carries
no allocation data -- treating an unbacked RUN claim as untrustworthy
rather than manufacturing a phantom.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
sched_job_exit_fun() (renamed sched_job_finish_fun, sched.c) subscribes
to HOOK_DETAIL_JOB_EXIT, which fires for EVERY job finish, both success
and failure -- record_job_finish_fact() (job.c) decides DONE vs EXIT
from finish->exit_code BEFORE firing this same hook detail either way
(mirrors persist.c's own HOOK_DETAIL_JOB_EXIT -> WAL_JOB_FINISH
mapping). This function discarded info->exit_code and sent only a bare
jobid, forcing sched_handle_notify() (scheduler/src/handler.c) to
unconditionally map EVERY finish -- including a clean, successful one
-- to JOB_STAT_EXIT. A subsequent HOOK_DETAIL_JOB_UPDATE notify usually
self-corrected this moments later via the full job state, but if that
second notify was dropped during a scheduler reconnect gap (the same
window bug-1731/bug-1734 already documented), the scheduler was left
permanently stuck showing EXIT for a job the controller correctly
recorded as DONE.
Live-reproduced on a real 3-node HA cluster + worker: job
0-8-1788143749-0[38] genuinely completed successfully (worker log:
"completed successfully", controller joblist: DONE) but the
scheduler's own replay showed EXIT.
Fix: carry {jobid, exit_code} across the wire via the existing
JobFinishInfo shape (the same struct apiserver_job_exit_fun already
reuses for this identical hook) instead of a bare UInt64, so
sched_handle_notify() can make the identical exit_code==0-vs-not
decision job.c itself makes. Renamed sched_job_exit_fun to
sched_job_finish_fun to reflect what it actually does.
Live-verified: a 51-job run (all exit 0) showed 100% DONE on the
scheduler side after the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid handling (sched_add_job()
EEXIST) assumed a redelivery is always a harmless no-op -- true for
bug-1714's array-push TOCTOU race (both copies start identically
PEND), but not for a scheduler reconnect's full resync
(on_scheduler_join() -> xcesched_push_jobs(), sched.c), which
re-sends every currently-committed job through this same path
regardless of how far its state has moved on since this scheduler
last saw it.
Live-reproduced on a real 3-node HA cluster + worker: the scheduler's
outbound xcecli connection repeatedly failed over under a 50-element
array-job load; each reconnect's resync redelivered jobs that had
already finished (PEND/RUN -> DONE) while disconnected -- confirmed
via ctrl-1.log's existing bug-1731 "c->sched.ep is NULL, notify
SKIPPED" diagnostic firing for the same jobids during the gap, making
the resync that job's only remaining chance to reach the scheduler.
Discarding it unconditionally permanently froze the tracked job at
its stale state: controller shows DONE, scheduler stuck at RUN/PEND
forever.
Fix: on EEXIST, look up the existing tracked job and merge in the
redelivery's state if different (same terminal-transition
accounts/allocs cleanup XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE
already do), instead of silently dropping it. Only state is merged;
broader field reconciliation is out of scope, same as the existing
allocs/accounts scope note this code path already carries. Bundled a
requested logging fix at the same site (raw jobid -> fmt_jobid()).
Verified live end to end: rebuilt scheduler, reran the same load
against the same cluster (reconnect storm still occurred, proving the
fix doesn't depend on suppressing it), both test arrays now show
consistent DONE state matching the controller exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
test_recordstart_symlink_escape_replies_error expected "invalid record
file name" (the -EINVAL string), but sched_validate_record_filename()'s
canonical-path containment check has always, correctly, returned
-EACCES ("permission denied accessing record file") for a symlink
escaping record_dir -- confirmed against test_recording_path_validation.c's
own direct unit test for the identical scenario, which already asserts
-EACCES and has always passed.
Not environment-dependent as originally suspected (this project's
buglog carried it as a uid-1000-vs-root permission-check-ordering
theory across 7+ sightings since Aug 15) -- it fails identically
regardless of uid, since escape-containment and filesystem-permission
errors were never actually conflated in production code, only in this
test's assertion. No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
