Clear the cache inside core_begin_session_internal() so a job cached as
unsatisfiable in one session is re-evaluated fresh in the next (Req 7.2).
An initial weak-symbol cast-based approach was rejected: it corrupted
xce_sched_sim_driver_t.alt_events_file, since core_begin_session_internal()
is shared between driver.c (real xce_scheduler_t) and sim_loop.c (a
structurally different, deliberately isolated sim driver struct with no
cache field at the same byte offset) -- the strong cache_clear() symbol
won unconditionally for both, invisible to ASAN and untested by any
test_sim_*.c file.
Fixed via a driver-registered callback: xce_sched_core_t gains a
cache_clear_cb hook (NULL by default, per xce_sched_core_init()'s
full-struct memset), set only in main.c's driver-mode branch --
mirroring the existing on_decision/xce_core_set_decision_cb() pattern.
--sim mode returns before that registration is ever reached, so its hook
stays NULL and core_begin_session_internal()'s guarded call is a no-op.
The real xce_scheduler_t-casting adapter lives in alloc_cache.c rather
than driver.c, since ~11 tests white-box-include driver.c without
linking alloc_cache.c.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Wrap sched_make_primary_alloc_() -- the single choke point all three
primary-resreq shapes (standalone, compound-nested, alter-nested via
compound delegation) already funnel through -- with a cache lookup
before allocation and a cache insert on exhaustion. On a hit, pend the
job immediately via the same make_pendreason()/add_job_preason() path
every exhaustion case already uses, without invoking either allocator
(Req 7.3, 7.4, 7.5).
A nested primary has no real per-primary schedobj, so
sched_primary_cache_obj_() synthesizes one on the stack per dispatch for
cache-key purposes; verified memory-safe since alloc_cache.c's
build_key_() copies the fingerprint/identity by value synchronously
before the stack frame unwinds.
Log bug-1768 (compound/alter-nested primaries never get real slot/rusage
availability checks -- seqjob/parjob's prepare_allocation hooks no-op
whenever ctx->schedobj->tag != XCE_RESREQ_PRIMARY) and bug-1769
(sched_make_alter_allocation() type-confuses xce_compound_schedobj_t*
as xce_sched_obj_t*, a ~112-byte heap over-read, currently untested).
Both are pre-existing and out of this feature's scope; discovered and
confirmed while building this task's dispatch tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Task 4.3's multi-host merge itself was already implemented and verified
by task 4.2. The one unproven clause -- job-level shared-resource
allocation computed once regardless of host count -- is now covered by
test_shared_resource_debited_once_regardless_of_host_count, using a
PER_JOB (not PER_TASK) global resource so a per-bucket-repeated
computation can't coincidentally sum to the correct total. Mutation
testing (a deliberately injected duplicate per-host shared debit)
confirms the test fails exactly as expected; production code required
no change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Replace parjob's return-0 stub with a real multi-host allocator: for each
candidate plan, walk buckets in the given (descending) order, claiming a
distinct host per bucket via a fresh host_iter excluding hosts already
claimed elsewhere in the same plan attempt. On a bucket failure, reverse
every earlier successful claim's provisional debit via
sched_ctx_uncommit_host() before advancing to the next plan (Req 4.2, 4.4,
4.5, 4.6, 4.7).
Fix parjob_prepare_allocation() to compute each bucket's want from that
bucket's own slot count (ctx->nslots) instead of a fixed value, and reset
ctx->alloc_req.host at the top of each bucket's scan so host_iter's
pre-filter can't inherit a stale want from a differently-sized prior
bucket (safe: sched_try_alloc_on_host_() always repopulates it from the
real per-bucket want before its own authoritative check runs).
Add a maxTasks<=1 guard matching Req 4 AC1's precondition, needed because
a snapshot-restored job with unpopulated maxTasks==0 (pre-existing,
tracked separately as bug-1767) would otherwise reach parjob's now-real
allocator instead of being silently ignored as before.
On full success, merges every claimed bucket into one xce_job_alloc_t via
make_job_alloc()'s existing runtime-sized host array, satisfying task
4.3's multi-host commit scope as well.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add the real xce_alloc_plan_iter_t for parjob: for host-count k=1..maxTasks
(ascending, most-preferred-first), generate one balanced partition of
maxTasks into k buckets (floor/ceil split, remainder to the first rem
buckets), which is descending by construction. Pure function of maxTasks
alone -- never dereferences sched -- with a real pthread_testcancel()
cancellation point and stable post-exhaustion NULL (Req 2.1-2.6, 4.1).
my_alloc()/parjob_prepare_allocation() remain untouched stubs; wiring the
iterator into real allocation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Add a dedicated equivalence suite (test_seqjob_rewrap_equivalence.c)
covering multi-host fixtures the trivial single-host test doesn't reach:
select-filter-only rejection, resource-only rejection, a combined
select-then-resource-then-success ordering, and per-host debit
correctness. The iteration-order test is grounded in the deleted
pre-rewrap sched_collect_workers() code path (verified via git show
against the parent commit) and the unmodified xlist.c primitive, not
merely in host_iter.c's own traversal, so it proves equivalence to the
old behavior rather than internal self-consistency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Restructure seqjob's my_alloc() as a trivial single-bucket case over the
shared xce_alloc_plan_iter_t/xce_host_iter_t contracts, replacing its own
inline worker-collection loop. Guard seqjob_prepare_allocation() to
maxTasks == 1, closing the fan-out gap for the parallel allocator's later
fix (Req 9.1, 4.6).
Export sched_try_alloc_on_host_() from resource.c (drop static/va_list,
plain (worker, sched, ctx) signature) so seqjob calls the real per-host
try-alloc leaf instead of a private duplicate; delete the now-dead
sched_try_alloc_on_hosts() variadic wrapper. Also wires host_iter.c into
the production scheduler_srcs list (task 2.1 only added it to testsuite).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds alloc_cache.c: xce_cache_key_t, cache_key_ops (the corrected
three-way, 0-means-equal .cmp -- an inverted-boolean version was a
real bug found during design review), and cache_lookup/insert/clear
against a real kvtab_t (constructed matching xce_select.c's
sel_node_table_init() precedent). Cache values are a boxed reason-
category int, not a bare hit/miss sentinel, satisfying Req 7 AC5's
requirement that a hit reproduce the original pend-reason category.
unsatisfiable_cache lives on xce_scheduler_t per Req 7 AC1.
Task 2.2 (scheduler-allocation spec). Requirements 7.1, 7.3, 7.4, 7.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds host_iter.c: the one shared xce_host_iter_t implementation,
lazily filtering sched->core.workers by select expression
(sched_meet_host_selection, unchanged) and resource availability
(sched_check_host_satisfy, unchanged), with a real cancellation
point per candidate visited. xce_alloc_plan_t gains a ctx field
(xce_sched_ctx_t*) so the iterator can reach the scheduler/select
expression/staged resource-want -- additive to task 1.1's approved
interface, NULL-guarded at init(). Test proves the core design
property this spec exists to establish: a host too small for one
search remains a valid candidate for a later, smaller search via a
fresh iterator instance over the same worker set.
Task 2.1 (scheduler-allocation spec). Requirements 3.1-3.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds sched_ctx_uncommit_host() (resource.c), the structural inverse
of sched_try_alloc_on_host_()'s existing XVEC_ADD_FLOAT commit call,
scoped to ctx->uncomitted only per requirements.md's Rollback scope
discussion. Test exercises the real sched_ctx_get_uncomit_host()
(schedobj.c) via white-box #include rather than a private
reimplementation, matching test_picker_config_fields.c's established
precedent.
Task 1.4 (scheduler-allocation spec). Requirement 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds xce_alloc_mode_t (XCE_ALLOC_NORMAL=0, XCE_ALLOC_PREEMPT, with
commented-out RESERVE/BACKFILL placeholders) and alloc_mode on
xce_sched_ctx_t, zero-defaulted by sched_ctx_init()'s existing
memset. Purely additive: nothing branches on the field yet -- this
spec only declares the extension point scheduler-preemption needs.
Task 1.3 (scheduler-allocation spec). Requirement 9.5.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds rusage_hash to xce_primary_resreq_t (computed once by
parse_resreq_primary() via a new hash_rusages() helper, hashing
rusage content rather than raw struct bytes since resname is a
non-interned strdup'd pointer), the xce_resreq_fingerprint_t value
type, and resreq_fp on xce_primary_schedobj_t, populated once at real
job-ingest time in core_build_primary_schedobj().
Task 1.2 (scheduler-allocation spec). Requirement 7.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Defines xce_alloc_plan_t (partition of a job's task count into
per-host slot buckets), xce_alloc_plan_iter_t, and xce_host_iter_t
(init/next/fini vtables, stack-allocate-then-init-in-place convention
matching sched_ctx_init) as reusable abstractions any allocator can
adopt. Proven usable by a trivial no-op reference implementation and
a fresh-per-bucket host_iter composition test, no allocator wired in
yet.
Task 1.1 (scheduler-allocation spec). Requirements 2.1, 3.1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
An idle scheduler (nothing queued to publish) never noticed a dead
controller connection: sched_needs_rejoin was only ever set as a side
effect of an active client-library call, and an idle round makes none.
After any controller failover with nothing already in flight, jobs
submitted to the new primary stayed PEND indefinitely.
Adds xcecli_is_connection_broken() (libxce) -- a pure, socket-free read
of the transport's already-async-maintained dead/alive flag, set for
free by the existing RDHUP disconnect handler with zero application
calls. sched_check_pending_rejoin() now retries xcecli_join() on either
the existing reactive flag or this new passive signal, so a healthy
idle scheduler still makes zero extra network calls while a genuinely
dead connection self-heals within one ~1s round -- live-verified
against a 3-node HA cluster (rejoin fired ~300ms after new-primary
election with nothing submitted in between).
Also closes out bug-1735 (job memory/CPU usage reporting): confirmed
already fixed on this branch since 2026-08-30, just never marked in
the bug log.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
Adds test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch,
mirroring bug-1764's actual live-observed shape: a worker with one job
driven to a terminal state and two jobs genuinely still RUN, all real
allocations to the SAME worker. After reconciling a stray DISPATCH
entry for the terminal job's id, info worker excludes it while still
listing the two running jobs, and info summary's running count agrees.
Round 1 review rejected an initial version for using a mismatched host
string on the stray entry ("workerarr-stray" instead of "workerarr"),
which made the info worker/info summary assertions pass vacuously
regardless of whether the fix existed, since repl_info_worker_one()
filters assigned jobs by an exact host-string match. Round 2 fixed the
host string and confirmed via an independently-reproduced mutation
test (fix reverted, the test's own accounting-level pre-check
bypassed) that the REPL-level assertions now genuinely, independently
detect the bug.
design.md's Testing Strategy is corrected: it originally said no new
test was needed, reasoning the existing display-path suites already
covered this; tasks.md's own task 2.2 correctly required a
purpose-built test for this exact scenario instead.
Full scheduler test suite: 63/64 OK, sole failure the pre-existing,
unrelated bug-1758. This is the final task of the
scheduler-sim-reconcile-terminal-guard spec; all implementation tasks
(1.1-2.2) are now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds two failing tests formalizing bug-1764's reproduction: a job
driven to JOB_STAT_DONE (and, separately, JOB_STAT_EXIT) via a real
xce_core_push_events() JOB_STATE_CHANGE call correctly has its
allocs/accounts nulled by the existing end-of-life cleanup, but
sim_reconcile_unaccounted_jobs() then wrongly resurrects both fields
when presented with a stray DISPATCH entry for that same jobid, since
its trigger checks only job->accounts == NULL with no state check.
The pre-existing eager-capture regression test,
test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
already covers Requirement 1.2's "PEND job's early DISPATCH still
captured" case explicitly and needed no duplicate. No production code
touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 6 failing tests for the Event Echo Preamble Deduplicator
(Requirements 1.1-1.4, 2.2): a second echoed event sharing its
predecessor's rendered timestamp+session omits its preamble; two
events across a session boundary both print their own preamble; the
critical raw-vs-rendered nuance (Requirement 1.3) - two events whose
raw nanosecond timestamps genuinely differ but whose rendered,
second-granularity timestamp and session are identical still dedupe,
with a fixture-premise self-check proving the two raw values really
do render identically; the first echoed event in a fresh session
always prints its preamble; a structural event under continue mode
after suppressed routine events follows the same rule; event content
and ordering are unaffected regardless of the preamble decision.
Lands in a new test_repl_step_verbosity.c (1010 lines, 6 tests) rather
than extending test_repl_event_echo.c (705 lines) - proactive split
per this file family's established practice, since appending would
have pushed that file well past the project's ~1000-line trigger.
Registered in testsuite/meson.build. No production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds failing tests for Requirement 7 (Discoverable Help and Breakpoint
Event Types): help's four labeled sections in correct order, one
consistent column width across every listed command, info config's
placement alongside its info siblings, set units's placement alongside
set speed/output/verify, the break line's complete BP_KIND_TABLE-derived
event-type listing, and break's own unrecognized-event-type error
listing the same complete set. Each new test hardcodes BP_KIND_TABLE's
current 12 tokens rather than referencing the not-yet-implemented
repl_bp_kind_names() accessor, so RED comes from genuine assertion
failures against today's behavior rather than a link error.
New tests for Requirements 7.1-7.4 land in a new test_repl_help.c
rather than test_repl_execution_control.c (949 lines going in, within
reach of this spec's own ~1000-line split trigger already flagged at
task 2.1) - proactive split per this spec's established practice,
documented in design.md. break's Requirement 7.5 error-message test
lands in test_repl_breakpoints_inspection.c exactly as originally
planned, alongside the pre-existing test_break_unrecognized_event_type_rejected
(left unmodified).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Rewrites info worker's/info resource's resource-listing loops from a
flat, unconditional fprintf into a 3-pass, section-grouped table
(Consumable/Capacity, Load Metrics, Static Hardware Attributes),
units_mode-aware value rendering (raw/human/both, including swp's
MB->bytes conversion), whole-number formatting with no trailing .00
outside the consumable value=/used= pair, and jobid-range-compression
adoption for info worker's assigned-job listing (Requirements 6.4-6.6,
4.5). All 10 of task 8.1's tests pass. Preserves every named prior-bug
invariant (bug-1727's avail_v-as-total, bug-1613's consumable/
non-consumable split, bug-1593's GLOBAL/SHARED filter, bug-1688's
.caps-bound iteration).
Fixes one pre-existing test (bug-1688, predating this spec) whose
"value=5.00" assertion for a whole-number non-consumable resource
directly conflicts with Requirement 6.5's intentional no-trailing-.00
change - same class of fix as task 4.2's pre-existing-test updates.
repl_info.c grew from 985 to 1267 lines (58% over the 800-line cap,
worse than any other file this spec touched) - but unlike repl.c/
repl_break.c, it was already 23% over cap before this spec began.
Review empirically confirmed a narrow helper-only extraction doesn't
achieve compliance (still 41% over), and a compliance-achieving split
would mean restructuring ~690 lines of pre-existing display logic
this feature doesn't otherwise touch - a larger scheduler-core-level
decision, accepted as a residual rather than folded into this task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 10 tests in a new test_repl_resource_table.c covering info
worker's/info resource's resource-listing table/grouping layout,
units_mode-aware rendering (raw/human/both, including swp's MB->bytes
conversion and a no-mutation check on the underlying stored value),
whole-number-vs-fractional formatting, and info worker's assigned-job
listing adopting the real repl_fmt_jobid_range() (Requirements
6.4-6.6, 4.5). All 10 fail as expected against today's flat,
unconditional per-resource fprintf loop and uncompressed listing.
Found and fixed a genuine ASAN stack-use-after-return in the test's
own fixture code (not production): WORKERS_PUSH's handler moves a raw
pointer into a job's resvals rather than deep-copying, so the fixture
must heap-allocate rather than stack-allocate the structures it feeds
through that path - independently reproduced by review.
New file rather than extending test_repl_breakpoints_inspection.c:
landing this content there first pushed it to 3215 lines, 3.2x this
file family's own ~1000-line split trigger already used to justify
splitting test_repl_round_rollup.c/test_repl_event_echo.c for
structurally identical reasons.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements repl_is_bytes_valued_resource()/repl_format_bytes_human()
(mirroring xce_test's confirmed is_bytes_valued_resource()/
format_bytes_human() precedent exactly) and the units_mode setting
(explicitly defaulted to both, not left to zero-init) wired into
set units raw|human|both alongside the existing set speed/output/verify
branches (Requirements 6.1-6.3). All 8 of task 7.1's tests pass.
Split repl_format_ts_human(), the Jobid Range Compressor (task 1.2),
and these two new helpers out of repl.c into a new repl_fmt.c: adding
them in place pushed repl.c to 929 lines (16.1% over the 800-line
cap), and unlike task 6.2's single-function split attempt (which
didn't even achieve compliance), bundling these three pure-formatter
groups together does - repl.c settles at 587 lines, repl_fmt.c at 402,
both comfortably under cap. Required a matching #include line in the
20 test files that white-box #include repl.c, plus extending
test_sim_isolation.c's forbidden-include/symbol scan to cover the new
file, following the repl_rollup.c split precedent from task 2.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 5 pure-function tests in a new test_repl_resource_formatting.c
for repl_is_bytes_valued_resource()/repl_format_bytes_human()
(classification, adaptive B/KB/MB/GB/TB/PB scaling including exact
boundary values, swp's MB->bytes conversion), and 3 tests in
test_repl_snapshot_params.c for set units raw|human|both parsing
(Requirements 6.1-6.3). All fail as expected: link failure for the
not-yet-existing helpers, compile failure for the not-yet-existing
units_mode field/enum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 2 tests in a new test_repl_prompt_context.c covering the dynamic
prompt's plain-before-first-session baseline and its session/timestamp
content in both step and continue modes (Requirement 5.5).
Required genuinely new test infrastructure: libedit's readline() only
writes its prompt text when both stdin and stdout are real ptys, which
no existing REPL test in this codebase's freopen()-from-file harness
can observe. Built a PTY-based capture technique (openpty(), ECHO
disabled via termios) instead - independently reproduced via a
standalone probe and stress-tested 15x with zero flakiness during
review.
New file rather than extending test_repl_execution_control.c (already
at 948 lines, over the 800-line cap), per this spec's established
split precedent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Gates sim_print_decision() so the bare per-decision line no longer
prints under continue mode, making the Round Rollup's summary the
sole source of decision output (Requirement 2.3, closing bug-1610).
step mode and non-interactive/headless runs are completely
unaffected (Requirements 5.1, 5.4). Implemented as a file-local
inline wrapper in sim_loop.c rather than repl.h, since the gated
function is static to that file and repl.h can't see repl_state's
internals without leaking them to every other repl.h consumer.
Fixes 3 pre-existing tests (predating this spec, from scheduler-core)
that pinned the now-intentionally-changed bare-line behavior under
continue mode - each fixed differently based on its actual purpose
rather than uniformly: two kept continue mode with their anchor
switched to the Round Rollup's summary (since they specifically test
continue-mode breakpoint interruption, which step mode would
trivialize), one switched to step mode (since its actual invariant -
print-before-breakpoint-check call order - is mode-independent and
continue mode no longer has any observable anchor for it).
Review caught a second, unguarded sim_print_decision() call site in
sim_reconcile_unaccounted_jobs() (added by an earlier, separately-
merged spec) that design.md's boundary never accounted for - per the
review, this is actually the dominant real-world path for DISPATCH
decisions in replay mode, so leaving it unguarded would have defeated
Requirement 2.3's "sole source of decision output" claim for the
common case. Gated it through the same wrapper; added 2 new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 3 tests in a new test_sim_print_decision_gate.c covering the old
unconditional per-decision echo (sim_loop.c's sim_print_decision())
no longer duplicating the Round Rollup's per-round summary under
`continue` mode (Requirement 2.3), while `step` mode and headless
(non-interactive) batch runs stay completely unaffected (Requirements
5.1, 5.4). This is exactly the duplication bug-1610 reported.
1 of 3 tests fails as expected RED evidence (continue-mode bare
"[sim] PEND jobid=..." line still appears alongside the Round Rollup's
own summary today); the other 2 (step-mode and headless regression
checks) correctly pass since they pin already-correct, unaffected
behavior.
New file rather than extending either sibling: test_repl_round_rollup.c
is already 1022 lines (222 over the 800-line cap), and
test_repl_event_echo.c (704 lines) deliberately keeps its fixtures
job-free (bug-1595 discipline), which this task's decision-producing
fixtures cannot follow.
No production code touched (sim_loop.c/repl.c/repl_rollup.c/repl_break.c
untouched) - tests only, per this task's RED-phase boundary. The actual
gate implementation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 4 tests in a new test_repl_event_echo.c covering mode-aware event
echo suppression and ending an active collapsed range on structural
events/breakpoint hits (Requirements 1.1-1.4, 3.5, 3.6). Job-free
fixtures (QUEUE_UPDATE/WORKERS_PUSH only) sidestep bug-1595's
jobid-reuse hazard entirely rather than merely avoiding it. 3 of 4
tests fail as expected RED evidence; the 4th (step-mode regression
check) correctly passes since it exercises unchanged behavior.
New file rather than extending test_repl_execution_control.c (already
over the 800-line cap at 937 lines) - applying the same file-size
lesson from tasks 2.1/2.2 proactively, decided before implementation
rather than after a review round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements the round-boundary decision summary, collapsing of repetitive
rounds, and the shared repl_round_collapse_flush() entry point
(Requirements 2.1, 2.2, 2.4, 3.1-3.4, 3.6). Fingerprint composition
matches design.md exactly: dispatch/preempt counts from the existing
per-round tally, pend-jobid set state-derived from core->jobs (not
decision events), so Requirement 2.4 holds independent of bug-1611.
All 7 of task 2.1's RED-phase tests now pass.
Went through three review rounds:
- Round 1 approved the logic but rejected repl.c's size (1160 lines,
45% over the 800-line cap). Remediated by splitting the Round Rollup
functions into a new repl_rollup.c, following this codebase's own
established repl.c/repl_break.c/repl_info.c/repl_params.c domain-split
precedent.
- Round 2 approved the split but found the new file wasn't added to the
"simulation isolation" regression guard (sched_sim_isolation_lib +
test_sim_isolation.c's forbidden-include/symbol scans), plus two stale
repl.c-location doc comments left over from the move. Both fixed.
- Round 3 found repl.c still 17 lines over cap post-split (808 after
trimming, from 817) and rejected on that basis. Verified empirically
that closing the remaining gap isn't achievable via comment trimming
alone (even deleting both comments entirely only reaches 801) without
another disproportionate file split for single-digit lines. Accepted
808 lines (1% over) as the final state, consistent with round 1's own
explicit judgment that a small residual is not a blocker once the
original 45%-over violation is resolved - documented in design.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 7 tests in a new test_repl_round_rollup.c covering per-round
decision summaries, collapsing of repetitive rounds, and the shared
repl_round_collapse_flush() entry point (Requirements 2.1-2.4, 3.1-3.4,
3.6). repl_round_collapse_flush() itself is not yet implemented (task
2.2) - build fails at link time, confirming RED.
Split into its own file rather than the originally-planned
test_repl_execution_control.c: that file was already at the project's
800-line cap before this task, and landing these tests there would
have pushed it to 1563 lines. design.md's File Structure Plan is
corrected accordingly (mirrors the existing test_repl_snapshot_params.c
/test_repl_breakpoints_inspection.c split in this same file family).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
delete <id> used atoi(), which silently coerces a non-numeric or
trailing-garbage token to 0 instead of reporting an error, e.g.
"delete abc" printed "no breakpoint with id 0" rather than rejecting
the input. Switched to strtol()+endptr validation, mirroring the same
"fail loud" discipline repl_cmd_step() already established in this
file family. Added test_delete_non_numeric_id_rejected covering
non-numeric, zero, negative, and trailing-garbage ids.
Logged as bug-1761 (buglog.json entry already committed alongside an
unrelated task's changes earlier in this branch's history).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements repl_fmt_jobid_range() in repl.c (per design.md's "no new
production .c file" commitment), grouping job-array identifiers by base
and compressing contiguous index runs into [lo-hi] ranges. All 5 of task
1.1's RED-phase tests now pass.
Went through one review round: an initial GREEN implementation placed
the function in a new file (repl_jobid_range.c) to sidestep a linker
issue in the lean test target from task 1.1. Review found and verified
a cleaner fix: link the test against the full scheduler_srcs object list
(mirroring the existing sched_scheduler_connect_test precedent), which
resolves the same symbols without a new file. Remediated accordingly;
bug-1762 in buglog.json documents the investigation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds failing unit tests for repl_fmt_jobid_range() covering contiguous
ranges, non-contiguous sub-ranges, multiple array bases, mixed array/
non-array ids, and a too-small output buffer (Requirement 4, AC 4.1-4.5).
The function itself is not yet implemented (task 1.2) - build fails at
link time with undefined reference to repl_fmt_jobid_range, confirming
the RED phase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Extend the IS_WORKER_UNAVAIL-style candidate-host filter to also skip
a host with XCE_WORKER_RECONCILING set (Req 5.1), naturally
re-including it once the bit clears with no separate mechanism (Req
5.2). Leaves the scheduler's decision-publish code and its
symbol-guard regression test untouched.
Fixes bug-1751: design.md's cited filter site (scheduler.c's
sched_reset_worker_(), via sched_begin_session()) is dead code on both
live drivers - driver.c/sim_loop.c actually call
xce_core_begin_session() -> core_session.c's core_reset_workers(), an
independently-maintained mirror not listed in design.md's File
Structure Plan. Applied the identical disjunct to both sites so the
task's own observable-completion criterion holds in production, not
just on the literally-cited dead path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
A job with no rusage[mem=...] request (reserved=0) was being SKIPPED
entirely during reservation-vs-usage reconciliation, on the mistaken
assumption that "no reservation" meant "nothing needs charging". It
doesn't: this codebase's admission check has no live OS read at
decision time (sched_check_host_satisfy_(), resource.c) -- it compares
the worker's last-reported "mem" value against this scheduler's OWN
tracked resInUse tally, built entirely from summed xce_res_account_t
entries. Skipping an unreserved job left its real, live memory
consumption permanently invisible to that tally -- an actual
over-subscription risk (the scheduler could admit more onto a host
than it can support, believing memory was free that a live process was
genuinely using), not merely a display gap.
Fix: charged = max(reserved, actual) applies to EVERY job, reserved or
not. ensure_host_account_by_resname() (resource.c) now creates a fresh
reserved=0 account entry (resolving residx via key_find() against
core->resmgr, same pattern sched_update_res_accounts() already uses
for a fresh account) instead of skipping when a job has no existing
entry for a resource. Returns NULL (a genuine no-op) only when the
resource isn't registered in the cluster's config at all.
Rewrote resource-accounting.md to correct the mechanism it documented:
"available" is NOT a live OS read (mem vs the similarly-named but
static maxmem was worth naming explicitly too) -- it's worker's
last-reported mem minus this scheduler's own resInUse tally, and that
tally must include every job's real usage regardless of reservation.
Also documented a known, pre-existing property surfaced while
correcting this: since mem's own worker report already reflects real
consumption and the scheduler's charge is a separate layer that
doesn't get cleared on a fresh report, tracking can become
conservative over time (under-utilization risk, never over-
subscription) -- the same direction slots already accepts, and a
separate, larger question than this fix's scope.
Replaced test_reconcile_job_resource_usage_skips_when_no_mem_request
(asserted the now-wrong skip behavior) with two tests: one proving an
unreserved job's real usage is created and charged, one proving a
genuine no-op when "mem" isn't a registered resource at all. Full
scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
repl_info_worker_one()/repl_info_resource_one() (repl_info.c) printed
value=avail+used for a CONSUMABLE resource, on the assumption that
avail_v holds a live "remaining capacity" figure that shrinks as jobs
run. That assumption was false: w->resvals[idx] (and
core->sharedInst[idx] for shared resources) is the resource's STATIC
TOTAL capacity, self-reported by the worker at join/heartbeat time and
never decremented as jobs consume it -- confirmed against the actual
admission-control formula this same field feeds (resource.c's
unsatisfy_(): `total[i] < inuse[i] + uncomitted[i] + wanted[i]` treats
it as a total, not a pre-reduced remaining figure). Printing avail+used
therefore double-counted `used` on top of a value that already
included it.
Live-reproduced against a real 3-node cluster + worker: a fully-packed
22-slot worker showed "value=44.00 used=22.00" instead of the correct
"value=22.00 used=22.00" (22 never-decremented total + 22 used = 44,
matching this bug's mechanism exactly). This was previously
investigated as bug-1727 ("recorded for future investigation only") --
confirmed its hypothesis (a) and closed it here rather than opening a
new entry.
Fixed by printing avail_v as-is (no addition) at both call sites.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
bug-1738: a scheduler-side notify could clobber an already-terminal
job state. The controller's own terminal-transition guard
(job_state_is_terminal(), job.c) treats DONE/EXIT as permanent once
set -- apply_job_fail_transition()'s POST_EXEC/Q_POST_EXEC branch only
ever MERGES a JOB_STAT_PERR flag onto whatever DONE/EXIT already
holds, never re-deciding DONE vs EXIT after the fact. The scheduler
had no equivalent guard: HOOK_DETAIL_JOB_FAIL's schedFailJob notify
(sched_job_fail_fun, sched.c) unconditionally requests JOB_STAT_EXIT
with no regard for the job's current state, so a job that finished
successfully (DONE) and then hit an unrelated post-exec script failure
would have the scheduler's view forcibly clobbered back to EXIT.
Fixed with one shared helper, job_state_regresses_terminal(), applied
at both XCE_CORE_EVENT_JOB_STATE_CHANGE and the JOBS_PUSH duplicate-
jobid resync merge (a full resync can just as easily redeliver a stale
snapshot as a genuinely newer one) -- once a job is DONE/EXIT, neither
path can move it to a different terminal state.
bug-1739 (Symptom 1, "phantom RUN"): the bug-1734 fix's own JOBS_PUSH
merge adopted a redelivery's RUN state without transferring the real
allocation data core_build_job() had already attached to the
redelivered clone -- leaving existing->state==RUN with
existing->allocs==NULL, a job the scheduler's own resource/slot
bookkeeping has no record of. Live-reproduced: `info summary` showed
30 elements of an array marked RUN, but `info worker <name>` only
listed 22 real slot assignments for a 22-core worker. Fixed by
transferring job->accounts/job->allocs onto the existing job on a RUN
merge (freeing whatever it previously held), and refusing the merge
entirely (WARN-logged, existing state kept) if the redelivery carries
no allocation data -- treating an unbacked RUN claim as untrustworthy
rather than manufacturing a phantom.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
sched_job_exit_fun() (renamed sched_job_finish_fun, sched.c) subscribes
to HOOK_DETAIL_JOB_EXIT, which fires for EVERY job finish, both success
and failure -- record_job_finish_fact() (job.c) decides DONE vs EXIT
from finish->exit_code BEFORE firing this same hook detail either way
(mirrors persist.c's own HOOK_DETAIL_JOB_EXIT -> WAL_JOB_FINISH
mapping). This function discarded info->exit_code and sent only a bare
jobid, forcing sched_handle_notify() (scheduler/src/handler.c) to
unconditionally map EVERY finish -- including a clean, successful one
-- to JOB_STAT_EXIT. A subsequent HOOK_DETAIL_JOB_UPDATE notify usually
self-corrected this moments later via the full job state, but if that
second notify was dropped during a scheduler reconnect gap (the same
window bug-1731/bug-1734 already documented), the scheduler was left
permanently stuck showing EXIT for a job the controller correctly
recorded as DONE.
Live-reproduced on a real 3-node HA cluster + worker: job
0-8-1788143749-0[38] genuinely completed successfully (worker log:
"completed successfully", controller joblist: DONE) but the
scheduler's own replay showed EXIT.
Fix: carry {jobid, exit_code} across the wire via the existing
JobFinishInfo shape (the same struct apiserver_job_exit_fun already
reuses for this identical hook) instead of a bare UInt64, so
sched_handle_notify() can make the identical exit_code==0-vs-not
decision job.c itself makes. Renamed sched_job_exit_fun to
sched_job_finish_fun to reflect what it actually does.
Live-verified: a 51-job run (all exit 0) showed 100% DONE on the
scheduler side after the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid handling (sched_add_job()
EEXIST) assumed a redelivery is always a harmless no-op -- true for
bug-1714's array-push TOCTOU race (both copies start identically
PEND), but not for a scheduler reconnect's full resync
(on_scheduler_join() -> xcesched_push_jobs(), sched.c), which
re-sends every currently-committed job through this same path
regardless of how far its state has moved on since this scheduler
last saw it.
Live-reproduced on a real 3-node HA cluster + worker: the scheduler's
outbound xcecli connection repeatedly failed over under a 50-element
array-job load; each reconnect's resync redelivered jobs that had
already finished (PEND/RUN -> DONE) while disconnected -- confirmed
via ctrl-1.log's existing bug-1731 "c->sched.ep is NULL, notify
SKIPPED" diagnostic firing for the same jobids during the gap, making
the resync that job's only remaining chance to reach the scheduler.
Discarding it unconditionally permanently froze the tracked job at
its stale state: controller shows DONE, scheduler stuck at RUN/PEND
forever.
Fix: on EEXIST, look up the existing tracked job and merge in the
redelivery's state if different (same terminal-transition
accounts/allocs cleanup XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE
already do), instead of silently dropping it. Only state is merged;
broader field reconciliation is out of scope, same as the existing
allocs/accounts scope note this code path already carries. Bundled a
requested logging fix at the same site (raw jobid -> fmt_jobid()).
Verified live end to end: rebuilt scheduler, reran the same load
against the same cluster (reconnect storm still occurred, proving the
fix doesn't depend on suppressing it), both test arrays now show
consistent DONE state matching the controller exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
test_recordstart_symlink_escape_replies_error expected "invalid record
file name" (the -EINVAL string), but sched_validate_record_filename()'s
canonical-path containment check has always, correctly, returned
-EACCES ("permission denied accessing record file") for a symlink
escaping record_dir -- confirmed against test_recording_path_validation.c's
own direct unit test for the identical scenario, which already asserts
-EACCES and has always passed.
Not environment-dependent as originally suspected (this project's
buglog carried it as a uid-1000-vs-root permission-check-ordering
theory across 7+ sightings since Aug 15) -- it fails identically
regardless of uid, since escape-containment and filesystem-permission
errors were never actually conflated in production code, only in this
test's assertion. No production code changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Found while chasing bug-1714's crash (root cause still open -- see that
entry): two related, independently-confirmed defects, not proven to be
bug-1714's own cause but real bugs on their own merits.
xce_core_push_events()'s JOBS_PUSH case never checked sched_add_job()'s
return value -- if key_add() failed (duplicate jobid), the freshly-built
xce_sched_job_t was handed to picker_iface->on_job_push() anyway, even
though it was never registered in core->jobs. An orphaned job the
picker knows about but sched_get_job() can never find again. Now checks
the return, frees the orphan (at core_build_job()'s own established
cleanup granularity -- xce_sched_job_t has no complete destructor
anywhere in this codebase, kill_sched_job() only frees ->rusages, so
this intentionally doesn't invent a more thorough one), and skips the
picker call.
This makes it safe to fix (not the other way around): sched_on_job_array()'s
per-element dedup check (handler.c) has a TOCTOU race -- a job is only
added to core->jobs later, by the driver thread, so a redelivered array
push arriving on the request thread before the driver catches up isn't
caught. Documented as a best-effort optimization, not a correctness
boundary -- the actual authoritative uniqueness check is sched_add_job()'s
key_add(), which the JOBS_PUSH fix above now handles correctly.
TDD: new test reproduces the duplicate-jobid path (RED: on_job_push
fired twice, Expected 1 Was 2), GREEN after the fix. Full scheduler
regression: 54/55, only the known bug-1561-adjacent flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
xce_sched_core_load_snapshot() left every restored worker's resInUse
(and core->sharedInUse) at zero -- zarray_init() in the workers-
restoration block clears them, and nothing re-summed against the
restored job allocations. A caller inspecting worker/resource state
between a snapshot load and the first scheduling session (e.g. the
REPL's `info worker` before the first `step`) saw a misleadingly empty
"used" total for jobs already known to be RUNNING. Added one
sched_account_jobs(core) call after workers/jobs are restored -- a
correct idempotent-from-zero re-derivation using the same
sched_charge_account() chain the next real session would use anyway.
Must not be called again without an intervening reset (accumulates,
does not assign) -- verified no such double-call exists in this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOB_UPDATE (core_events.c) is the only currently-live path
a controller job-finish notification reaches, but unlike its sibling
XCE_CORE_EVENT_JOB_STATE_CHANGE case, it never released job->accounts/
job->allocs on a terminal transition, leaking them for the scheduler
process's lifetime. Mirrors the existing JOB_STATE_CHANGE cleanup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Task 4.2: extend test_e2e_original_motivation_long_replay with PREEXIST_ID
(job 70013, dedicated worker worker-e2e-pre) -- a job submitted,
live-dispatched, and confirmed RUN entirely before sched_record_start(),
so its dispatch decision can never appear in the replay stream. Proves
its "used=9.00" resource contribution is correct starting from the very
first replayed round, recoverable only via snapshot-load allocation
restoration (tasks 2.1/2.2), not replay-based reconciliation -- the
complementary case to the file's existing ANCHOR_ID scenario.
Completes and fixes a pre-existing uncommitted WIP addendum to this file:
removed a redundant enqueue_queue_update() call that was masking a
genuine, unrelated latent bug in core_upsert_queue() (rtti makeString()
pointer freed via plain free(), same class as bug-1503/bug-1600) --
noted in tasks.md's Implementation Notes as a follow-up, not fixed here
(test-only task boundary).
Verified via fault injection: gating snapshot_load.c's allocation
restore reproduces the exact pre-fix failure at this test's new
assertion, then cleanly reverts.
Requirements: 1.1, 1.2, 1.3
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BSiC2cPVH2VSBdtFq4P4ju
Task 4.1: new test proving a snapshot-restored PENDING job (valid
resreq) reaches the real, live sched_allocator() thread's
sched_make_allocation() entry point without crashing -- the exact
bug-1569-class failure task 1.2 closed for the snapshot-load path.
New file (not an extension of test_snapshot_save_load.c) since driving
the real allocator needs scheduler.c's full dependency closure, too
heavy for that file's deliberately lean core-only TU. Follows the
already-established pattern from test_scheduler_connect.c /
test_scheduler_backpressure.c / test_scheduler_dispatch_commit.c: link
the real scheduler.c, wrap xcecli_start/xcecli_join, start a real
sched_init(), feed jobs directly into core.jbuf to drive the live
allocator thread. Wired into meson.build as a new executable/test.
Verified with two fault-injection experiments (reverted after):
disabling task 1.2's schedobj construction call makes this test fail
cleanly via its own assertion; going further and removing that
assertion reproduces a genuine ASAN-caught SEGV in the real allocator
path -- concrete proof this test would have caught bug-1569's failure
shape. Both polling loops (jbuf drain, decision wait) are bounded
(2s timeout, 5ms poll) -- no hang risk; confirmed non-flaky across 10
consecutive runs.
Requirements: 2.2, 2.4
Task 3.2: adds test_snapshot_load_leaves_running_job_without_allocation_at_zero_resinuse,
covering the specific Req 3.5 scenario the existing PENDING-job test didn't
-- a RUNNING job whose allocation was never captured (simulating a
pre-feature-saved snapshot). With a real worker/resource present (making
the assertion non-vacuous), confirms: load succeeds with no error,
job->schedobj is still valid (task 1.2, independent of allocation data),
job->allocs/accounts stay NULL, and resInUse correctly reads 0.0 via the
real accounting chain -- a clean degrade, no special-casing, matching
today's pre-feature behavior exactly.
Test-only task; no production code changed. Per this spec's own
"pre-release product" scope note, no archived legacy snapshot file is
needed as a fixture -- a freshly-built wire job with allocs unset
exercises the identical code path.
Requirements: 3.5
Task 3.1: comprehensive integration test proving tasks 1.1/1.2/2.1/2.2
work together end to end -- a RUNNING job (multi-host, multi-resource,
plus a genuine shared-resource-pool component) and a PENDING job saved
together and loaded into a second, independent core. Asserts:
- Both jobs get a valid, non-NULL schedobj in the restored core.
- The running job's allocation/accounting content-match the source,
including the shared-pool component.
- Worker resInUse and core sharedInUse, computed via the real
sched_charge_account() chain (white-box included from resource.c,
following the established test_resource_accounting.c/test.c
link-only-stub convention), match exactly between source and
restored cores -- the actual proof of Req 1.2/1.3, not just that
job->accounts holds the right values.
- Destroying the source core's job list afterward does not corrupt or
crash the restored core (Req 1.4's independence guarantee exercised
via real teardown, not just pointer-distinctness checks).
Test-only task; no production code changed. Independently verified via
a live RED/GREEN toggle against task 2.2's load-side restoration block.
Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.3, 2.4
Task 2.2: xce_sched_core_load_snapshot() now restores job->allocs via
dup_job_alloc() (never aliasing the wire job's own copy, since it
belongs to a const, potentially-shared snapshot struct) and rebuilds
job->accounts via the existing make_job_account(), positioned right
after task 1.2's schedobj construction and before sched_add_job().
Independent of resreq parse success -- not nested inside that guard.
Absent allocation data is the normal case: no WARN, no error.
New tests prove a full cross-core round trip (save core_a, load into
independent core_b): restored allocs/accounts content-match the
source, with three-way pointer independence (restored vs source-core
vs saved-wire) proving the deep-copy guarantee end to end. A companion
test confirms a job with no allocation loads cleanly.
Note: this task's implementation was completed directly after two
implementer subagent dispatches hit transient API interruptions; one
auto-resumed and reported a false READY_FOR_REVIEW (claiming 9/9
passing tests while the actual production code was never written,
only the test file). Caught by fresh-evidence verification before
trusting the report -- confirmed via an independent build+test run
showing a genuine failure, then fixed and re-verified by an
independent reviewer who reproduced RED-to-GREEN itself.
Requirements: 1.1, 1.2, 1.3, 1.4, 3.5
Task 2.1: build_wire_job() now deep-copies job->allocs into the wire
job's already-existing allocs field via dup_job_alloc() (task 1.1) when
the live job has an allocation, leaving it unset otherwise -- matching
how every other optional field in this function is handled. Never
aliases job->allocs.
New tests prove: a running job with a multi-host allocation saves with
matching content (host names, resource values) and a pointer distinct
from the live job's own allocation; a job with no allocation saves with
the wire field left NULL, no error.
Requirements: 1.1
Task 1.2: snapshot_load.c gains a local, static field-for-field port of
core_events.c's core_build_schedobj()/core_build_primary_schedobj()/
core_build_compound_schedobj()/core_build_alter_schedobj() (the bug-1569
fix for the JOBS_PUSH path). Every job restored from a snapshot whose
resreq parsed successfully now gets a valid, non-NULL job->schedobj,
closing the same NULL-deref crash class for the snapshot-load path.
Guarded on job->resreq != NULL so the pre-existing, unrelated
resreq-parse-failure path is untouched (still leaves schedobj NULL,
same as today -- not fixed here, per design.md's documented boundary).
New tests prove non-NULL schedobj with correct tag/union member for
primary, compound, and alter resource requests, plus a regression test
confirming the parse-failure path's behavior is unchanged.
Requirements: 2.1, 2.2, 2.3, 2.4
