lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): sim resource over-commit, byte-unit display, sync job-state-at-capture

Browse source 
- bug-1799/1801: fix resInUse over-commit in sim replay -- sched_account_jobs's
  per-round re-derive pass now skips RECONCILING/UNAVAIL workers the same way
  core_reset_workers already does (bug-1751), closing the gap that let a
  worker's used slots exceed its own capacity across a resync replay
- bug-1804: revise scheduler-sim-eager-reconciliation spec (user-directed) so
  a captured job's job->state is set to RUN synchronously at dispatch capture,
  mirroring sched_commit_decision's normal-mode semantics (bug-1624), instead
  of deferring until its separately-recorded JOB_STATE_CHANGE event replays
- bug-1800: render bytes-valued CONSUMABLE resources (mem/swp/tmp) with the
  same human-scaled unit formatting already used for maxmem/maxswap/maxtmp
- bug-1802/1803: add then revert a break-on-ANY-event breakpoint kind after
  live feedback that it over-fires inside busy sessions; replace it with a
  prescan-based "next non-dummy session" hint in info summary instead

Full scheduler suite: 77/77 tests pass (asan profile).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-12 14:06:41 +00:00
parent ffca959ed9
commit 25fb5c115a
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
22 changed files with 1827 additions and 303 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/scheduler/include/sim.h
View file

@ -253,4 +253,23 @@ extern int sched_sim_driver_start(xce_sched_sim_driver_t* sim,
extern int sched_sim_driver_run(xce_sched_sim_driver_t* sim,
const sched_sim_args_t* args);
/*
* sim_prescan_next_nonempty_session - user-requested (2026-09-11): looks
* ahead in sim->replay_file (from its CURRENT read position) for the next
* ReplaySession boundary preceded by at least one real event -- i.e. the
* next "non-dummy" session -- without consuming/advancing the stream.
* `*out_sessions_ahead` is the number of session boundaries from right now
* to reach it (exactly the count `step N` needs), NOT an absolute session
* number -- this function only sees the raw replay stream, not
* xce_sched_repl_state_t's session_count (an opaque type here, fully
* defined only in repl_internal.h). repl_info.c's `info summary` adds this
* to rs->session_count itself for display, and reports the result so an
* operator can `step N` straight there instead of single-stepping through
* a run of dummy (virtual-clock-only) sessions first. See sim.c's own doc
* comment on this function for the full contract, return values, and
* scope decisions.
*/
extern int sim_prescan_next_nonempty_session(xce_sched_sim_driver_t* sim,
uint64_t* out_sessions_ahead);
#endif

127
src/scheduler/src/repl_info.c
View file

@ -292,14 +292,20 @@ static void repl_info_queue(xce_sched_sim_driver_t* sim, const char* arg)
* show -- `info resource`'s GLOBAL/SHARED-only scope (bug-1593) may
* genuinely have none of a given category for a given replay.
*
* The Consumable/Capacity value=/used= pair deliberately keeps its
* existing unconditional "%.2f" formatting (bug-1601/1613/1727's own
* pinned shapes, test_repl_breakpoints_inspection.c) and is NOT
* units_mode-aware -- Req 6.5's whole-number rule is explicitly scoped to
* non-consumable values only (test_repl_resource_table.c's own file header
* comment), and no bytes-valued CONSUMABLE resource is exercised anywhere
* in this codebase's tests today. Only a Load Metrics (non-consumable
* numeric) value goes through repl_format_numeric_resource_value() below.
* The Consumable/Capacity value=/used= pair keeps its existing
* unconditional "%.2f" formatting for a NON-bytes-valued resource (slots,
* licenses, ...) -- bug-1601/1613/1727's own pinned shapes,
* test_repl_breakpoints_inspection.c -- since Req 6.5's whole-number rule
* is explicitly scoped to non-consumable values only
* (test_repl_resource_table.c's own file header comment). A bytes-valued
* CONSUMABLE resource (mem/tmp/swp -- a live worker's own memory/swap/tmp
* capacity, all flagged CONSUMABLE in practice) DOES go through
* repl_format_numeric_resource_value() for both value= and used=, same as
* every other bytes-valued field in this listing (maxmem/maxswap/maxtmp,
* and mem/tmp/swp themselves when not flagged consumable via the Load
* Metrics path below) -- see
* repl_format_consumable_value_and_used()'s own doc comment for the exact
* split and the user-reported gap this closes (2026-09-11).
*/
typedef enum {
XCE_REPL_RES_SECTION_NONE = 0,
@ -414,6 +420,43 @@ static void repl_format_numeric_resource_value(xce_repl_units_mode_t mode,
}
}
/*
* repl_format_consumable_value_and_used - formats a CONSUMABLE numeric
* resource's value=/used= pair for repl_info_worker_one()'s per-host case
* and repl_info_resource_one()'s GLOBAL/SHARED case. A byte/megabyte-valued
* resource (repl_is_bytes_valued_resource(), or `swp`'s documented MB
* special case -- the SAME classification repl_format_numeric_resource_value()
* already applies to the Load Metrics section) renders BOTH `avail` and
* `used` through that same formatter, respecting `mode` -- so
* `used=87960930222080.00` (a live-reported bug, 2026-09-11) instead
* becomes e.g. `used=87960930222080.000000 (80.00 TB)` under the default
* "both" mode, matching how maxmem/maxswap/maxtmp already render alongside
* it. Every other consumable resource (slots, licenses, ...) keeps the
* existing pinned "%.2f"/"%.2f" shape unchanged -- bug-1601/1613/1727's own
* tests assert this exact format and must not regress.
*/
static void repl_format_consumable_value_and_used(xce_repl_units_mode_t mode,
const char* rname,
double avail, double used,
char* value_buf,
size_t value_bufsz,
char* used_buf,
size_t used_bufsz)
{
bool is_swp = (rname != NULL) && (strcmp(rname, "swp") == 0);
if (repl_is_bytes_valued_resource(rname) || is_swp) {
repl_format_numeric_resource_value(mode, rname, avail, value_buf,
value_bufsz);
repl_format_numeric_resource_value(mode, rname, used, used_buf,
used_bufsz);
return;
}
snprintf(value_buf, value_bufsz, "%.2f", avail);
snprintf(used_buf, used_bufsz, "%.2f", used);
}
/* repl_current_units_mode - sim->repl_state may legitimately be NULL (a few
* white-box tests dispatch `info worker`/`info resource` without ever
* calling repl_state_ensure()); default to the same XCE_REPL_UNITS_BOTH
@ -656,18 +699,25 @@ static void repl_info_worker_one(xce_sched_sim_driver_t* sim,
* user's exact reported numbers (value=44.00
* used=22.00 on a 22-slot worker at full
* occupancy: 22 (never-decremented total) + 22
* (used) = 44). Print avail_v as-is; NOT
* units_mode-aware (this section's own doc comment
* above explains why). */
* (used) = 44). Print avail_v as-is. A
* byte-valued resource (mem/tmp/swp) additionally
* renders both fields units_mode-aware, via
* repl_format_consumable_value_and_used() -- see
* that function's own doc comment. */
double avail = avail_v->u.fvalue;
double used = ((used_v != NULL) &&
(used_v->which == Value_u_fvalue))
? used_v->u.fvalue
: 0.0;
char value_str[80];
char used_str[80];
repl_format_consumable_value_and_used(
units_mode, rname, avail, used, value_str,
sizeof(value_str), used_str, sizeof(used_str));
fprintf(stdout,
"[sim] resource %s: value=%.2f used=%.2f\n",
rname, avail, used);
"[sim] resource %s: value=%s used=%s\n",
rname, value_str, used_str);
} else {
char valbuf[80];
@ -988,9 +1038,22 @@ static void repl_info_resource_one(xce_sched_sim_driver_t* sim,
* never decremented as jobs consume it (confirmed against
* resource.c's unsatisfy_() admission formula, which reads
* it as `total` and compares against `inuse` separately).
* Print it as-is instead of adding `used` on top. */
fprintf(stdout, "[sim] value=%.2f used=%.2f\n",
v->u.fvalue, used);
* Print it as-is instead of adding `used` on top. A
* byte-valued resource additionally renders both fields
* units_mode-aware, via the same
* repl_format_consumable_value_and_used() helper
* repl_info_worker_one() uses. */
{
char value_str[80];
char used_str[80];
repl_format_consumable_value_and_used(
repl_current_units_mode(sim), r->name, v->u.fvalue,
used, value_str, sizeof(value_str), used_str,
sizeof(used_str));
fprintf(stdout, "[sim] value=%s used=%s\n", value_str,
used_str);
}
} else {
/* Requirement 6.1/6.2/6.5: same Load Metrics rendering
* repl_info_worker_one() uses -- units_mode-aware for a
@ -1176,6 +1239,38 @@ static void repl_info_summary(xce_sched_sim_driver_t* sim)
"(%d online), %d queues, %d resources\n",
n_jobs, n_pending, n_running, n_workers, n_online, n_queues,
n_resources);
/*
* User-requested (2026-09-11, replacing a reverted `break ANY`
* breakpoint kind -- see repl_internal.h's doc comment on
* xce_repl_bp_kind_t): orient the operator toward the next session
* actually worth looking at, instead of leaving them to single-step
* through a run of dummy (virtual-clock-only, zero-event) sessions one
* at a time. sim_prescan_next_nonempty_session() (sim.c) is a pure
* read-only peek -- silently omitted (not an error) whenever it can't
* answer (no replay file, e.g. a bare in-memory test fixture with no
* `.replay` backing; or a genuine I/O error), matching
* sched_sim_driver_start()'s own "purely informational, silently
* skipped on failure" precedent for its sibling prescan.
*/
{
uint64_t sessions_ahead = 0;
int rc2 = sim_prescan_next_nonempty_session(sim, &sessions_ahead);
if (rc2 == 0) {
xce_sched_repl_state_t* rs = sim->repl_state;
uint64_t current = (rs != NULL) ? rs->session_count : 0;
fprintf(stdout,
"[sim] next non-dummy session: %llu -- run 'step %llu' "
"to reach it\n",
(unsigned long long)(current + sessions_ahead),
(unsigned long long)sessions_ahead);
} else if (rc2 == 1) {
fprintf(stdout,
"[sim] next non-dummy session: none remaining\n");
}
}
}
/*

11
src/scheduler/src/repl_internal.h
View file

@ -77,6 +77,17 @@ typedef enum {
* xce_core_event_t members -- BEGIN_ROUND is the ReplaySession boundary
* itself, DECISION is a drained xce_decision_t -- each gets its own
* breakpoint-kind value and its own matching path (repl_break.c).
*
* bug-1802 (2026-09-11): an XCE_REPL_BP_ANY convenience kind ("stop at the
* next session with any event, skipping dummy ones") was added then
* reverted the same session -- live use showed it fires on EVERY matching
* event inside an already-busy session too, forcing many single-event
* `step`s to get back OUT of that session instead of the single hop the
* feature was meant to provide. Replaced with `info summary` reporting the
* next non-dummy session number up front (repl_info.c's
* repl_info_summary(), sim.c's sim_prescan_next_nonempty_session()) so an
* operator can `step N` directly there instead -- no breakpoint needed at
* all. See buglog.json bug-1802/bug-1803 for the full history.
*/
typedef enum {
XCE_REPL_BP_CONFIG_PUSH = 1,

29
src/scheduler/src/resource.c
View file

@ -605,6 +605,31 @@ int sched_ctx_uncommit_host(xce_sched_ctx_t* ctx, int idx,
return xvec_idval(uncomit, committed, XVEC_SUB_FLOAT);
}
/*
* bug-1799 fix (2026-09-11): `w->resInUse` is a per-round "reset, then
* fully re-derive from every currently-RUN job" quantity -- core_session.c's
* core_reset_workers() (the reset half) deliberately SKIPS zeroing a
* worker's resInUse while it is IS_WORKER_UNAVAIL/IS_WORKER_RECONCILING
* (bug-1751, worker-rejoin-resync task 5), so its last known-good in-flight
* usage survives until the worker's real state is confirmed. sched_account_
* jobs() (scheduler.c) is the re-derive half -- called once per round via
* this function (sched_charge_account()) -- and had NO matching skip: it
* unconditionally re-added every RUN job's charge on top of whatever the
* (possibly not-reset) resInUse already held, so a worker that stayed
* RECONCILING/UNAVAIL across several rounds had its resInUse grow by one
* extra full charge PER ROUND instead of being cleanly re-summed --
* eventually exceeding the resource's real capacity (live-observed: a
* 22-slot worker showing used=26.00). Skipping the per-host charge for
* exactly the same two status bits core_reset_workers() already skips the
* reset for restores the "resum only what reset also touched" symmetry:
* while a worker is UNAVAIL/RECONCILING, its resInUse is simply left
* FROZEN at its last value (neither reset nor re-summed) until the status
* bit clears, at which point normal reset+resum resumes and self-corrects
* on the very next round. Scoped to the per-HOST loop only -- core->
* sharedInUse's own reset (core_reset_shared_resval(), core_session.c) is
* unconditional (no per-worker gating exists for it), so the shared-
* resource charge two lines below is intentionally left untouched.
*/
static int sched_update_account(xce_sched_core_t* core,
xce_job_account_t* account, int charge)
{
@ -626,6 +651,10 @@ static int sched_update_account(xce_sched_core_t* core,
continue;
}
if (IS_WORKER_UNAVAIL(w->status) || IS_WORKER_RECONCILING(w->status)) {
continue;
}
sched_update_res_accounts(core, &(w->resInUse), 0, haccount->naccount,
haccount->accounts, charge);
}

119
src/scheduler/src/sim.c
View file

@ -1,4 +1,5 @@
#include <errno.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@ -408,6 +409,124 @@ static int sim_prescan_session_count(xce_sched_sim_driver_t* sim,
return 0;
}
/*
* sim_prescan_next_nonempty_session - user-requested (2026-09-11, replacing
* a reverted `break ANY` breakpoint kind -- see repl_internal.h's own doc
* comment on xce_repl_bp_kind_t for why): looks ahead in sim->replay_file
* from the CURRENT read position (exactly what sim_loop.c's main loop will
* read next) for the next ReplaySession boundary that has at least one
* ReplayEvent preceding it in the stream -- a "non-dummy" session, as
* opposed to one that only advances the virtual clock with nothing else
* happening (e.g. a run of sessions that just tick the clock forward with
* no JOB_PUSH/WORKERS_PUSH/etc. in between, a common shape in a real
* captured replay). repl_info.c's repl_info_summary() reports the result so
* an operator can `step N` straight there instead of single-stepping
* through every dummy session first.
*
* Same ftell()/fseek()-around-a-shape-only-scan technique
* sim_prescan_session_count() above establishes -- see that function's own
* doc comment for why this is safe (capn_init_fp() is always unpacked, one
* plain fread() per message, no internal read-ahead buffer). Rewinds the
* file back to the exact starting position before returning in every case
* (found, not found, or error) -- this is a read-only peek, never an
* advance, so the real event loop's own subsequent read is unaffected.
*
* `*out_sessions_ahead` is populated with the number of ReplaySession
* boundaries from RIGHT NOW to (and including) the next non-dummy one --
* i.e. exactly the count `step N` needs -- only when a non-dummy session is
* actually found before EOF. Deliberately relative, not absolute: this
* function only sees sim->replay_file's raw message stream, not
* sim->repl_state (an opaque, repl_internal.h-only-complete type sim.c
* has no business depending on -- "Simulation isolation" boundary, same
* rationale repl_break.c/repl_info.c's own file headers document for the
* reverse direction). The caller (repl_info.c, which DOES have
* repl_state's full definition) adds this to rs->session_count itself to
* get the absolute "session=N" number for display.
*
* Returns: 0 if a non-dummy session was found (*out_sessions_ahead set,
* always >= 1); 1 if every remaining session before EOF is dummy
* (*out_sessions_ahead untouched); a negative errno on an I/O error
* (ftell/fseek failure).
*
* Scope decisions: (1) like sim_prescan_session_count(), only sees
* sim->replay_file's own stream -- a run under --sim-alt-events substitutes
* a different event stream at run time (sim_loop.c) that this scan does
* not see. (2) if called while paused MID-session (e.g. an event
* breakpoint fired partway through a busy session's own event batch), this
* scan cannot see events already consumed before the current file
* position, so it answers "how many MORE session boundaries from right
* now," not "is the in-progress session itself non-dummy" -- correct and
* exactly what `step N` needs for the overwhelmingly common case (paused
* cleanly at a session boundary between `step`/`continue` calls), a narrow
* gap only for the rarer mid-session-breakpoint case.
*/
int sim_prescan_next_nonempty_session(xce_sched_sim_driver_t* sim,
uint64_t* out_sessions_ahead)
{
FILE* f;
long resume_at;
uint64_t boundaries_seen = 0;
bool saw_event = false;
if ((sim == NULL) || (sim->replay_file == NULL)
|| (out_sessions_ahead == NULL)) {
return -EINVAL;
}
f = sim->replay_file;
resume_at = ftell(f);
if (resume_at < 0) {
return -errno;
}
for (;;) {
struct capn c;
capn_ptr root;
if (capn_init_fp(&c, f, 0) != 0) {
break; /* end of file (or malformed trailing data) */
}
if (msg_root_has_shape(&c, &root, ReplaySession_word_count,
ReplaySession_pointer_count)) {
boundaries_seen++;
capn_free(&c);
if (saw_event) {
if (fseek(f, resume_at, SEEK_SET) != 0) {
return -errno;
}
*out_sessions_ahead = boundaries_seen;
return 0;
}
continue;
}
if (msg_root_has_shape(&c, &root, ReplayEvent_word_count,
ReplayEvent_pointer_count)) {
saw_event = true;
capn_free(&c);
continue;
}
if (msg_root_has_shape(&c, &root, SnapshotEntry_word_count,
SnapshotEntry_pointer_count)) {
capn_free(&c); /* mid-replay checkpoint -- not a real event */
continue;
}
/* Unrecognized shape -- end of the message stream. */
capn_free(&c);
break;
}
if (fseek(f, resume_at, SEEK_SET) != 0) {
return -errno;
}
return 1;
}
int sched_sim_driver_start(xce_sched_sim_driver_t* sim,
const sched_sim_args_t* args)
{

55
src/scheduler/src/sim_loop.c
View file

@ -594,15 +594,26 @@ static void* sim_picker_thread_fn(void* arg)
* double-application guard. Any job in `sim->core.jobs` with `job->accounts
* == NULL` is eligible to be searched for a matching recorded DISPATCH
* decision this round, regardless of whether it is PEND or RUN -- a job's
* recorded DISPATCH decision and its RUN-state confirmation can legitimately
* land many rounds apart in the original recording (the dominant real-world
* case), and this function is called unconditionally every round, so a
* not-yet-running job is simply re-checked round after round until its
* originating decision is found (or the replay ends). Actual resource
* charging remains fully deferred to sched_account_jobs()'s own unchanged
* `JOB_IS_RUN` gate (scheduler.c) -- capturing a job's allocation here while
* it is not yet RUN is inert until that gate later includes it; this
* function performs no charging itself. Requirement 3.3's "report only the
* recorded DISPATCH decision and its separately-recorded RUN-state
* confirmation event can legitimately land many rounds apart in the
* original recording, and this function is called unconditionally every
* round, so a not-yet-captured job is simply re-checked round after round
* until its originating decision is found (or the replay ends).
*
* job->state transition (REVISED 2026-09-11, scheduler-sim-eager-
* reconciliation Requirement 1.2/1.3, user-directed): a successful FOUND
* match ALSO sets job->state = JOB_STAT_RUN synchronously, in the same
* operation as the allocation capture -- mirroring scheduler.c's
* sched_commit_decision() (bug-1624) exactly, which treats a job as
* running from the scheduler's own perspective the instant it decides to
* dispatch, independent of any later, separately-recorded worker-
* confirmation signal. sched_account_jobs()'s own unchanged `JOB_IS_RUN`
* gate (scheduler.c) therefore includes this job's charge starting THIS
* round, not a later one when its confirmation event happens to replay.
* (Previously this function left job->state untouched here, deferring the
* RUN transition entirely to a later JOB_STATE_CHANGE/JOB_UPDATE replay --
* see requirements.md's Requirement 1 2026-09-11 revision for the full
* rationale trail.) Requirement 3.3's "report only the
* fresh one" guarantee needs no separate mechanism here: a job that
* independently receives a fresh live decision in this same round already
* has job->accounts set by sched_commit_decision() (scheduler.c, unchanged)
@ -774,6 +785,32 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
entry->body.dispatch->alloc = NULL;
job->accounts = make_job_account(job->allocs);
/* User-directed revision (2026-09-11, scheduler-sim-eager-
* reconciliation requirements.md Requirement 1.2/1.3): set
* job->state = JOB_STAT_RUN synchronously, in the SAME
* operation as the allocation capture above -- mirroring
* scheduler.c's sched_commit_decision() (bug-1624) exactly,
* which treats a job as running from the scheduler's own
* perspective the instant it decides to dispatch, independent
* of whatever later, separately-recorded signal confirms the
* worker actually started it. Previously this function left
* job->state untouched here, deferring the RUN transition
* entirely to a later-replayed JOB_STATE_CHANGE/JOB_UPDATE
* event -- live use showed this as a confusing multi-session
* "DISPATCH already printed, but info jobs still shows PEND"
* gap that does not match what the original live scheduler
* process's own job->state field actually held at this exact
* point. sched_account_jobs()'s own unchanged JOB_IS_RUN gate
* (scheduler.c) now includes this job's charge starting THIS
* round, not a later one -- no change needed there. A later
* replay of the job's own separately-recorded confirmation
* event still applies via core_events.c's normal
* XCE_CORE_EVENT_JOB_STATE_CHANGE path; that handler's
* unconditional `job->state = new_state;` reassignment
* (RUN -> RUN) is a verified-safe no-op, not a double-charge
* or a terminal-guard violation. */
job->state = JOB_STAT_RUN;
/* bug-1759/bug-1779: mirrors sched_allocator()'s own ordering
* (scheduler.c) as closely as this reconstruction allows --
* remove the job from the picker's candidate set now that it is

95
src/scheduler/testsuite/test_repl_breakpoints_inspection.c
View file

@ -1902,6 +1902,99 @@ static void test_event_breakpoint_pauses_repl_at_matching_event(void)
printf("test_event_breakpoint_pauses_repl_at_matching_event: PASS\n");
}
/*
* test_info_summary_reports_next_nondummy_session_when_present -
* user-requested (2026-09-11, replacing a reverted `break ANY` breakpoint
* kind -- see repl_internal.h's own doc comment on xce_repl_bp_kind_t):
* `info summary`, called at the very start (before any `step`/`continue`,
* i.e. session_count==0), must report the next session that decodes at
* least one real event, skipping two fully-empty dummy sessions (rounds
* 1000 and 2000), plus the exact `step` count to reach it.
*/
static void
test_info_summary_reports_next_nondummy_session_when_present(void)
{
char replay_path[PATH_MAX];
char aux_dir[PATH_MAX];
char script_path[PATH_MAX];
char capture_path[PATH_MAX];
FILE* f;
char* captured;
f = write_header_and_snapshot(replay_path, sizeof(replay_path),
"_nextnd");
TEST_ASSERT_NOT_NULL(f);
append_replay_session(f, 1000ULL); /* session 1: dummy */
append_replay_session(f, 2000ULL); /* session 2: dummy */
append_jobstatechange_event(f, 2500ULL, 444ULL); /* session 3's event */
append_replay_session(f, 3000ULL); /* session 3: real */
append_end_of_stream_marker(f);
fclose(f);
make_tmp_dir(aux_dir, sizeof(aux_dir), "_nextndaux");
snprintf(script_path, sizeof(script_path), "%s/script.txt", aux_dir);
snprintf(capture_path, sizeof(capture_path), "%s/capture.out", aux_dir);
write_text_file(script_path, "info summary\nquit\n");
captured = run_group_d_and_capture(replay_path, script_path, capture_path);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, "next non-dummy session: 3 -- run 'step 3' to "
"reach it"),
"info summary must report session 3 (skipping the two dummy "
"sessions 1 and 2) and the exact step count (3) to reach it from "
"the very start (session_count==0)");
free(captured);
printf("test_info_summary_reports_next_nondummy_session_when_present: "
"PASS\n");
}
/*
* test_info_summary_reports_none_remaining_when_all_dummy - the file has
* exactly two sessions, BOTH dummy, and nothing after them -- `info
* summary` must say so plainly rather than reporting a bogus session
* number or silently omitting the line.
*/
static void
test_info_summary_reports_none_remaining_when_all_dummy(void)
{
char replay_path[PATH_MAX];
char aux_dir[PATH_MAX];
char script_path[PATH_MAX];
char capture_path[PATH_MAX];
FILE* f;
char* captured;
f = write_header_and_snapshot(replay_path, sizeof(replay_path),
"_nonend");
TEST_ASSERT_NOT_NULL(f);
append_replay_session(f, 1000ULL); /* session 1: dummy */
append_replay_session(f, 2000ULL); /* session 2: dummy */
append_end_of_stream_marker(f);
fclose(f);
make_tmp_dir(aux_dir, sizeof(aux_dir), "_nonendaux");
snprintf(script_path, sizeof(script_path), "%s/script.txt", aux_dir);
snprintf(capture_path, sizeof(capture_path), "%s/capture.out", aux_dir);
write_text_file(script_path, "info summary\nquit\n");
captured = run_group_d_and_capture(replay_path, script_path, capture_path);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, "next non-dummy session: none remaining"),
"info summary must plainly report that no non-dummy session "
"remains, not a bogus number or silence");
free(captured);
printf("test_info_summary_reports_none_remaining_when_all_dummy: "
"PASS\n");
}
static void test_decision_breakpoint_pauses_after_decision_output(void)
{
char replay_path[PATH_MAX];
@ -2149,6 +2242,8 @@ int main(void)
RUN_TEST(test_info_resource_all_reports_when_none_are_global_shared);
RUN_TEST(test_info_job_accepts_quoted_jobid_argument);
RUN_TEST(test_info_summary_reports_counts);
RUN_TEST(test_info_summary_reports_next_nondummy_session_when_present);
RUN_TEST(test_info_summary_reports_none_remaining_when_all_dummy);
RUN_TEST(test_info_session_reports_position_after_synthetic_events);
RUN_TEST(test_event_breakpoint_pauses_repl_at_matching_event);

402
src/scheduler/testsuite/test_repl_resource_table.c
View file

@ -1714,6 +1714,404 @@ test_info_resource_units_mode_both_renders_raw_and_human_for_bytes_resource(
"bytes_resource: PASS\n");
}
/*
* User-reported (2026-09-11, live replay session): a CONSUMABLE byte-valued
* per-host resource (mem/tmp/swp) rendered its value=/used= pair as a raw
* `%.2f` byte count (e.g. "used=87960930222080.00"), unlike every other
* bytes-valued field in this same listing (maxmem/maxswap/maxtmp, and mem/
* tmp/swp themselves when NOT flagged consumable, via the Load Metrics
* path) which already render human-scaled. repl_info_worker_one()'s own
* pre-fix header comment explicitly scoped this out ("no bytes-valued
* CONSUMABLE resource is exercised anywhere in this codebase's tests
* today") -- this test (and its "raw"/swp/info-resource siblings below)
* closes that gap. A non-bytes-valued consumable (slots/licenses) must keep
* its existing pinned "%.2f"/"%.2f" shape unchanged -- see
* test_info_worker_reports_consumable_value_and_used (bug-1727,
* test_repl_breakpoints_inspection.c), left untouched by this fix.
*/
static void
test_info_worker_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource(
void)
{
xce_sched_sim_driver_t sim;
xce_core_event_t events[1];
xce_resource_t wire_resource;
xce_resource_t* r;
xce_worker_t wire_worker;
xce_worker_t* worker_ptrs[1];
xce_val_t val;
xce_name_val_t nv;
xce_name_val_t* nv_ptrs[1];
xce_val_t used_val;
xce_sched_worker_t* sw;
char* captured;
char expected_value[64];
char expected_used[64];
char expected_line[160];
double avail;
double used;
int rc;
avail = 8589934592.0; /* 8 GB */
used = 4294967296.0; /* 4 GB */
memset(&sim, 0, sizeof(sim));
rc = xce_sched_core_init(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = "mem";
wire_resource.valtype = XCE_VAL_NUMERIC;
wire_resource.flags = XCE_RESF_CONSUMABLE;
r = makeResource(sim.core.resmgr, &wire_resource);
TEST_ASSERT_NOT_NULL(r);
memset(&val, 0, sizeof(val));
val.which = Value_u_fvalue;
val.u.fvalue = avail;
memset(&nv, 0, sizeof(nv));
nv.name = "mem";
nv.val = &val;
nv_ptrs[0] = &nv;
memset(&wire_worker, 0, sizeof(wire_worker));
wire_worker.name = "hostA";
wire_worker.status = 0;
wire_worker.nresvals_to_sched = 1;
wire_worker.resvals_to_sched = nv_ptrs;
worker_ptrs[0] = &wire_worker;
memset(events, 0, sizeof(events));
events[0].type = XCE_CORE_EVENT_WORKERS_PUSH;
events[0].u.workers_push.nworkers = 1;
events[0].u.workers_push.workers = worker_ptrs;
rc = xce_core_push_events(&(sim.core), events, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
sw = sched_get_worker(&(sim.core), "hostA");
TEST_ASSERT_NOT_NULL(sw);
memset(&used_val, 0, sizeof(used_val));
used_val.which = Value_u_fvalue;
used_val.u.fvalue = used;
zarray_put(&(sw->resInUse), r->idx, dup_val(&used_val));
sim.repl_state = repl_state_ensure(&sim);
TEST_ASSERT_NOT_NULL(sim.repl_state);
sim.repl_state->units_mode = XCE_REPL_UNITS_BOTH;
captured = capture_dispatch(&sim, "info worker hostA");
repl_format_bytes_human(avail, expected_value, sizeof(expected_value));
repl_format_bytes_human(used, expected_used, sizeof(expected_used));
snprintf(expected_line, sizeof(expected_line), "value=%f (%s) used=%f (%s)",
avail, expected_value, used, expected_used);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, expected_line),
"a CONSUMABLE byte-valued resource (mem/tmp/swp) must render both "
"value= and used= human-scaled under units_mode=both, the same "
"as every other bytes-valued field in this listing");
free(captured);
printf(
"test_info_worker_units_mode_both_renders_raw_and_human_for_"
"consumable_bytes_resource: PASS\n");
}
/*
* AC 6.2 'raw' counterpart for a CONSUMABLE byte-valued resource: under
* units_mode=raw, value=/used= must show ONLY the bare wire numbers, no
* "GB"/human suffix -- mirrors
* test_info_worker_units_mode_raw_renders_bare_wire_value_without_mutating_stored_value's
* contract, applied to the consumable value=/used= pair instead of a
* single value= line.
*/
static void
test_info_worker_units_mode_raw_renders_bare_values_for_consumable_bytes_resource(
void)
{
xce_sched_sim_driver_t sim;
xce_core_event_t events[1];
xce_resource_t wire_resource;
xce_resource_t* r;
xce_worker_t wire_worker;
xce_worker_t* worker_ptrs[1];
xce_val_t val;
xce_name_val_t nv;
xce_name_val_t* nv_ptrs[1];
xce_val_t used_val;
xce_sched_worker_t* sw;
char* captured;
char expected_line[160];
double avail;
double used;
int rc;
avail = 8589934592.0;
used = 4294967296.0;
memset(&sim, 0, sizeof(sim));
rc = xce_sched_core_init(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = "mem";
wire_resource.valtype = XCE_VAL_NUMERIC;
wire_resource.flags = XCE_RESF_CONSUMABLE;
r = makeResource(sim.core.resmgr, &wire_resource);
TEST_ASSERT_NOT_NULL(r);
memset(&val, 0, sizeof(val));
val.which = Value_u_fvalue;
val.u.fvalue = avail;
memset(&nv, 0, sizeof(nv));
nv.name = "mem";
nv.val = &val;
nv_ptrs[0] = &nv;
memset(&wire_worker, 0, sizeof(wire_worker));
wire_worker.name = "hostA";
wire_worker.status = 0;
wire_worker.nresvals_to_sched = 1;
wire_worker.resvals_to_sched = nv_ptrs;
worker_ptrs[0] = &wire_worker;
memset(events, 0, sizeof(events));
events[0].type = XCE_CORE_EVENT_WORKERS_PUSH;
events[0].u.workers_push.nworkers = 1;
events[0].u.workers_push.workers = worker_ptrs;
rc = xce_core_push_events(&(sim.core), events, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
sw = sched_get_worker(&(sim.core), "hostA");
TEST_ASSERT_NOT_NULL(sw);
memset(&used_val, 0, sizeof(used_val));
used_val.which = Value_u_fvalue;
used_val.u.fvalue = used;
zarray_put(&(sw->resInUse), r->idx, dup_val(&used_val));
sim.repl_state = repl_state_ensure(&sim);
TEST_ASSERT_NOT_NULL(sim.repl_state);
sim.repl_state->units_mode = XCE_REPL_UNITS_RAW;
captured = capture_dispatch(&sim, "info worker hostA");
snprintf(expected_line, sizeof(expected_line), "value=%f used=%f", avail,
used);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, expected_line),
"AC 6.2 'raw': a CONSUMABLE byte-valued resource must render bare "
"wire values for both value= and used= under units_mode=raw");
TEST_ASSERT_NULL_MESSAGE(
strstr(captured, "GB"),
"AC 6.2 'raw': must show no human-readable unit suffix");
free(captured);
printf(
"test_info_worker_units_mode_raw_renders_bare_values_for_"
"consumable_bytes_resource: PASS\n");
}
/*
* AC 6.1's swp-is-megabytes special case, applied to the CONSUMABLE
* value=/used= pair: both fields must be converted MB->bytes before
* human-scaling, mirroring
* test_info_worker_units_mode_converts_swp_megabytes_before_human_scaling's
* contract for the Load Metrics (non-consumable) path.
*/
static void
test_info_worker_units_mode_converts_swp_megabytes_for_consumable_resource(
void)
{
xce_sched_sim_driver_t sim;
xce_core_event_t events[1];
xce_resource_t wire_resource;
xce_resource_t* r;
xce_worker_t wire_worker;
xce_worker_t* worker_ptrs[1];
xce_val_t val;
xce_name_val_t nv;
xce_name_val_t* nv_ptrs[1];
xce_val_t used_val;
xce_sched_worker_t* sw;
char* captured;
char expected_value_human[64];
char expected_used_human[64];
double avail_mb;
double used_mb;
int rc;
avail_mb = 16384.0; /* 16384 MB -> 16 GB */
used_mb = 4096.0; /* 4096 MB -> 4 GB */
memset(&sim, 0, sizeof(sim));
rc = xce_sched_core_init(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = "swp";
wire_resource.valtype = XCE_VAL_NUMERIC;
wire_resource.flags = XCE_RESF_CONSUMABLE;
r = makeResource(sim.core.resmgr, &wire_resource);
TEST_ASSERT_NOT_NULL(r);
memset(&val, 0, sizeof(val));
val.which = Value_u_fvalue;
val.u.fvalue = avail_mb;
memset(&nv, 0, sizeof(nv));
nv.name = "swp";
nv.val = &val;
nv_ptrs[0] = &nv;
memset(&wire_worker, 0, sizeof(wire_worker));
wire_worker.name = "hostA";
wire_worker.status = 0;
wire_worker.nresvals_to_sched = 1;
wire_worker.resvals_to_sched = nv_ptrs;
worker_ptrs[0] = &wire_worker;
memset(events, 0, sizeof(events));
events[0].type = XCE_CORE_EVENT_WORKERS_PUSH;
events[0].u.workers_push.nworkers = 1;
events[0].u.workers_push.workers = worker_ptrs;
rc = xce_core_push_events(&(sim.core), events, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
sw = sched_get_worker(&(sim.core), "hostA");
TEST_ASSERT_NOT_NULL(sw);
memset(&used_val, 0, sizeof(used_val));
used_val.which = Value_u_fvalue;
used_val.u.fvalue = used_mb;
zarray_put(&(sw->resInUse), r->idx, dup_val(&used_val));
sim.repl_state = repl_state_ensure(&sim);
TEST_ASSERT_NOT_NULL(sim.repl_state);
sim.repl_state->units_mode = XCE_REPL_UNITS_HUMAN;
captured = capture_dispatch(&sim, "info worker hostA");
repl_format_bytes_human(avail_mb * 1024.0 * 1024.0, expected_value_human,
sizeof(expected_value_human));
repl_format_bytes_human(used_mb * 1024.0 * 1024.0, expected_used_human,
sizeof(expected_used_human));
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, expected_value_human),
"swp's value= must be converted MB->bytes before human-scaling, "
"same as the Load Metrics path");
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, expected_used_human),
"swp's used= must be converted MB->bytes before human-scaling too");
free(captured);
printf(
"test_info_worker_units_mode_converts_swp_megabytes_for_"
"consumable_resource: PASS\n");
}
/*
* info resource's counterpart: a GLOBAL/SHARED CONSUMABLE byte-valued
* resource must render its value=/used= pair human-scaled too, mirroring
* test_info_resource_units_mode_both_renders_raw_and_human_for_bytes_resource
* (the non-consumable case) and this file's own info-worker tests above.
*/
static void
test_info_resource_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource(
void)
{
xce_sched_sim_driver_t sim;
xce_core_event_t events[1];
xce_resource_t wire_mem;
xce_resource_t* r;
xce_val_t val_mem;
xce_resval_t rv_mem;
xce_resval_t* rv_ptrs[1];
xce_val_t* stored;
xce_val_t used_val;
char* captured;
char expected_value[64];
char expected_used[64];
char expected_line[160];
double avail;
double used;
int rc;
avail = 8589934592.0; /* 8 GB */
used = 2147483648.0; /* 2 GB */
memset(&sim, 0, sizeof(sim));
rc = xce_sched_core_init(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&wire_mem, 0, sizeof(wire_mem));
wire_mem.name = "mem";
wire_mem.valtype = XCE_VAL_NUMERIC;
wire_mem.flags = XCE_RESF_SHARED | XCE_RESF_CONSUMABLE;
r = makeResource(sim.core.resmgr, &wire_mem);
TEST_ASSERT_NOT_NULL(r);
memset(&val_mem, 0, sizeof(val_mem));
val_mem.which = Value_u_fvalue;
val_mem.u.fvalue = avail;
memset(&rv_mem, 0, sizeof(rv_mem));
rv_mem.name = "mem";
rv_mem.val = &val_mem;
rv_ptrs[0] = &rv_mem;
memset(events, 0, sizeof(events));
events[0].type = XCE_CORE_EVENT_RESVALS_FULL;
events[0].u.resvals_full.n = 1;
events[0].u.resvals_full.resvals = rv_ptrs;
rc = xce_core_push_events(&(sim.core), events, 1);
TEST_ASSERT_EQUAL_INT(0, rc);
memset(&used_val, 0, sizeof(used_val));
used_val.which = Value_u_fvalue;
used_val.u.fvalue = used;
zarray_put(&(sim.core.sharedInUse), r->idx, dup_val(&used_val));
sim.repl_state = repl_state_ensure(&sim);
TEST_ASSERT_NOT_NULL(sim.repl_state);
sim.repl_state->units_mode = XCE_REPL_UNITS_BOTH;
captured = capture_dispatch(&sim, "info resource mem");
repl_format_bytes_human(avail, expected_value, sizeof(expected_value));
repl_format_bytes_human(used, expected_used, sizeof(expected_used));
snprintf(expected_line, sizeof(expected_line), "value=%f (%s) used=%f (%s)",
avail, expected_value, used, expected_used);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, expected_line),
"a GLOBAL/SHARED CONSUMABLE byte-valued resource must render both "
"value= and used= human-scaled under units_mode=both");
stored = (xce_val_t*)zarray_get(&(sim.core.sharedInst), r->idx);
TEST_ASSERT_NOT_NULL(stored);
TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
(float)avail, (float)stored->u.fvalue,
"AC 6.4: displaying a resource under any units_mode must never "
"mutate the underlying stored/wire value");
free(captured);
printf(
"test_info_resource_units_mode_both_renders_raw_and_human_for_"
"consumable_bytes_resource: PASS\n");
}
int main(void)
{
UNITY_BEGIN();
@ -1724,6 +2122,9 @@ int main(void)
RUN_TEST(test_info_worker_units_mode_human_renders_scaled_value_without_mutating_stored_value);
RUN_TEST(test_info_worker_units_mode_both_renders_raw_and_human_without_mutating_stored_value);
RUN_TEST(test_info_worker_units_mode_converts_swp_megabytes_before_human_scaling);
RUN_TEST(test_info_worker_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource);
RUN_TEST(test_info_worker_units_mode_raw_renders_bare_values_for_consumable_bytes_resource);
RUN_TEST(test_info_worker_units_mode_converts_swp_megabytes_for_consumable_resource);
RUN_TEST(test_info_worker_assigned_job_listing_compresses_contiguous_array_indices);
RUN_TEST(test_info_worker_assigned_job_listing_appends_slot_count_when_uniform);
RUN_TEST(test_info_worker_assigned_job_listing_excludes_terminal_job_despite_stale_allocs);
@ -1731,6 +2132,7 @@ int main(void)
RUN_TEST(test_info_resource_groups_resources_into_labeled_table_sections_in_order);
RUN_TEST(test_info_resource_load_metric_whole_number_omits_trailing_zeros);
RUN_TEST(test_info_resource_units_mode_both_renders_raw_and_human_for_bytes_resource);
RUN_TEST(test_info_resource_units_mode_both_renders_raw_and_human_for_consumable_bytes_resource);
return UNITY_END();
}

81
src/scheduler/testsuite/test_resource_accounting.c
View file

@ -369,6 +369,86 @@ static void test_commit_job_alloc_identical_across_core_containers(void)
TEST_ASSERT_EQUAL_FLOAT(shared_sched[0], shared_sim[0]);
}
/*
* test_charge_account_skips_reconciling_or_unavail_worker_host_charge -
* bug-1799 fix: sched_charge_account()'s per-host resInUse charge must be
* skipped for a worker currently IS_WORKER_RECONCILING or IS_WORKER_UNAVAIL.
* core_reset_workers() (core_session.c, bug-1751) already skips RESETTING
* such a worker's resInUse every round; sched_account_jobs()'s own
* per-round re-derive pass reaches resInUse only through this same
* sched_charge_account()->sched_update_account() path, so without a
* matching skip here it re-added a full charge ON TOP of the not-reset
* total every round the worker stayed RECONCILING/UNAVAIL -- resInUse grew
* unboundedly and could exceed the resource's real capacity (live-observed:
* a 22-slot worker showing used=26.00). core->sharedInUse has no
* per-worker gating at all (its own reset, core_reset_shared_resval(), is
* unconditional) and must keep charging normally regardless of any one
* host's status.
*/
static void
test_charge_account_skips_reconciling_or_unavail_worker_host_charge(void)
{
xce_scheduler_t sched;
xce_job_account_t* acct;
xce_sched_worker_t* w;
float shared_val[1];
float host_val[1];
int shared_idx[1] = { SHARED_RESIDX };
int host_idx[1] = { HOST_RESIDX };
int rc;
memset(&sched, 0, sizeof(sched));
setup_core_with_worker(&(sched.core), WORKER_NAME);
w = sched_get_worker(&(sched.core), WORKER_NAME);
TEST_ASSERT_NOT_NULL(w);
acct = make_test_account(WORKER_NAME, SHARED_RESIDX, SHARED_USED,
SHARED_RESERVED, HOST_RESIDX, HOST_USED,
HOST_RESERVED);
/* --- RECONCILING: shared charges normally, per-host charge skipped --- */
w->status = XCE_WORKER_RECONCILING;
rc = sched_charge_account(&(sched.core), acct);
TEST_ASSERT_EQUAL_INT(0, rc);
get_resvals(&(sched.core.sharedInUse), 1, shared_idx, shared_val);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
EXPECTED_SHARED_CHARGE, shared_val[0],
"shared resources have no per-worker gating and must still charge "
"while the host is RECONCILING");
get_resvals(&(w->resInUse), 1, host_idx, host_val);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, host_val[0],
"bug-1799: a RECONCILING worker's per-host resInUse must NOT be "
"charged -- core_reset_workers() already skips resetting it, so "
"charging on top would grow it every round it stays RECONCILING");
/* --- UNAVAIL: same per-host skip --- */
w->status = XCE_WORKER_UNAVAIL;
rc = sched_charge_account(&(sched.core), acct);
TEST_ASSERT_EQUAL_INT(0, rc);
get_resvals(&(w->resInUse), 1, host_idx, host_val);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, host_val[0],
"bug-1799: an UNAVAIL worker's per-host resInUse must NOT be "
"charged either -- the same reset-skip condition covers both bits");
/* --- status clears: normal charging resumes on the very next call --- */
w->status = 0;
rc = sched_charge_account(&(sched.core), acct);
TEST_ASSERT_EQUAL_INT(0, rc);
get_resvals(&(w->resInUse), 1, host_idx, host_val);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
EXPECTED_HOST_CHARGE, host_val[0],
"once the worker's status bit clears, charging must resume "
"normally on the very next call -- this is a per-call skip, not a "
"permanent block");
}
/*
* resource-accounting.md's reservation-vs-actual-usage reconciliation for
* a REAL (gauge-style) resource, e.g. "mem" -- sched_reconcile_host_
@ -1142,6 +1222,7 @@ int main(void)
UNITY_BEGIN();
RUN_TEST(test_charge_account_identical_across_core_containers);
RUN_TEST(test_commit_job_alloc_identical_across_core_containers);
RUN_TEST(test_charge_account_skips_reconciling_or_unavail_worker_host_charge);
RUN_TEST(
test_reconcile_host_resource_usage_below_reservation_keeps_reservation);
RUN_TEST(

185
src/scheduler/testsuite/test_sim_contention_parity.c
View file

@ -1380,49 +1380,70 @@ test_contention_parity_matches_real_normal_mode_recording_round_by_round(
}
/* ======================================================================
* scheduler-sim-eager-reconciliation task 3.1 (Req 1.1-1.4): a resource-
* contention fixture where a job's recorded DISPATCH decision and its
* JOB_STATE_CHANGE-to-RUN confirmation are separated by SEVERAL rounds --
* not the same-round shape the test above already covers (there,
* scheduler-sim-eager-reconciliation task 3.1 (Req 1.1-1.4), REVISED
* 2026-09-11 (user-directed, requirements.md Requirement 1's revision):
* a resource-contention fixture where a job's recorded DISPATCH decision
* and its JOB_STATE_CHANGE-to-RUN confirmation are separated by SEVERAL
* rounds -- not the same-round shape the test above already covers (there,
* JOB_STATE_CHANGE lands in the SAME round window as the matching DISPATCH
* decision, which is exactly the shape the OLD, pre-task-1.1 trigger gate
* (`!JOB_IS_RUN(job->state)` in sim_reconcile_unaccounted_jobs(), removed
* by task 1.1) already handled correctly). This is the genuinely NEW
* scenario the OLD gate could never reconcile: jobs A and C are PEND at
* the exact round their own DISPATCH decision is recorded (round 1), and
* only reach RUN several rounds later (rounds 4 and 5 respectively) via a
* bare JOB_STATE_CHANGE event with no DISPATCH decision of its own -- by
* which point round 1's own decoded_session (the only place their
* DISPATCH ever existed) is long freed. Under the OLD gate, job->accounts
* would stay NULL forever for both jobs, and sched_account_jobs()'s own
* JOB_IS_RUN + accounts!=NULL charging gate would therefore never charge
* them -- resInUse would read a PERMANENTLY-ZERO 0.0 for the rest of the
* replay, wrongly making the worker's full WORKER_CAPACITY look available
* even once both jobs are genuinely running. See this task's own Status
* Report for the live revert-rebuild confirmation that this exact fixture
* reproduces that failure against a reverted tree.
* by task 1.1) already handled correctly). Jobs A and C are PEND at the
* exact round their own DISPATCH decision is recorded (round 1), and only
* reach a SEPARATELY-RECORDED RUN confirmation several rounds later
* (rounds 4 and 5 respectively) via a bare JOB_STATE_CHANGE event with no
* DISPATCH decision of its own -- by which point round 1's own
* decoded_session (the only place their DISPATCH ever existed) is long
* freed.
*
* Pre-task-1.1: job->accounts stayed NULL forever for both jobs (the OLD
* `!JOB_IS_RUN` gate never re-searched a PEND job past its own round), so
* resInUse read a PERMANENTLY-ZERO 0.0 for the rest of the replay.
*
* Task 1.1 through 2026-09-07: `sim_reconcile_unaccounted_jobs()` captured
* job->allocs/job->accounts at round 1 regardless of state, but LEFT
* job->state untouched -- charging stayed deferred to sched_account_jobs()'s
* unchanged JOB_IS_RUN gate until the SEPARATELY-recorded JOB_STATE_CHANGE
* event replayed at round 4/5, several rounds after capture.
*
* 2026-09-11 revision (current, this fixture): capture now ALSO sets
* job->state = JOB_STAT_RUN synchronously (mirroring scheduler.c's
* sched_commit_decision(), bug-1624) -- so BOTH jobs A and C are
* genuinely RUN and charged starting round 1 itself, the SAME round their
* DISPATCH decision is captured, not deferred to rounds 4/5. The later,
* separately-recorded JOB_STATE_CHANGE-to-RUN events at rounds 4/5 are
* still replayed (this fixture keeps them, unchanged, to prove they
* remain safe/idempotent) but no longer change resInUse at all -- they
* re-affirm an already-RUN job. This test's core value survives
* unchanged: prove capacity accounting is CORRECT across a realistic
* multi-round dispatch-to-confirmation gap, not permanently wrong for the
* rest of the replay (the pre-task-1.1 bug) -- but "correct" now means
* "already fully accounted from round 1", not "deferred until
* confirmation."
*
* Fixture shape (reuses this file's own established constants verbatim --
* WORKER_CAPACITY=10.0, JOBA_WANT=8.0, JOBB_WANT=5.0, JOBC_WANT=2.0,
* JOBD_WANT=1.0 -- never re-typed):
* Round 1 (begin_round=2000): A/B/C pushed PEND; A(8.0) and C(2.0)
* dispatched (A+C=10.0, exactly saturating capacity), B genuinely
* denied. NEITHER A NOR C's JOB_STATE_CHANGE appears anywhere near
* this round.
* dispatched (A+C=10.0, exactly saturating capacity) -- BOTH
* genuinely RUN and charged from this round onward (2026-09-11
* revision), B genuinely denied. NEITHER A NOR C's JOB_STATE_CHANGE
* appears anywhere near this round.
* Round 2, 3 (begin_round=3000/4000): pure filler -- B is genuinely
* re-offered and re-denied each round (builtin_fcfs's own documented
* "no-op, retried next round" contract), widening the gap before A/C's
* eventual confirmations below to genuinely "several rounds", and
* proving the deferred-zero state from round 1 persists correctly
* across MULTIPLE idle rounds, not just one.
* Round 4 (begin_round=5000): job A's JOB_STATE_CHANGE to RUN -- 3
* rounds after its own round-1 DISPATCH. Job C is still PEND -- only
* PARTIAL usage (8.0 of 10.0) must be reflected from this round on.
* Round 5 (begin_round=6000): job C's JOB_STATE_CHANGE to RUN -- 4
* rounds after ITS OWN round-1 DISPATCH (a deliberately DIFFERENT
* separation than job A's 3 rounds, proving this isn't a single-job
* coincidence). Both jobs now genuinely RUN -- FULL usage (10.0 of
* 10.0) must be reflected from this round on.
* eventual (now redundant) confirmations below to genuinely "several
* rounds", and proving the round-1 charge persists correctly across
* MULTIPLE idle rounds, not just one.
* Round 4 (begin_round=5000): job A's SEPARATELY-recorded JOB_STATE_
* CHANGE to RUN replays -- 3 rounds after its own round-1 DISPATCH.
* A safe no-op re-affirmation (already RUN since round 1); resInUse
* stays at the SAME full 10.0 it already reached at round 1.
* Round 5 (begin_round=6000): job C's SEPARATELY-recorded JOB_STATE_
* CHANGE to RUN replays -- 4 rounds after ITS OWN round-1 DISPATCH (a
* deliberately DIFFERENT separation than job A's 3 rounds, proving
* this isn't a single-job coincidence). Also a safe no-op; resInUse
* is unchanged, still 10.0.
*
* The single test function below drives this SAME fixture shape three
* times, truncated after round 1, round 4, and round 5 respectively (via
@ -1597,41 +1618,43 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
g_capturing = 0;
g_round_idx = 0;
/* ---- Checkpoint 1: right after round 1 -- A and C are dispatched but
* still PEND (neither has reached RUN yet). Capacity must look FULLY
* free: a hypothetical job wanting the entire WORKER_CAPACITY must be
* granted, since nothing is genuinely running yet (Requirement 1.2, at
* the capacity-check OUTCOME level, not just a raw resInUse read). ---- */
/* ---- Checkpoint 1 (REVISED 2026-09-11): right after round 1 -- A and
* C are captured AND already genuinely RUN (synchronous capture-time
* transition, mirroring sched_commit_decision()/bug-1624). Capacity
* must already look FULLY SATURATED: a hypothetical job wanting the
* entire WORKER_CAPACITY must be DENIED, since both A and C are
* already counted from this round onward (Requirement 1.2/1.3, at the
* capacity-check OUTCOME level, not just a raw resInUse read). Under
* the PRE-2026-09-11 design this same check was granted (1) -- the
* inversion itself is the proof the revision landed. ---- */
run_multiround_contention_fixture(&sim1, 1, "_mr1");
rc = check_host_satisfy_want(&sim1, WORKER_CAPACITY);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, rc,
"round 1: A and C are dispatched-but-still-PEND -- capacity must "
"look FULLY free (a job wanting the entire WORKER_CAPACITY must be "
"granted), since neither job has genuinely reached RUN yet "
"(Requirement 1.2)");
0, rc,
"round 1 (2026-09-11 revision): A and C are captured AND already "
"genuinely RUN, synchronously with capture -- capacity must "
"already look FULLY SATURATED (a job wanting the entire "
"WORKER_CAPACITY must be DENIED), not deferred until a later "
"confirmation round (Requirement 1.2/1.3)");
/* ---- Checkpoint 2: right after round 4 -- job A confirmed RUN, 3
* rounds after its own round-1 DISPATCH; job C still PEND. Capacity
* must reflect PARTIAL usage: exactly JOBC_WANT (2.0) of headroom
* remains -- a job wanting exactly that must be granted, one wanting
* more (JOBB_WANT, 5.0) must be denied. This is the strongest single
* proof that combined accounting is correct, not just "some job got
* counted": an implementation that dropped job A's contribution (the
* OLD same-round-only bug) would wrongly grant JOBB_WANT here too. ---- */
/* ---- Checkpoint 2 (REVISED 2026-09-11): right after round 4 -- job
* A's separately-recorded confirmation event has now ALSO replayed,
* but changes nothing: A and C were both already RUN since round 1.
* Capacity must STILL be fully saturated -- even JOBC_WANT (2.0),
* previously grantable under the deferred design, must now be denied,
* since C itself has been counted since round 1 too. ---- */
run_multiround_contention_fixture(&sim4, 4, "_mr4");
rc = check_host_satisfy_want(&sim4, JOBC_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, rc,
"round 4: only job A (8.0) is genuinely RUN -- exactly JOBC_WANT "
"(2.0) of headroom must remain and must be granted");
0, rc,
"round 4 (2026-09-11 revision): both A and C have been genuinely "
"RUN and fully charged since round 1 -- even JOBC_WANT (2.0) must "
"now be denied, since there is no remaining headroom");
rc = check_host_satisfy_want(&sim4, JOBB_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 4: JOBB_WANT (5.0) exceeds the 2.0 remaining headroom -- "
"must be denied. A capacity check that wrongly dropped job A's "
"already-RUN 8.0 usage (the OLD same-round-only bug) would wrongly "
"grant this");
"round 4: JOBB_WANT (5.0) must also be denied -- capacity has been "
"fully saturated by A+C since round 1");
/* ---- Checkpoint 3: the full 5-round replay -- per-round resInUse
* snapshots (Requirement 1.1-1.4 across EVERY round, not just
@ -1661,37 +1684,41 @@ test_contention_capacity_check_correct_across_multi_round_dispatch_confirm_gap(
"starting snapshot -- resInUse must read 0.0 (task 4.6's own "
"load-time call, not a replayed round)");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[1],
"round 1: A/C dispatched but still PEND -- resInUse must read 0.0, "
"not fabricated or prematurely charged (Requirement 1.2)");
WORKER_CAPACITY, g_round_resinuse[1],
"round 1 (2026-09-11 revision): A(8.0) and C(2.0) are BOTH "
"captured and synchronously transitioned to RUN this round -- "
"resInUse must already reflect the FULL combined 10.0, not "
"deferred to a later confirmation round (Requirement 1.2/1.3)");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[2],
"round 2: idle filler round -- the deferred-zero state from round "
"1 must persist, not drift, across an EXTRA round");
WORKER_CAPACITY, g_round_resinuse[2],
"round 2: idle filler round -- the round-1 charge must persist "
"unchanged across an EXTRA round, not drift or reset");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, g_round_resinuse[3],
"round 3: a SECOND idle filler round -- still 0.0, widening the "
"gap before A's confirmation below to genuinely several rounds");
WORKER_CAPACITY, g_round_resinuse[3],
"round 3: a SECOND idle filler round -- still the full 10.0, "
"widening the gap before A's now-redundant confirmation below to "
"genuinely several rounds");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
JOBA_WANT, g_round_resinuse[4],
"round 4: job A's JOB_STATE_CHANGE to RUN just replayed, 3 rounds "
"after A's own round-1 DISPATCH -- resInUse must now reflect A's "
"8.0 usage (Requirement 1.3), and ONLY A's -- job C is still PEND");
WORKER_CAPACITY, g_round_resinuse[4],
"round 4: job A's separately-recorded JOB_STATE_CHANGE-to-RUN "
"event replays, 3 rounds after A's own round-1 DISPATCH -- a safe "
"no-op re-affirmation (A has been RUN and charged since round 1); "
"resInUse must remain exactly 10.0, not drop, not double-charge");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
WORKER_CAPACITY, g_round_resinuse[5],
"round 5: job C's JOB_STATE_CHANGE to RUN just replayed, 4 rounds "
"after C's own round-1 DISPATCH (a DIFFERENT separation than A's 3 "
"rounds, proving this isn't a single-job coincidence) -- resInUse "
"must now reflect the FULL combined 10.0 usage of both A and C "
"together");
"round 5: job C's separately-recorded JOB_STATE_CHANGE-to-RUN "
"event replays, 4 rounds after C's own round-1 DISPATCH (a "
"DIFFERENT separation than A's 3 rounds, proving this isn't a "
"single-job coincidence) -- also a safe no-op; resInUse must "
"remain exactly 10.0, not double-charge to 20.0");
rc = check_host_satisfy_want(&sim5, JOBD_WANT);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, rc,
"round 5: capacity is fully saturated by A+C (10.0 of 10.0) -- "
"even the smallest positive want (JOBD_WANT) must be denied. Under "
"the OLD same-round-only reconciliation gate, neither A nor C "
"would EVER have been captured (their DISPATCH and RUN-"
"round 5: capacity is fully saturated by A+C (10.0 of 10.0, since "
"round 1) -- even the smallest positive want (JOBD_WANT) must be "
"denied. Under the PRE-task-1.1 reconciliation gate, neither A nor "
"C would EVER have been captured (their DISPATCH and RUN-"
"confirmation rounds never coincide), so resInUse would read a "
"PERMANENTLY-ZERO 0.0 here and this check would WRONGLY grant "
"admission");

158
src/scheduler/testsuite/test_sim_reconcile_unaccounted.c
View file

@ -1036,46 +1036,43 @@ static void test_reconcile_null_args_are_a_safe_noop(void)
}
/* ---------------------------------------------------------------------- */
/* Test 12 (scheduler-sim-eager-reconciliation task 2.1, Req 1.2, 1.3, */
/* 1.4): resource-in-use accounting must defer to the job's REAL running */
/* state, not its capture time. task 1.1 widened sim_reconcile_unaccounted_ */
/* jobs()'s trigger so a job's allocation is captured (job->allocs/ */
/* job->accounts set) as soon as its recorded DISPATCH decision is found, */
/* regardless of job->state -- this test proves the OTHER half of that */
/* design's correctness: sched_account_jobs() (scheduler.c, UNCHANGED, */
/* real body pulled in above) still gates charging on JOB_IS_RUN(state), */
/* so a captured-but-still-PEND job contributes nothing to w->resInUse */
/* (Req 1.2) until it genuinely transitions to RUN, at which point the */
/* SAME captured allocation (never re-captured, never re-typed) is charged */
/* (Req 1.3). Req 1.4 ("matches normal-mode's equivalent-point reporting") */
/* is satisfied by construction: this test drives the exact same */
/* sched_account_jobs() normal mode's own driver.c calls at the identical */
/* round-boundary position (design.md's Architecture Integration note) -- */
/* no separate normal-mode comparison fixture is needed to prove a shared */
/* function behaves consistently with itself. */
/* Test 12 (scheduler-sim-eager-reconciliation task 4.1/4.2, Req 1.2, 1.3, */
/* 1.4, REVISED 2026-09-11, user-directed): a captured job's resource */
/* consumption must be reflected in resInUse starting the SAME round its */
/* dispatch decision is captured, not deferred to a later round's */
/* separately-recorded RUN-state confirmation event. sim_reconcile_ */
/* unaccounted_jobs() now sets job->state = JOB_STAT_RUN synchronously, */
/* in the same operation as the allocation capture -- mirroring */
/* scheduler.c's sched_commit_decision() (bug-1624) exactly. This */
/* INVERTS the pre-2026-09-11 version of this test (formerly */
/* test_captured_pend_job_excluded_from_resinuse_until_run), which */
/* asserted the OPPOSITE: that a captured-but-PEND job stayed PEND and */
/* contributed zero to resInUse until a SEPARATE, later job->state = RUN */
/* write. See requirements.md's Requirement 1 2026-09-11 revision and */
/* research.md's corresponding entry for the full rationale trail. */
/* ---------------------------------------------------------------------- */
static void
test_captured_pend_job_excluded_from_resinuse_until_run(void)
test_captured_job_included_in_resinuse_same_round_as_capture(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_sched_worker_t* w;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
xce_core_event_t ev;
int residx;
float got;
int rc;
int push_rc;
fixture_init(&sim);
residx = register_slots_resource(&(sim.core));
w = add_worker(&(sim.core), "worker1", residx, 10.0f);
/* PEND-state job, no known allocation yet -- exactly task 1.1's own
* widened trigger shape: its recorded DISPATCH decision has been found
* this round, but its RUN-state confirmation has not been replayed
* yet. */
/* PEND-state job, no known allocation yet -- its recorded DISPATCH
* decision is about to be found this round. */
job = add_job(&(sim.core), 7030ULL, JOB_STAT_PEND, NULL);
TEST_ASSERT_NULL(job->accounts);
TEST_ASSERT_NULL(job->allocs);
@ -1088,49 +1085,21 @@ test_captured_pend_job_excluded_from_resinuse_until_run(void)
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, job->allocs,
"sanity: the PEND job's recorded DISPATCH decision must be captured "
"(task 1.1's own widened gate) before this test's own checkpoints "
"mean anything");
"the job's recorded DISPATCH decision must be captured before this "
"test's own checkpoints mean anything");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "sanity: job->accounts must be built from the "
"captured allocation even though the job is still "
"PEND");
job->accounts,
"job->accounts must be built from the captured allocation");
TEST_ASSERT_TRUE_MESSAGE(
JOB_STAT_PEND == job->state,
"sanity: the job must still genuinely be PEND at this checkpoint -- "
"otherwise the next assertion (resInUse == 0) would be trivially "
"true for the wrong reason");
JOB_STAT_RUN == job->state,
"Requirement 1.2 (2026-09-11 revision): capturing a job's dispatch "
"decision must ALSO transition it to RUN synchronously, mirroring "
"scheduler.c's sched_commit_decision() (bug-1624) -- no separate "
"job->state write should be needed here");
/* Checkpoint 1 (Requirement 1.2): recompute resInUse from scratch via
* the REAL sched_account_jobs() -- a captured-but-PEND job must
* contribute nothing. */
rc = sched_account_jobs(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
got = -1.0f;
rc = get_resvals(&(w->resInUse), 1, &residx, &got);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
0.0f, got,
"Requirement 1.2: a job whose allocation was captured early (while "
"still PEND) must contribute ZERO to its worker's resInUse until it "
"genuinely transitions to RUN -- sched_account_jobs()'s own "
"JOB_IS_RUN gate (scheduler.c, unchanged by this feature) is the "
"sole mechanism deferring this, and must still be effective now "
"that capture happens earlier than before");
/* Transition the SAME job to RUN -- direct field write, same white-box
* technique test_sim_contention_parity.c's own capacity_aware_
* allocator_thread_fn() uses to model a worker-confirmed state change
* without driving the full plugin_job_state_change() machinery (out of
* this file's own boundary -- scheduler.c is not white-box included). */
job->state = JOB_STAT_RUN;
/* Checkpoint 2 (Requirement 1.3): recompute resInUse again -- the SAME
* captured allocation (jalloc, never re-captured -- sim_reconcile_
* unaccounted_jobs() is not called a second time here, proving this
* checkpoint's nonzero value comes from the EARLIER capture, not a
* fresh one) must now be charged. */
/* Checkpoint 1 (Requirement 1.3, 2026-09-11 revision): recompute
* resInUse via the REAL sched_account_jobs() -- the SAME round as
* capture must ALREADY reflect the charge, not a later one. */
rc = sched_account_jobs(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
@ -1139,14 +1108,65 @@ test_captured_pend_job_excluded_from_resinuse_until_run(void)
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
4.0f, got,
"Requirement 1.3: once the captured-early job genuinely transitions "
"to RUN, its worker's resInUse must reflect its allocation from "
"that point forward -- using the EXACT allocation amount (4.0) "
"captured earlier by sim_reconcile_unaccounted_jobs(), not a "
"freshly re-captured or re-typed value");
"Requirement 1.3 (2026-09-11 revision): a captured job's resInUse "
"charge must appear the SAME round its dispatch decision is "
"captured -- sched_account_jobs()'s unchanged JOB_IS_RUN gate now "
"sees the job as RUN immediately, since capture itself sets "
"job->state = JOB_STAT_RUN synchronously");
/* Checkpoint 2 (Requirement 1.4, new 2026-09-11): a LATER, separately-
* recorded JOB_STATE_CHANGE-to-RUN event for the SAME job (the
* historical worker-confirmation signal, still present in a real
* recording and still replayed) must be a safe, idempotent
* re-affirmation -- no double-charge, no crash, allocs/accounts
* untouched (since !JOB_IS_RUN(new_state) is false for RUN, the
* handler's cleanup branch is skipped). Drives the REAL
* XCE_CORE_EVENT_JOB_STATE_CHANGE handler (core_events.c), same
* technique test_reconcile_then_release_via_job_state_change_is_asan_
* clean() above already establishes. */
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 7030ULL;
ev.u.job_state_change.old_state = JOB_STAT_RUN;
ev.u.job_state_change.new_state = JOB_STAT_RUN;
push_rc = xce_core_push_events(&(sim.core), &ev, 1);
TEST_ASSERT_EQUAL_INT(0, push_rc);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, job->allocs,
"Requirement 1.4: a redundant RUN->RUN confirmation event must not "
"touch the already-captured allocation");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts,
"Requirement 1.4: a redundant RUN->RUN confirmation event must not "
"free the already-captured accounts");
/* core_reset_workers() (core_session.c, white-box #include'd above)
* mirrors the REAL per-round sequence sched_sim_driver_run() always
* uses (xce_core_begin_session() zeroes resInUse -> sched_account_jobs()
* re-sums it from scratch) -- production NEVER calls sched_account_jobs()
* a second time without this reset in between, so this call is what
* makes the following recompute a faithful round-4-boundary proof
* instead of an artificial double-charge test-methodology artifact. */
core_reset_workers(&(sim.core));
rc = sched_account_jobs(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
got = -1.0f;
rc = get_resvals(&(w->resInUse), 1, &residx, &got);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
4.0f, got,
"Requirement 1.4: after a proper reset-then-resum round boundary, "
"the redundant RUN->RUN confirmation event must NOT have changed "
"what gets charged -- still exactly 4.0, the same single "
"allocation, not fabricated or duplicated");
printf(
"test_captured_pend_job_excluded_from_resinuse_until_run: PASS\n");
"test_captured_job_included_in_resinuse_same_round_as_capture: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
@ -2121,7 +2141,7 @@ int main(void)
RUN_TEST(
test_reconcile_reports_reconstructed_decision_through_existing_channel);
RUN_TEST(test_reconcile_null_args_are_a_safe_noop);
RUN_TEST(test_captured_pend_job_excluded_from_resinuse_until_run);
RUN_TEST(test_captured_job_included_in_resinuse_same_round_as_capture);
RUN_TEST(
test_reconcile_still_skips_already_captured_job_regardless_of_state);
RUN_TEST(test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries);