lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/src/scheduler/testsuite/test_sim_reconcile_unaccounted.c
Rongsong Shen 8931833132
feat(scheduler-sim-decision-trace): compile trace sources into REPL/sim tests and the sim isolation checks 
Task 5.1: the 26 white-box tests that include repl.c or sim_loop.c now
include the REPL trace sources (and the core trace sources where no
hooked source brings them); sched_sim_isolation_lib carries all trace
sources and the sim isolation test scans the REPL trace files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 14:56:31 +00:00

3027 lines
136 KiB
C
Raw Blame History

/*
* test_sim_reconcile_unaccounted.c - contract test for
* sim_reconcile_unaccounted_jobs() (sim_loop.c, scheduler-sim-decision-
* parity spec, tasks 2.2-2.4, "Recorded-Decision Reconciliation" component,
* design.md; Requirement 1.1, 1.2, 3.1, 3.2, 3.3, 4.1, 4.2). Also covers
* scheduler-sim-reconcile-terminal-guard task 1.1 (RED phase, Requirement
* 1.1, 1.2, 1.3; bug-1764): the two tests at the very bottom of this file
* (test_reconcile_never_reallocates_a_{done,exit}_job_despite_stray_
* dispatch_entry) prove a job driven to JOB_STAT_DONE/JOB_STAT_EXIT via
* xce_core_push_events() must never have its allocs/accounts resurrected by
* a later round's stray/duplicate recorded DISPATCH entry -- both are
* EXPECTED TO FAIL against today's sim_reconcile_unaccounted_jobs(), whose
* trigger (job->accounts == NULL alone) has no job->state check yet (task
* 1.2 adds it). test_reconcile_captures_non_run_state_job_when_dispatch_
* recorded() above already serves as this feature's non-terminal-case
* regression guard (Requirement 1.2) and needed no duplication.
* Also covers scheduler-sim-eager-reconciliation task 2.1 (Requirement 1.2, 1.3, 1.4):
* resource-in-use accounting via the REAL sched_account_jobs() (scheduler.c,
* unchanged by that spec, verbatim-copied in below -- see this file's own
* "White-box #include set" paragraph) correctly defers to a captured job's
* REAL running state, not its (now possibly much earlier) capture time.
*
* Scope: the once-per-round trigger (RUN-state job with job->accounts ==
* NULL), the jobid-based search through this round's
* decoded_session->decisions, the "not found" path -- leave the job
* unaccounted, report usage unavailable (== job->accounts staying NULL,
* which sched_account_jobs() already tolerates safely), never treat this as
* an error, and stay safe to re-check next round -- AND (task 2.3/2.4) a
* FOUND match's allocation reconstruction (ownership transfer, array
* compaction) plus feeding the reconstructed decision through the same
* three-call reporting channel (sched_record_session_add_decision(),
* sim_print_decision(), xce_repl_check_decision()) a freshly-drained
* decision uses, and that an already-accounted job produces NO duplicate
* report through this mechanism (Requirement 3.3).
*
* **Correction (post-review, scheduler-sim-repl-verbosity task 4.2
* remediation):** the reporting channel's console-print leg is now
* sim_print_decision_gate(sim, d), not a bare sim_print_decision(d) --
* review found this file's own reconciliation call site had been missed by
* task 4.2's original single-call-site scope (bug-1610). The two tests at
* the bottom of this file (test_reconcile_continue_mode_with_repl_state_
* suppresses_bare_decision_line / test_reconcile_step_mode_with_repl_
* state_still_prints_bare_decision_line) pin that gate's contract at THIS
* call site specifically, using the same repl_state_ensure()-based
* direct-call fixture technique test_repl_round_rollup.c's own fixtures
* establish. test_reconcile_reports_reconstructed_decision_through_
* existing_channel above (task 2.4's original test) deliberately leaves
* sim->repl_state NULL and continues to assert the bare line IS present --
* empirically confirmed unaffected by this remediation, since a NULL
* repl_state falls through the gate's own NULL-safety default ("never
* load-bearing, degrade to print") exactly like every other REPL hook in
* this codebase.
*
* sim_reconcile_unaccounted_jobs() is `static` and not yet called from
* anywhere in production code (task 3 wires it into
* sched_sim_driver_run()'s round-boundary sequence) -- called directly here
* instead, standalone, per design.md's own Testing Strategy.
*
* White-box #include set and stub set: originally IDENTICAL to
* test_sim_event_loop.c's own (same rationale -- sim_loop.c's compiled TU
* unconditionally references every one of these as externs, whether or not
* any single test here calls sched_sim_driver_run() itself). This file
* never calls sched_sim_driver_run()/spawns the picker thread, so (unlike
* test_sim_event_loop.c) no `static` storage workaround is needed for its
* own `xce_sched_sim_driver_t` fixtures. Task 2.1 (scheduler-sim-eager-
* reconciliation) additionally #includes ../src/resource.c and replaces the
* PREVIOUS no-op sched_account_jobs() stub with a REAL reimplementation
* (body copied verbatim from scheduler.c, same precedent test_sim_
* contention_parity.c's own copy documents -- scheduler.c itself is never
* white-box #included in this directory, too large a link-time dependency
* surface, see test_resource_accounting.c's own header comment) -- this
* file's own task 2.1 test is the first test here to actually call that
* function, and a no-op stub would defeat its entire premise. Pulling in
* resource.c makes sched_worker_release_host_ctx() a REAL (non-stub) symbol
* too (previously a local no-op here) and adds one new link-satisfying-only
* stub, sched_ctx_get_uncomit_host() -- see the stub definitions below for
* both.
*
* Fixture technique: xce_sched_core_init() plus core_build_job()/
* sched_add_job() (core_events.c/core.c, both white-box #included) to place
* a job directly into sim.core.jobs -- same technique
* write_header_and_snapshot_with_pending_job() (test_sim_event_loop.c)
* already establishes for baking a job straight into a core without a real
* `.replay` file. `decoded_session` (replay_session_t) and its
* `decisions[]` (replay_decision_t*) are hand-built directly in memory
* (calloc), not round-tripped through capnp -- sim_reconcile_unaccounted_
* jobs() only ever reads these plain heap structures, so no encode/decode
* step is needed to exercise it. Fixtures are not torn down (matches this
* whole suite's own "detect_leaks=0" convention, meson.build).
*/
#include <errno.h>
#include <limits.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <unity/unity.h>
#include "controller.capnp.h"
#include "kvmgr.h"
#include "resobj.h"
#include "scheduler.h"
#include "xstring.h"
/* Pull in the real functions under test into this TU (white-box #include,
* same combined set test_sim_event_loop.c's own header comment documents
* and justifies, symbol by symbol). */
#include "../src/core.c"
#include "../src/core_clock.c"
#include "../src/core_events.c"
#include "../src/core_session.c"
#include "../src/decision.c"
#include "../src/plugin.c"
#include "../src/plugins/builtin_fcfs.c"
#include "../src/plugins/limit.c" /* task 4.2 (scheduler-sim-live-allocation
* spec): pulled in for REAL (same file test_repl_limit_table.c's own
* fixtures already white-box #include) so this file's reconciliation tests
* can prove sim_reconcile_unaccounted_jobs()'s new plugin_job_election()
* call (sim_loop.c) genuinely registers a reconciled job's GLOBAL limit
* usage -- a stub returning NULL from limit_new_plugin() (this file's own
* previous stand-in, removed below) would defeat that test's entire
* premise. This supersedes the four link-satisfying stubs immediately
* below this include block in earlier revisions of this file
* (limit_get_configured_limits()/limit_get_usage_instances()/
* limit_new_plugin()/limit_seed_config()), which are now real symbols
* defined by this include and would collide at link time if left in
* place. */
#include "../src/recording.c"
#include "../src/recording_session.c"
#include "../src/repl.c"
#include "../src/repl_fmt.c"
#include "../src/repl_rollup.c"
#include "../src/repl_break.c"
#include "../src/repl_info.c"
#include "../src/repl_params.c"
#include "../src/repl_trace_render.c"
#include "../src/repl_trace_render_outcome.c"
#include "../src/repl_trace_store.c"
#include "../src/repl_trace.c"
#include "../src/repl_trace_emit.c"
#include "../src/resource.c"
/* Hooked sources compile SCHED_TRACE bodies (scheduler-sim-decision-trace). */
#include "../src/sched_trace.c"
#include "../src/sched_trace_scan.c"
#include "../src/sched_trace_capture.c"
#include "../src/sched_trace_outcome.c"
#include "../src/sim.c"
#include "../src/sim_loop.c"
#include "../src/sim_picker.c" /* bug-1567: sim_loop.c calls sim_bootstrap_picker() (sim_loop.c-cap sibling) as an extern -- undefined reference at link time otherwise */
#include "../src/sim_limit.c" /* bug-1825: sched_sim_driver_run() now also calls sim_bootstrap_limit_plugin() (sim_loop.c-cap sibling, sim_picker.c's own bug-1567 precedent) as an extern -- undefined reference at link time otherwise */
#include "../src/sim_record.c"
#include "../src/sim_verify.c"
#include "../src/snapshot.c"
#include "../src/snapshot_load.c"
#include "../src/snapshot_wire.c"
void kill_sched_job(void* arg) { (void)arg; }
void kill_sched_queue(void* arg) { (void)arg; }
void kill_sched_worker(void* arg) { (void)arg; }
void kill_sched_user(void* arg) { (void)arg; }
void kill_sched_project(void* arg) { (void)arg; }
void kill_sched_resval(xce_val_t* val) { (void)val; }
/* task 4.2 (scheduler-sim-live-allocation spec): limit_get_configured_
* limits()/limit_get_usage_instances()/limit_new_plugin()/limit_seed_
* config() are now REAL symbols, pulled in for real by the "../src/plugins/
* limit.c" white-box #include above -- the four link-satisfying stubs that
* previously stood in for them here (each returning NULL/0, since this
* file's fixtures never registered a real limit plugin) are removed, not
* left alongside the real definitions, which would collide at link time.
* sim_bootstrap_limit_plugin() (sim_limit.c, already included) now
* genuinely registers the limit plugin exactly as production does when a
* test's fixture calls it, which the new task 4.2 tests below rely on. */
/* sched_worker_release_host_ctx() is now a REAL (non-stub) symbol pulled in
* from resource.c (task 2.1 addition, below) -- no local stand-in needed or
* permitted (a local definition here would collide at link time). Unused by
* any test in this file (no worker is ever released), listed only because
* resource.c defines it -- same precedent test_resource_accounting.c's own
* header comment documents. */
/* sched_fill_limit_requests() (plugin_prepare_allocation()'s, plugin.c, the
* "limit" plugin category, linker dependency) is likewise now a REAL
* (non-stub) symbol pulled in from resource.c -- no local stand-in needed
* or permitted (a local definition here would collide at link time, same
* as sched_worker_release_host_ctx() above). Never invoked by any test in
* this file (no real allocation is ever driven), listed only because
* resource.c defines it. */
/* resource.c's sched_try_alloc_on_host_()'s own link-time dependency --
* never actually invoked (this file never drives real allocation, only
* calls sched_account_jobs()/sched_charge_account() directly). Same
* link-satisfying-only stub test_sim_contention_parity.c's own copy
* establishes needing once resource.c is white-box #included. */
int sched_ctx_get_uncomit_host(xvec_t** v, xce_sched_ctx_t* c, int i)
{
(void)v;
(void)c;
(void)i;
return 0;
}
/* Task 4.2 (scheduler-sim-live-allocation spec): plugins/limit.c's
* limit_job_election() calls reset_sched_obj() (../src/schedobj.c, not part
* of this white-box TU -- schedobj.c is not included here) -- a genuine
* link-time dependency of limit.c, not previously reachable in this file
* before this task's #include "../src/plugins/limit.c" addition above.
* Stubbed as a no-op purely to satisfy the linker, same precedent
* test_repl_limit_table.c's own identical stub establishes for the same
* symbol/reason. This is behaviorally correct here, not just a link
* placeholder: every job this file's fixtures build goes through the real
* core_build_job() (core_events.c, white-box #included above), which
* already xlist_init()s/zarray_init()s obj->glimits/hlimits fresh and
* empty -- the real reset_sched_obj() would only ever clear an EARLIER
* match's leftovers, which no job in this file's fixtures ever has (each
* test's job is freshly built and matched against its limit at most
* once), so skipping that clear changes nothing observable here. */
void reset_sched_obj(xce_sched_obj_t* obj) { (void)obj; }
/* Task 2.1 addition (scheduler-sim-eager-reconciliation, Req 1.2/1.3): a
* REAL (not no-op) reimplementation of scheduler.c's own sched_account_jobs()
* (scheduler.c:1258, scheduler-sim-decision-parity task 1.1's relocated
* xce_sched_core_t* signature) -- body copied verbatim, same precedent
* test_sim_contention_parity.c's own copy of this function documents
* (scheduler.c itself is never white-box #included in this directory --
* ~1400 lines, too large a link-time dependency surface, see
* test_resource_accounting.c's own header comment). This file's task 2.1
* test exists specifically to observe what THIS function does to
* w->resInUse for a captured-but-not-yet-RUN job vs. the same job once it
* transitions to RUN -- a no-op stub would defeat that test's entire
* premise, exactly like test_sim_contention_parity.c's own rationale for
* needing the real body. This supersedes the file's previous no-op stand-in
* (task 1.1 and earlier tests here never called this function themselves,
* so the no-op was harmless for them; task 2.1 is the first test in this
* file to call it). */
int sched_account_jobs(xce_sched_core_t* core)
{
iterator_t it;
int rc = 0;
rc = iter_make(&it, core->jobs);
if (rc < 0) {
return rc;
}
for (;;) {
xce_sched_job_t* job = NULL;
rc = iter_next(&it, (void**)&job);
if ((rc < 0) || (rc == RTTI_ITER_NO_DATA)) {
break;
}
if (!JOB_IS_RUN(job->state)) {
continue;
}
if (job->accounts != NULL) {
sched_charge_account(core, job->accounts);
}
}
if (rc == RTTI_ITER_NO_DATA) {
rc = 0;
}
iter_destroy(&it);
return rc;
}
/* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same
* copy every white-box test in this directory that does not link
* schedobj.c already establishes. */
int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out)
{
xce_sched_plugin_t* plugin = NULL;
int rc = 0;
if ((core == NULL) || (out == NULL)) {
return -EINVAL;
}
*out = NULL;
if (core->job_pick_plugin == NULL) {
return -EINVAL;
}
lock(core->plugins, RTTI_LOCK_READ);
rc = key_find(core->plugins, core->job_pick_plugin, (void**)&plugin);
unlock(core->plugins);
if (rc <= 0) {
return (rc < 0) ? rc : -ENOENT;
}
if ((plugin == NULL) || (plugin->picker_iface == NULL)
|| (plugin->picker_iface->next_job == NULL)
|| (plugin->picker_iface->begin_session == NULL)
|| (plugin->picker_iface->end_session == NULL)
|| (plugin->picker_iface->on_dispatch == NULL)
|| (plugin->picker_iface->on_pend == NULL)) {
return -EINVAL;
}
*out = plugin;
return 0;
}
/* scheduler-sim-live-allocation spec, task 4.1: sim_loop.c's real
* sched_sim_driver_run() (white-box #include'd into this TU above) now
* spawns sched_job_picker()/sched_allocator() (sched_election.c) directly
* instead of its own, now-removed sim_picker_thread_fn() -- an undefined
* reference at link time in this white-box TU otherwise, since
* sched_election.c's own sched_allocator() calls the REAL
* sched_make_allocation()/sched_commit_decision() (scheduler.c), neither
* part of this file's own hand-picked white-box #include set (this file's
* own header comment lists that set). This file's own test(s) do not
* exercise genuine resource-matching allocation, so this link-satisfying
* pair faithfully reproduces the exact PEND-only contract the REMOVED
* sim_picker_thread_fn() itself used to provide -- split across the SAME
* two-thread jbuf hand-off the real production pair now uses (one
* producer, one consumer, matching the project's jbuf capacity-1
* rendezvous invariant) instead of the old single fused thread -- so this
* file's own existing assertions are unaffected by this task's production
* change. Genuine resource-based dispatch through the real, unmodified
* sched_job_picker()/sched_allocator() pair is proved separately by
* test_sim_live_allocation.c (task 4.1's own dedicated fixture, which
* links the real scheduler_srcs instead of white-boxing individual files,
* precisely so it can reach the real allocation chain this stub
* deliberately does not). */
void* sched_job_picker(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
xce_sched_plugin_t* picker = NULL;
for (;;) {
int go = 0;
uint64_t jobid = 0;
cbuf_get(&(core->wbuf), &go);
if (sched_resolve_job_picker(core, &picker) < 0) {
jobid = 0;
cbuf_put(&(core->jbuf), &jobid);
continue;
}
picker->picker_iface->begin_session(picker);
for (;;) {
xce_sched_job_t* job = picker->picker_iface->next_job(picker);
if (job == NULL) {
break;
}
jobid = job->jobid;
cbuf_put(&(core->jbuf), &jobid);
}
jobid = 0;
cbuf_put(&(core->jbuf), &jobid);
picker->picker_iface->end_session(picker);
}
return NULL;
}
void* sched_allocator(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
xce_sched_plugin_t* picker = NULL;
for (;;) {
uint64_t jobid = 0;
xce_decision_t* decision = NULL;
xce_sched_job_t* job;
cbuf_get(&(core->jbuf), &jobid);
if (jobid == 0) {
decision = NULL;
cbuf_put(&(core->decisions), &decision);
continue;
}
job = sched_get_job(core, jobid);
if (job == NULL) {
continue;
}
(void)sched_resolve_job_picker(core, &picker);
if (picker != NULL) {
picker->picker_iface->on_pend(picker, job->jobid);
}
decision = make_pend_decision(make_pendreason(job->jobid));
cbuf_put(&(core->decisions), &decision);
}
return NULL;
}
/* scheduler-sim-live-allocation spec, task 4.1 (bug-1829 fix): sim_loop.c's
* real sched_sim_driver_run() (white-box #include'd into this TU above) now
* also calls sim_bootstrap_seqjob_allocator()/sim_bootstrap_parjob_
* allocator() (sim_alloc.c) -- undefined references at link time in this
* white-box TU otherwise, since neither symbol is part of this file's own
* hand-picked white-box #include set. This file's own test(s) exercise the
* link-satisfying sched_job_picker()/sched_allocator() stub pair above
* (which never reaches real primary allocation), so a true no-op is the
* correct, behavior-preserving stand-in here -- the real bootstrap logic is
* exercised separately by test_sim_live_allocation.c (task 4.1's own
* dedicated fixture, which links the real scheduler_srcs). */
void sim_bootstrap_seqjob_allocator(xce_sched_sim_driver_t* sim)
{
(void)sim;
}
void sim_bootstrap_parjob_allocator(xce_sched_sim_driver_t* sim)
{
(void)sim;
}
void setUp(void) { return; }
void tearDown(void) { return; }
/* ---------------------------------------------------------------------- */
/* Fixture helpers */
/* ---------------------------------------------------------------------- */
static void fixture_init(xce_sched_sim_driver_t* sim)
{
int rc;
memset(sim, 0, sizeof(*sim));
rc = xce_sched_core_init(&(sim->core));
TEST_ASSERT_EQUAL_INT(0, rc);
}
/* Bakes one job directly into `core->jobs` (no `.replay` file, no capnp --
* same core_build_job()+sched_add_job() technique test_sim_event_loop.c's
* own write_header_and_snapshot_with_pending_job() establishes). Passing a
* non-NULL `allocs` reproduces the "already accounted" job shape (mirrors
* core_build_job()'s own `if (j->allocs != NULL) { ...; job->accounts =
* make_job_account(job->allocs); }` branch, core_events.c) -- pass NULL for
* the ordinary "unaccounted RUN-state job" shape this file's central tests
* target. */
static xce_sched_job_t* add_job(xce_sched_core_t* core, uint64_t jobid,
unsigned int state, xce_job_alloc_t* allocs)
{
xce_queue_t wire_queue;
xce_sched_queue_t* q;
job_submit_data_t submit;
xce_job_t wire_job;
xce_sched_job_t* job;
int queue_rc;
memset(&wire_queue, 0, sizeof(wire_queue));
wire_queue.queue = (char*)"normal";
wire_queue.status = QUEUE_STAT_OPEN;
wire_queue.priority = 1;
q = core_build_queue(&wire_queue);
TEST_ASSERT_NOT_NULL(q);
/* Tolerate -EEXIST: a test placing MULTIPLE jobs on the same core (task
* 1.2's own multi-job isolation test is the first here to do so) calls
* add_job() more than once against the same `core`, and the "normal"
* queue this helper (re-)builds every call is already registered from
* the first call -- that is expected, not a fixture bug, so only a
* genuine failure (neither success nor EEXIST) is a hard assertion
* failure. The freshly-built-but-unused `q` on the EEXIST path is
* simply leaked, matching this whole suite's own "fixtures are not
* torn down, detect_leaks=0" convention documented in this file's own
* header comment. */
queue_rc = sched_add_queue(core, q);
TEST_ASSERT_TRUE_MESSAGE(
(queue_rc >= 0) || (queue_rc == -EEXIST),
"sched_add_queue must succeed, or fail with -EEXIST when this "
"fixture core's \"normal\" queue was already registered by an "
"earlier add_job() call against the same core");
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)"normal";
submit.user = (char*)"alice";
submit.mintasks = 1;
submit.maxtasks = 1;
memset(&wire_job, 0, sizeof(wire_job));
wire_job.jobid = jobid;
wire_job.state = state;
wire_job.submit = &submit;
wire_job.allocs = allocs;
job = core_build_job(core, &wire_job);
TEST_ASSERT_NOT_NULL(job);
TEST_ASSERT_GREATER_OR_EQUAL(0, sched_add_job(core, job));
return job;
}
static replay_decision_t* make_dispatch_entry(uint64_t jobid)
{
xce_job_alloc_t* alloc = make_job_alloc(jobid, 0, NULL);
sched_dispatch_decision_t* dispatch
= (sched_dispatch_decision_t*)calloc(1, sizeof(*dispatch));
replay_decision_t* rd = (replay_decision_t*)calloc(1, sizeof(*rd));
dispatch->alloc = alloc;
rd->body_which = ReplayDecision_body_dispatch;
rd->body.dispatch = dispatch;
return rd;
}
static replay_decision_t* make_pend_entry(uint64_t jobid)
{
sched_pend_decision_t* pend
= (sched_pend_decision_t*)calloc(1, sizeof(*pend));
replay_decision_t* rd = (replay_decision_t*)calloc(1, sizeof(*rd));
pend->jobid = jobid;
pend->reason = NULL;
rd->body_which = ReplayDecision_body_pend;
rd->body.pend = pend;
return rd;
}
static replay_session_t* make_decoded_session(uint64_t begin_round,
replay_decision_t* entry)
{
replay_session_t* s = (replay_session_t*)calloc(1, sizeof(*s));
s->begin_round = begin_round;
s->summary = NULL;
if (entry == NULL) {
s->decisions = NULL;
s->n_decisions = 0;
return s;
}
s->decisions = (replay_decision_t**)calloc(1, sizeof(*(s->decisions)));
s->decisions[0] = entry;
s->n_decisions = 1;
return s;
}
/* task 2.3 fixture helper: a real, one-host/one-resource xce_job_alloc_t --
* same field shape job_alloc_eq() (xcealloc.h) already validates (design.md's
* Allowed Dependencies), built via the real xcealloc.c production helpers
* (make_alloc/make_allocs/make_host_alloc/make_host_allocs/make_job_alloc),
* not a hand-rolled struct literal -- so a reconciled job's `job->accounts`
* shape can be checked against real, non-trivial resource values.
* `from` stays at make_alloc()'s time(NULL) default: these recorded
* DISPATCH entries only feed sim_reconcile_unaccounted_jobs() and are never
* compared against a live replay's allocation (whose `from` would be the
* session start in whole seconds, libxce-alloc-preason-equality
* Decision 8). */
static xce_job_alloc_t* make_test_job_alloc_with_host(uint64_t jobid,
const char* host,
const char* resname,
float value)
{
xce_alloc_t* a = make_alloc((char*)resname, NULL, 0, value, 0);
xce_alloc_t** allocs = make_allocs(1, a);
xce_host_alloc_t* h = make_host_alloc((char*)host, 1, allocs);
xce_host_alloc_t** hallocs = make_host_allocs(1, h);
return make_job_alloc(jobid, 1, hallocs);
}
/* task 2.3 fixture helper: wraps an already-built xce_job_alloc_t (as
* opposed to make_dispatch_entry() above, which builds its own empty
* (nhosts=0) alloc) into a DISPATCH-shaped replay_decision_t. */
static replay_decision_t* make_dispatch_entry_from_alloc(xce_job_alloc_t* alloc)
{
sched_dispatch_decision_t* dispatch
= (sched_dispatch_decision_t*)calloc(1, sizeof(*dispatch));
replay_decision_t* rd = (replay_decision_t*)calloc(1, sizeof(*rd));
dispatch->alloc = alloc;
rd->body_which = ReplayDecision_body_dispatch;
rd->body.dispatch = dispatch;
return rd;
}
/* Task 2.1 fixture helpers (scheduler-sim-eager-reconciliation, Req
* 1.2/1.3): register a real, consumable "slots" resource and add a real
* worker owning a finite amount of it -- same technique
* test_sim_contention_parity.c's own register_slots_resource()/add_worker()
* establish (that file's own header comment documents why the resource must
* be pre-registered: reconciliation's real make_job_account()/
* make_res_account() path always produces residx == -1, lazily resolved by
* sched_update_res_accounts()'s own key_find(core->resmgr, ...) against
* whatever the resource was registered as -- pre-registering here means
* this fixture's own worker resvals/resInUse setup and the reconciled
* job's eventual account both resolve to the SAME index). */
static int register_slots_resource(xce_sched_core_t* core)
{
xce_resource_t* res_template
= (xce_resource_t*)calloc(1, sizeof(*res_template));
xce_resource_t* registered;
TEST_ASSERT_NOT_NULL(res_template);
res_template->name = strdup("slots");
res_template->valtype = XCE_VAL_NUMERIC;
res_template->ordtype = XCE_ORD_DECR;
res_template->flags = XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE
| XCE_RESF_RELEASE | XCE_RESF_DEFAULT_HOST;
registered = makeResource(core->resmgr, res_template);
TEST_ASSERT_NOT_NULL(registered);
return registered->idx;
}
/* Adds a real xce_sched_worker_t owning `capacity` units of `residx`, added
* directly via sched_add_worker() -- sched_add_worker() itself does not
* initialize resvals/resInUse, callers own it (same convention every
* worker-adding fixture in this directory establishes). resInUse starts
* genuinely empty (zarray_init only, nothing put()) -- Requirement 1.2's
* own "still zero" checkpoint below depends on this starting state, not on
* an explicit zero being written. */
static xce_sched_worker_t* add_worker(xce_sched_core_t* core, const char* name,
int residx, float capacity)
{
xce_sched_worker_t* w;
int rc;
float val[1] = { capacity };
w = (xce_sched_worker_t*)calloc(1, sizeof(*w));
TEST_ASSERT_NOT_NULL(w);
w->name = strdup(name);
TEST_ASSERT_NOT_NULL(w->name);
rc = zarray_init(&(w->resvals), 8, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
rc = zarray_init(&(w->resInUse), 8, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
rc = put_resvals(&(w->resvals), 1, &residx, val);
TEST_ASSERT_TRUE(rc >= 0);
rc = sched_add_worker(core, w);
TEST_ASSERT_TRUE(rc >= 0);
return w;
}
/* ---------------------------------------------------------------------- */
/* Test 1 (the task's own primary Observable): a RUN-state job with no */
/* known allocation and no matching recorded decision in this round's */
/* window is left unaccounted (usage unavailable), and the call returns */
/* normally -- no error, nothing fabricated (Requirement 4.1, 4.2). */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_leaves_run_job_unaccounted_when_no_recorded_decision(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
replay_session_t* decoded;
fixture_init(&sim);
/* RUN-state, no known allocation -- e.g. a job dispatched before the
* snapshot this (hypothetical) replay started from, exactly the
* scenario this task's own Observable Completion text describes. */
job = add_job(&(sim.core), 7001ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
TEST_ASSERT_NULL(job->allocs);
/* This round's decoded recorded-decision data has nothing for this
* jobid at all -- the replay's own window never captured it. */
decoded = make_decoded_session(1000ULL, NULL);
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"no matching recorded decision -- job->accounts must stay NULL "
"(usage unavailable), never fabricated");
TEST_ASSERT_NULL_MESSAGE(job->allocs, "job->allocs must likewise stay "
"NULL -- nothing to reconstruct "
"from");
printf("test_reconcile_leaves_run_job_unaccounted_when_no_recorded_"
"decision: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 2: a job left unresolved this round is safe to re-check again on */
/* a later round, with no adverse effect (task's own explicit text). */
/* ---------------------------------------------------------------------- */
static void test_reconcile_is_safe_to_recheck_on_a_later_round(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
replay_session_t* decoded_round1;
replay_session_t* decoded_round2;
fixture_init(&sim);
job = add_job(&(sim.core), 7002ULL, JOB_STAT_RUN, NULL);
decoded_round1 = make_decoded_session(1000ULL, NULL);
sim_reconcile_unaccounted_jobs(&sim, decoded_round1, NULL);
TEST_ASSERT_NULL(job->accounts);
/* Round 2: still no matching recorded decision -- calling again must
* not error, crash, or otherwise behave differently just because this
* is a repeat check of the same job. */
decoded_round2 = make_decoded_session(2000ULL, NULL);
sim_reconcile_unaccounted_jobs(&sim, decoded_round2, NULL);
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"re-checking an unresolved job on a later round must remain safe "
"and leave it unaccounted, with no adverse effect");
printf(
"test_reconcile_is_safe_to_recheck_on_a_later_round: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 3: a job that already has known accounts (either reconciled */
/* earlier or freshly dispatched by the live path) is never re-touched -- */
/* the trigger condition (job->accounts == NULL) must gate correctly even */
/* when a matching recorded DISPATCH entry for the same jobid IS present */
/* this round. Extended (task 2.4) to also prove Requirement 3.3's "report */
/* only the fresh one" guarantee at the decision-reporting layer, not just */
/* at the job->accounts layer: neither the recording channel nor the */
/* step/continue console output may see a duplicate report for this job. */
/* ---------------------------------------------------------------------- */
static void test_reconcile_skips_job_that_already_has_accounts(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_account_t* accounts_before;
replay_session_t* decoded;
sched_record_session_ctx_t* record_ctx;
char tmp[] = "/tmp/xce_simreconcile_dedup_XXXXXX";
int fd = mkstemp(tmp);
int saved_stdout;
off_t len;
char* captured;
fixture_init(&sim);
job = add_job(&(sim.core), 7003ULL, JOB_STAT_RUN,
make_job_alloc(7003ULL, 0, NULL));
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "fixture setup: job must already have accounts");
accounts_before = job->accounts;
decoded = make_decoded_session(1000ULL, make_dispatch_entry(7003ULL));
record_ctx = sched_record_session_ctx_new(1000ULL);
TEST_ASSERT_NOT_NULL(record_ctx);
/* Same dup()/dup2() stdout-capture technique test_sim_verify.c's own
* capture_stdout_around()/run_sim_and_capture_stdout() establish,
* inlined here (this call takes more than the zero args that helper's
* function-pointer shape allows). */
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
"succeed");
fflush(stdout);
saved_stdout = dup(STDOUT_FILENO);
TEST_ASSERT_TRUE(saved_stdout >= 0);
TEST_ASSERT_TRUE(dup2(fd, STDOUT_FILENO) >= 0);
sim_reconcile_unaccounted_jobs(&sim, decoded, record_ctx);
fflush(stdout);
dup2(saved_stdout, STDOUT_FILENO);
close(saved_stdout);
len = lseek(fd, 0, SEEK_CUR);
TEST_ASSERT_TRUE(len >= 0);
captured = (char*)calloc(1, (size_t)len + 1);
TEST_ASSERT_NOT_NULL(captured);
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
if (len > 0) {
ssize_t nread = read(fd, captured, (size_t)len);
TEST_ASSERT_TRUE(nread == len);
}
close(fd);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
accounts_before, job->accounts,
"an already-accounted job must never be re-touched, even when a "
"matching recorded decision exists this round (Requirement 3.3's "
"double-application guard)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, decoded->n_decisions,
"an already-accounted job's matching entry must not be consumed/"
"compacted either -- the trigger never fires for it at all");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, record_ctx->n,
"Requirement 3.3: an already-accounted job (fresh live decision or "
"earlier reconciliation) must produce NO recorded-decision report "
"from this mechanism -- reporting only the fresh one, never a "
"duplicate");
TEST_ASSERT_NULL_MESSAGE(
strstr(captured, "[sim] DISPATCH"),
"Requirement 3.3: an already-accounted job must produce NO "
"step/continue console output from this mechanism either");
free(captured);
printf("test_reconcile_skips_job_that_already_has_accounts: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 4: a recorded PEND entry sharing the same jobid must never be */
/* mistaken for a match -- only a DISPATCH entry can legitimately account */
/* for a RUN-state job (design.md's Implementation Notes). */
/* ---------------------------------------------------------------------- */
static void test_reconcile_ignores_non_dispatch_entry_with_matching_jobid(
void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
replay_session_t* decoded;
fixture_init(&sim);
job = add_job(&(sim.core), 7004ULL, JOB_STAT_RUN, NULL);
decoded = make_decoded_session(1000ULL, make_pend_entry(7004ULL));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"a recorded PEND entry must never be treated as a DISPATCH match, "
"even when its jobid matches -- job stays unaccounted");
printf("test_reconcile_ignores_non_dispatch_entry_with_matching_jobid:"
" PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 5 -- INVERTED (2026-09-21, bug-1838, scheduler-driver-equivalence). */
/* Originally test_reconcile_captures_non_run_state_job_when_dispatch_ */
/* recorded, from scheduler-sim-eager-reconciliation task 1.1: proved the */
/* trigger was state-independent (a PEND job's recorded DISPATCH got */
/* captured just like an already-RUN job's). bug-1838 narrowed the trigger */
/* back to JOB_IS_RUN-only (see sim_reconcile_unaccounted_jobs()'s own */
/* updated header comment in sim_loop.c for the full rationale: */
/* scheduler-sim-live-allocation, completed after this spec, gave sim mode */
/* a genuine capacity-checked live picker/allocator path, making the */
/* PEND-inclusive widening redundant -- a freshly-PEND job now correctly */
/* flows to the SAME live path normal mode uses instead). This test is */
/* renamed and its assertions inverted to lock in the NEW behavior: a */
/* PEND-state job's recorded DISPATCH decision is left UNTOUCHED by */
/* reconciliation, even when a matching entry exists -- dispatching such a */
/* job is now the live picker/allocator's job, not exercised by this */
/* direct-call unit test (see test_sim_live_allocation.c for that proof). */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
int residx;
fixture_init(&sim);
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker1", residx, 1000.0f);
/* PEND-state, no known allocation yet, with a matching recorded
* DISPATCH decision available this round -- under bug-1838's RUN-only
* trigger, this must NOT be captured: reconciliation's remaining
* scope is a job already RUN with no known allocation (structurally
* invisible to the live picker, since next_job() never offers a RUN
* job) -- a freshly-PEND job like this one is a perfectly ordinary
* live-picker candidate instead. */
job = add_job(&(sim.core), 7005ULL, JOB_STAT_PEND, NULL);
TEST_ASSERT_NULL(job->accounts);
TEST_ASSERT_NULL(job->allocs);
jalloc = make_test_job_alloc_with_host(7005ULL, "worker1", "slots", 2.0f);
decoded
= make_decoded_session(1000ULL, make_dispatch_entry_from_alloc(jalloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NULL_MESSAGE(
job->allocs,
"bug-1838: a PEND-state job must NOT be captured by reconciliation "
"even when a matching recorded DISPATCH entry exists -- eligibility "
"is RUN-only again; the live picker/allocator (not exercised by "
"this direct-call unit test) is what decides a PEND job now");
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"job->accounts must stay NULL -- nothing here should fabricate an "
"allocation for a job reconciliation correctly left alone");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, decoded->n_decisions,
"the recorded entry must NOT be consumed -- it was never a "
"candidate match in the first place, so it is left exactly as "
"found for whatever (if anything) still wants to read it");
TEST_ASSERT_EQUAL_UINT_MESSAGE(
JOB_STAT_PEND, job->state,
"job->state must remain untouched (still PEND) -- reconciliation "
"never even attempted this job");
printf("test_reconcile_ignores_pend_state_job_even_when_dispatch_"
"recorded: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 7 (task 2.3): a matching recorded DISPATCH entry is reconstructed */
/* into job->allocs (ownership TAKEOVER -- same pointer, not a copy) and */
/* job->accounts (built from it), and the consumed entry is removed from */
/* decoded_session (Requirement 1.1, 1.2, 3.2). */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
int residx;
fixture_init(&sim);
/* bug-1836 fix: see test_reconcile_captures_non_run_state_job_when_
* dispatch_recorded's own comment above -- the capacity gate needs a
* REAL registered "worker1". */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker1", residx, 1000.0f);
job = add_job(&(sim.core), 7010ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
TEST_ASSERT_NULL(job->allocs);
jalloc = make_test_job_alloc_with_host(7010ULL, "worker1", "slots", 2.0f);
decoded
= make_decoded_session(1000ULL, make_dispatch_entry_from_alloc(jalloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, job->allocs,
"job->allocs must take OWNERSHIP of the recorded entry's alloc "
"(identical pointer), not a duplicate copy -- same pattern "
"scheduler.c's sched_commit_decision() uses");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts,
"job->accounts must be built (make_job_account()) from the "
"reconstructed job->allocs");
TEST_ASSERT_EQUAL_INT(1, job->accounts->nhosts);
TEST_ASSERT_EQUAL_STRING("worker1", job->accounts->accounts[0]->host);
TEST_ASSERT_EQUAL_FLOAT(2.0f,
job->accounts->accounts[0]->accounts[0]->reserved);
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, decoded->n_decisions,
"the now-fully-consumed recorded entry must be removed from "
"decoded_session->decisions");
printf("test_reconcile_reconstructs_allocs_and_accounts_from_matching_"
"dispatch: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 8 (task 2.3): compaction is SHIFT-DOWN, not swap-with-last -- the */
/* remaining recorded entries must keep their ORIGINAL RELATIVE ORDER, per */
/* design.md's Implementation Notes (a later --sim-verify positional walk */
/* depends on this). A 4-entry fixture with the match in the middle */
/* distinguishes the two strategies: swap-with-last would relocate the */
/* last entry (C) into the removed slot, producing [A, C, B] instead of the */
/* correct [A, B, C]. */
/* ---------------------------------------------------------------------- */
static void test_reconcile_compacts_via_shift_down_preserving_order(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
replay_session_t* decoded;
fixture_init(&sim);
job = add_job(&(sim.core), 7011ULL, JOB_STAT_RUN, NULL);
decoded = (replay_session_t*)calloc(1, sizeof(*decoded));
decoded->begin_round = 1000ULL;
decoded->summary = NULL;
decoded->decisions
= (replay_decision_t**)calloc(4, sizeof(*(decoded->decisions)));
decoded->decisions[0] = make_pend_entry(9001ULL); /* A */
decoded->decisions[1] = make_dispatch_entry(7011ULL); /* matched */
decoded->decisions[2] = make_pend_entry(9002ULL); /* B */
decoded->decisions[3] = make_pend_entry(9003ULL); /* C */
decoded->n_decisions = 4;
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NOT_NULL(job->accounts);
TEST_ASSERT_EQUAL_INT_MESSAGE(
3, decoded->n_decisions,
"exactly one entry (the matched DISPATCH) must be consumed");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(9001ULL,
decoded->decisions[0]->body.pend->jobid,
"entry A must stay first");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
9002ULL, decoded->decisions[1]->body.pend->jobid,
"entry B must be second (shift-down) -- swap-with-last would put C "
"here instead");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(9003ULL,
decoded->decisions[2]->body.pend->jobid,
"entry C must be third, untouched");
printf("test_reconcile_compacts_via_shift_down_preserving_order: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 9 (task 2.3, the task's own PRIMARY correctness bar): reconciling a */
/* job's allocation from a recorded DISPATCH entry, then releasing that */
/* job's allocation through its NORMAL end-of-life path -- */
/* XCE_CORE_EVENT_JOB_STATE_CHANGE (core_events.c) -- is clean: no */
/* double-free, no use-after-free. This directly exercises the */
/* move-then-null-then-free ownership handling design.md's Implementation */
/* Notes require: `entry->body.dispatch->alloc` must be nulled out BEFORE */
/* free_ReplayDecision(entry) frees the rest of the entry, or that free */
/* would free the exact memory job->allocs now owns, and this exact */
/* end-of-life release would then double-free it. Meaningful evidence for */
/* this only under the project's ASAN build (see meson.build/tasks.md), */
/* not as a plain functional assertion. */
/* ---------------------------------------------------------------------- */
static void test_reconcile_then_release_via_job_state_change_is_asan_clean(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
replay_session_t* decoded;
xce_core_event_t ev;
int push_rc;
int residx;
fixture_init(&sim);
/* bug-1836 fix: the capacity gate needs a REAL registered "worker1". */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker1", residx, 1000.0f);
job = add_job(&(sim.core), 7012ULL, JOB_STAT_RUN, NULL);
decoded = make_decoded_session(
1000ULL, make_dispatch_entry_from_alloc(make_test_job_alloc_with_host(
7012ULL, "worker1", "slots", 4.0f)));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NOT_NULL(job->accounts);
TEST_ASSERT_NOT_NULL(job->allocs);
TEST_ASSERT_EQUAL_INT(0, decoded->n_decisions);
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 7012ULL;
ev.u.job_state_change.old_state = JOB_STAT_RUN;
ev.u.job_state_change.new_state = JOB_STAT_DONE;
push_rc = xce_core_push_events(&(sim.core), &ev, 1);
TEST_ASSERT_EQUAL_INT(0, push_rc);
TEST_ASSERT_NULL_MESSAGE(
job->accounts, "released cleanly via the normal end-of-life path");
TEST_ASSERT_NULL_MESSAGE(
job->allocs, "released cleanly via the normal end-of-life path");
/* Also exercise decoded_session's own round-end cleanup path
* (free_ReplaySession_ptr(), the same call sched_sim_driver_run() makes
* every round) now that n_decisions correctly reflects only the
* remaining, still-owned entries (here: none) -- ASAN-clean under a
* leak-detecting run (this suite's registered meson test uses
* detect_leaks=0 per its own established fixture-teardown convention;
* this call is exercised here regardless, and was additionally verified
* with ASAN_OPTIONS=detect_leaks=1 against the standalone binary as
* part of this task's own validation). */
free_ReplaySession_ptr(&decoded);
printf("test_reconcile_then_release_via_job_state_change_is_asan_clean: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 10 (task 2.4, the task's own PRIMARY Observable): a matching */
/* recorded DISPATCH entry, once reconstructed (task 2.3), is fed through */
/* the exact same three-call decision-reporting channel a freshly-drained */
/* decision uses -- sched_record_session_add_decision() (proven via a real */
/* record_ctx's own accumulated decisions[]/n/n_dispatched fields, exactly */
/* the same struct sched_sim_driver_run()'s drain loop populates) and */
/* sim_print_decision() (proven via captured stdout -- the literal */
/* mechanism by which a decision "shows... in step/continue output", per */
/* this task's own Observable Completion text). xce_repl_check_decision() */
/* is also called (not independently observable here without a live REPL */
/* breakpoint fixture -- repl.c/repl_break.c/repl_info.c are unchanged and */
/* out of this task's boundary per design.md) but must not crash, which */
/* this test's mere completion already proves for both sim->interactive */
/* on and off. Preserves the original decision's type (DISPATCH) and */
/* resource-allocation detail (Requirement 3.2): the SAME job->allocs */
/* pointer the reconciliation step already reconstructed is what reaches */
/* both the recording channel and the console output. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_reports_reconstructed_decision_through_existing_channel(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
sched_record_session_ctx_t* record_ctx;
char tmp[] = "/tmp/xce_simreconcile_report_XXXXXX";
int fd = mkstemp(tmp);
int saved_stdout;
off_t len;
char* captured;
int residx;
fixture_init(&sim);
/* bug-1836 fix: the capacity gate needs a REAL registered "worker9". */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker9", residx, 1000.0f);
job = add_job(&(sim.core), 7020ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
jalloc = make_test_job_alloc_with_host(7020ULL, "worker9", "slots", 3.0f);
decoded
= make_decoded_session(1000ULL, make_dispatch_entry_from_alloc(jalloc));
record_ctx = sched_record_session_ctx_new(1000ULL);
TEST_ASSERT_NOT_NULL(record_ctx);
/* Exercise the REPL hook too (sim->interactive on) -- must remain a
* safe no-op with no repl_state allocated yet (repl.h's own "never
* load-bearing" precondition), proving xce_repl_check_decision() is
* actually reached and does not crash regardless of interactive mode. */
sim.interactive = true;
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
"succeed");
fflush(stdout);
saved_stdout = dup(STDOUT_FILENO);
TEST_ASSERT_TRUE(saved_stdout >= 0);
TEST_ASSERT_TRUE(dup2(fd, STDOUT_FILENO) >= 0);
sim_reconcile_unaccounted_jobs(&sim, decoded, record_ctx);
fflush(stdout);
dup2(saved_stdout, STDOUT_FILENO);
close(saved_stdout);
len = lseek(fd, 0, SEEK_CUR);
TEST_ASSERT_TRUE(len >= 0);
captured = (char*)calloc(1, (size_t)len + 1);
TEST_ASSERT_NOT_NULL(captured);
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
if (len > 0) {
ssize_t nread = read(fd, captured, (size_t)len);
TEST_ASSERT_TRUE(nread == len);
}
close(fd);
/* The reconciliation itself (task 2.3) still happened. */
TEST_ASSERT_EQUAL_PTR(jalloc, job->allocs);
TEST_ASSERT_NOT_NULL(job->accounts);
/* sched_record_session_add_decision() reporting (Requirement 3.1). */
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, record_ctx->n,
"the reconstructed DISPATCH decision must be reported through the "
"recording channel exactly once");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, record_ctx->n_dispatched,
"must be counted as a DISPATCH");
TEST_ASSERT_EQUAL_INT_MESSAGE(
ReplayDecision_body_dispatch, record_ctx->decisions[0]->body_which,
"must be recorded with the DISPATCH body shape");
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, record_ctx->decisions[0]->body.dispatch->alloc,
"Requirement 3.2: the recorded entry must carry the SAME "
"resource-allocation detail (identical pointer, borrowed -- same "
"convention recording_session.c's own header comment documents "
"for a freshly-drained decision) the reconciliation step "
"reconstructed, not a copy or a different value");
/* sim_print_decision() console output (Requirement 3.1 -- this IS the
* "shows... in step/continue output" mechanism). */
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, "[sim] DISPATCH jobid="),
"the reconstructed decision must be printed through the same "
"step/continue console output a freshly-computed decision uses");
free(captured);
printf("test_reconcile_reports_reconstructed_decision_through_existing_"
"channel: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 6: NULL `sim`/`decoded_session` is a safe no-op (design.md's */
/* Service Interface preconditions -- "never load-bearing" convention). */
/* ---------------------------------------------------------------------- */
static void test_reconcile_null_args_are_a_safe_noop(void)
{
xce_sched_sim_driver_t sim;
replay_session_t* decoded;
fixture_init(&sim);
decoded = make_decoded_session(1000ULL, NULL);
sim_reconcile_unaccounted_jobs(NULL, decoded, NULL);
sim_reconcile_unaccounted_jobs(&sim, NULL, NULL);
sim_reconcile_unaccounted_jobs(NULL, NULL, NULL);
/* Reaching this line without a crash IS the assertion. */
TEST_ASSERT_TRUE(1);
printf("test_reconcile_null_args_are_a_safe_noop: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 12 (scheduler-sim-eager-reconciliation task 4.1/4.2, Req 1.2, 1.3, */
/* 1.4, REVISED 2026-09-11, user-directed): a captured job's resource */
/* consumption must be reflected in resInUse starting the SAME round its */
/* dispatch decision is captured, not deferred to a later round's */
/* separately-recorded RUN-state confirmation event. sim_reconcile_ */
/* unaccounted_jobs() now sets job->state = JOB_STAT_RUN synchronously, */
/* in the same operation as the allocation capture -- mirroring */
/* scheduler.c's sched_commit_decision() (bug-1624) exactly. This */
/* INVERTS the pre-2026-09-11 version of this test (formerly */
/* test_captured_pend_job_excluded_from_resinuse_until_run), which */
/* asserted the OPPOSITE: that a captured-but-PEND job stayed PEND and */
/* contributed zero to resInUse until a SEPARATE, later job->state = RUN */
/* write. See requirements.md's Requirement 1 2026-09-11 revision and */
/* research.md's corresponding entry for the full rationale trail. */
/* */
/* REVISED AGAIN (2026-09-21, bug-1838): the job below now starts as */
/* JOB_STAT_RUN, not PEND -- bug-1838 narrowed reconciliation's trigger */
/* back to JOB_IS_RUN-only (a PEND job is no longer a candidate at all, */
/* regardless of a matching recorded decision -- see sim_loop.c's own */
/* updated header comment on sim_reconcile_unaccounted_jobs()). This */
/* test's own point -- a captured job's resInUse charge lands the SAME */
/* round as capture, not deferred -- is orthogonal to PEND-vs-RUN and */
/* still fully exercised by a RUN-starting job (the remaining genuine */
/* reconciliation case: already RUN, allocation still unknown). */
/* ---------------------------------------------------------------------- */
static void
test_captured_job_included_in_resinuse_same_round_as_capture(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_sched_worker_t* w;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
xce_core_event_t ev;
int residx;
float got;
int rc;
int push_rc;
fixture_init(&sim);
residx = register_slots_resource(&(sim.core));
w = add_worker(&(sim.core), "worker1", residx, 10.0f);
/* bug-1838: RUN-state job, no known allocation yet -- the remaining
* genuine reconciliation case (already RUN before this replay's own
* observation window, allocation never captured). Its recorded
* DISPATCH decision is about to be found this round. */
job = add_job(&(sim.core), 7030ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
TEST_ASSERT_NULL(job->allocs);
jalloc = make_test_job_alloc_with_host(7030ULL, "worker1", "slots", 4.0f);
decoded = make_decoded_session(1000ULL,
make_dispatch_entry_from_alloc(jalloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, job->allocs,
"the job's recorded DISPATCH decision must be captured before this "
"test's own checkpoints mean anything");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts,
"job->accounts must be built from the captured allocation");
TEST_ASSERT_TRUE_MESSAGE(
JOB_STAT_RUN == job->state,
"Requirement 1.2 (2026-09-11 revision): capturing a job's dispatch "
"decision must ALSO transition it to RUN synchronously, mirroring "
"scheduler.c's sched_commit_decision() (bug-1624) -- no separate "
"job->state write should be needed here");
/* Checkpoint 1 (Requirement 1.3, 2026-09-11 revision): recompute
* resInUse via the REAL sched_account_jobs() -- the SAME round as
* capture must ALREADY reflect the charge, not a later one. */
rc = sched_account_jobs(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
got = -1.0f;
rc = get_resvals(&(w->resInUse), 1, &residx, &got);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
4.0f, got,
"Requirement 1.3 (2026-09-11 revision): a captured job's resInUse "
"charge must appear the SAME round its dispatch decision is "
"captured -- sched_account_jobs()'s unchanged JOB_IS_RUN gate now "
"sees the job as RUN immediately, since capture itself sets "
"job->state = JOB_STAT_RUN synchronously");
/* Checkpoint 2 (Requirement 1.4, new 2026-09-11): a LATER, separately-
* recorded JOB_STATE_CHANGE-to-RUN event for the SAME job (the
* historical worker-confirmation signal, still present in a real
* recording and still replayed) must be a safe, idempotent
* re-affirmation -- no double-charge, no crash, allocs/accounts
* untouched (since !JOB_IS_RUN(new_state) is false for RUN, the
* handler's cleanup branch is skipped). Drives the REAL
* XCE_CORE_EVENT_JOB_STATE_CHANGE handler (core_events.c), same
* technique test_reconcile_then_release_via_job_state_change_is_asan_
* clean() above already establishes. */
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 7030ULL;
ev.u.job_state_change.old_state = JOB_STAT_RUN;
ev.u.job_state_change.new_state = JOB_STAT_RUN;
push_rc = xce_core_push_events(&(sim.core), &ev, 1);
TEST_ASSERT_EQUAL_INT(0, push_rc);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, job->allocs,
"Requirement 1.4: a redundant RUN->RUN confirmation event must not "
"touch the already-captured allocation");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts,
"Requirement 1.4: a redundant RUN->RUN confirmation event must not "
"free the already-captured accounts");
/* core_reset_workers() (core_session.c, white-box #include'd above)
* mirrors the REAL per-round sequence sched_sim_driver_run() always
* uses (xce_core_begin_session() zeroes resInUse -> sched_account_jobs()
* re-sums it from scratch) -- production NEVER calls sched_account_jobs()
* a second time without this reset in between, so this call is what
* makes the following recompute a faithful round-4-boundary proof
* instead of an artificial double-charge test-methodology artifact. */
core_reset_workers(&(sim.core));
rc = sched_account_jobs(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
got = -1.0f;
rc = get_resvals(&(w->resInUse), 1, &residx, &got);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
4.0f, got,
"Requirement 1.4: after a proper reset-then-resum round boundary, "
"the redundant RUN->RUN confirmation event must NOT have changed "
"what gets charged -- still exactly 4.0, the same single "
"allocation, not fabricated or duplicated");
printf(
"test_captured_job_included_in_resinuse_same_round_as_capture: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
/* bug-1836 regression (scheduler-driver-equivalence): a resource- */
/* contention fixture per .kiro/steering/scheduler-driver-equivalence.md's */
/* own testing discipline ("multiple jobs competing for capacity */
/* insufficient for all of them"). Live symptom this closes: a real */
/* 22-slot host took 44 concurrent single-slot jobs during replay, because */
/* sim_reconcile_unaccounted_jobs() force-applied every recorded DISPATCH */
/* decision unconditionally, never checking whether it still fit CURRENT */
/* capacity. Two RUN jobs (bug-1838: reconciliation's trigger is RUN-only */
/* again -- these model jobs already confirmed RUN via a replayed */
/* JOB_STATE_CHANGE event, awaiting their allocation data), each with a */
/* recorded DISPATCH decision wanting the SAME host's only unit of */
/* "slots" (capacity 1) -- normal mode would dispatch exactly one and PEND */
/* the other; this proves sim mode's reconciliation backstop now does the */
/* capacity check instead of force-applying both unconditionally. */
/* ---------------------------------------------------------------------- */
static void test_reconcile_respects_host_capacity_under_contention(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* jobA;
xce_sched_job_t* jobB;
xce_sched_worker_t* w;
xce_job_alloc_t* allocA;
xce_job_alloc_t* allocB;
replay_session_t* decoded;
int residx;
float got;
int rc;
int n_admitted;
int n_unaccounted;
fixture_init(&sim);
residx = register_slots_resource(&(sim.core));
w = add_worker(&(sim.core), "worker1", residx, 1.0f);
jobA = add_job(&(sim.core), 7040ULL, JOB_STAT_RUN, NULL);
jobB = add_job(&(sim.core), 7041ULL, JOB_STAT_RUN, NULL);
allocA = make_test_job_alloc_with_host(7040ULL, "worker1", "slots", 1.0f);
allocB = make_test_job_alloc_with_host(7041ULL, "worker1", "slots", 1.0f);
decoded = (replay_session_t*)calloc(1, sizeof(*decoded));
decoded->begin_round = 1000ULL;
decoded->summary = NULL;
decoded->decisions
= (replay_decision_t**)calloc(2, sizeof(*(decoded->decisions)));
decoded->decisions[0] = make_dispatch_entry_from_alloc(allocA);
decoded->decisions[1] = make_dispatch_entry_from_alloc(allocB);
decoded->n_decisions = 2;
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
n_admitted = ((jobA->accounts != NULL) ? 1 : 0)
+ ((jobB->accounts != NULL) ? 1 : 0);
n_unaccounted = ((jobA->accounts == NULL) ? 1 : 0)
+ ((jobB->accounts == NULL) ? 1 : 0);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, n_admitted,
"worker1 has capacity for exactly ONE of these two 1-slot jobs -- "
"reconciliation must admit exactly one, not both (the bug-1836 "
"over-commit) and not zero");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, n_unaccounted,
"the job that does not fit must be left unaccounted (accounts == "
"NULL) to retry on a later round, not force-dispatched");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, decoded->n_decisions,
"the entry for the job that did not fit must NOT be consumed -- it "
"stays available to retry a later round");
/* Whichever job was admitted must genuinely be RUN with accounts
* populated; the other must ALSO still be RUN (bug-1838: it entered
* this call already RUN, and a failed capacity gate never reverts
* state -- it is simply left RUN-but-unaccounted, exactly like a job
* whose confirming JOB_STATE_CHANGE arrived before its DISPATCH could
* be reconciled) -- distinguished from the admitted job only by
* accounts/allocs, never by job->state. */
if (jobA->accounts != NULL) {
TEST_ASSERT_TRUE(JOB_STAT_RUN == jobA->state);
TEST_ASSERT_TRUE(JOB_STAT_RUN == jobB->state);
} else {
TEST_ASSERT_TRUE(JOB_STAT_RUN == jobB->state);
TEST_ASSERT_TRUE(JOB_STAT_RUN == jobA->state);
}
/* The core correctness bar: resInUse (via the real, unchanged
* sched_account_jobs() call this function's own caller always makes
* immediately afterward in production) must reflect EXACTLY the one
* admitted job's charge -- 1.0, never 2.0 (the bug-1836 over-commit) --
* proving this function's internal peek+incremental-charge+undo left no
* residue AND that the one real charge landed correctly. */
rc = sched_account_jobs(&(sim.core));
TEST_ASSERT_EQUAL_INT(0, rc);
got = -1.0f;
rc = get_resvals(&(w->resInUse), 1, &residx, &got);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
1.0f, got,
"bug-1836: resInUse must reflect exactly the one admitted job's "
"charge (1.0), never both (2.0, the over-commit this fix closes) "
"and never neither (0.0, which would mean the fitting job was "
"wrongly rejected too)");
printf("test_reconcile_respects_host_capacity_under_contention: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 13 (scheduler-sim-eager-reconciliation task 1.2, Requirement 3.2): */
/* a job with job->accounts ALREADY set is never re-searched or */
/* re-captured on a later round's call, deliberately fixtured PEND here */
/* (different from test_reconcile_skips_job_that_already_has_accounts' */
/* RUN-state fixture above) -- even when a decoded session that round */
/* happens to also contain a matching-jobid DISPATCH entry. Proves the */
/* job->accounts != NULL guard alone is enough to block re-capture, */
/* independent of whatever the RUN-only trigger condition (bug-1838) */
/* would otherwise say about this job's state. */
/* */
/* UPDATED (2026-09-21, bug-1838): this test predates the RUN-only */
/* narrowing and originally demonstrated the accounts-guard held even */
/* under the (now-superseded) state-independent trigger. Since a PEND job */
/* is skipped by the state check alone today, this fixture no longer */
/* isolates the accounts guard specifically -- it still correctly proves */
/* "PEND + already captured" is never re-captured, just via a condition */
/* that is now doubly-guarded (state AND accounts) rather than singly. */
/* Kept as-is (not deleted) since it remains a true, valid regression */
/* lock; see test_reconcile_skips_job_that_already_has_accounts() for the */
/* RUN-state fixture that DOES still isolate the accounts guard alone. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_still_skips_already_captured_job_regardless_of_state(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_account_t* accounts_before;
xce_job_alloc_t* allocs_before;
replay_session_t* decoded_round1;
replay_session_t* decoded_round2;
fixture_init(&sim);
/* PEND-state, already captured (mirrors add_job()'s own "already
* accounted" fixture shape, same technique
* test_reconcile_skips_job_that_already_has_accounts() uses -- here
* deliberately with a non-RUN state). */
job = add_job(&(sim.core), 7040ULL, JOB_STAT_PEND,
make_job_alloc(7040ULL, 0, NULL));
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "fixture setup: job must already have accounts");
accounts_before = job->accounts;
allocs_before = job->allocs;
/* Round 1: nothing to do with this job either way -- establishes the
* "later round" framing the next call demonstrates. */
decoded_round1 = make_decoded_session(4000ULL, NULL);
sim_reconcile_unaccounted_jobs(&sim, decoded_round1, NULL);
TEST_ASSERT_EQUAL_PTR(accounts_before, job->accounts);
/* Round 2 ("a later round"): a recorded DISPATCH entry for THIS job's
* own jobid is present. If the job->accounts != NULL guard were ever
* bypassed under the widened, state-independent trigger, this would be
* re-captured. It must not be. */
decoded_round2
= make_decoded_session(5000ULL, make_dispatch_entry(7040ULL));
sim_reconcile_unaccounted_jobs(&sim, decoded_round2, NULL);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
accounts_before, job->accounts,
"an already-captured PEND-state job must never be re-captured on a "
"later round, even with a matching-jobid recorded DISPATCH entry "
"present -- job->accounts != NULL alone is sufficient to block "
"re-capture, regardless of what the RUN-only state check would "
"separately say about this job");
TEST_ASSERT_EQUAL_PTR_MESSAGE(
allocs_before, job->allocs,
"job->allocs likewise must stay exactly what it already was -- "
"never swapped for the later round's entry");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, decoded_round2->n_decisions,
"the matching entry must not be consumed either -- the trigger "
"never fires for an already-captured job at all");
printf("test_reconcile_still_skips_already_captured_job_regardless_of_"
"state: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 14 (scheduler-sim-eager-reconciliation task 1.2, Requirement 4.4). */
/* REVISED (2026-09-21, bug-1838): originally a JOB_STAT_PEND job (the */
/* "in-flight, no match yet" case task 1.1's now-superseded state- */
/* independent widening existed for). bug-1838 narrowed reconciliation */
/* back to JOB_IS_RUN-only, so a PEND job is never a candidate regardless */
/* of whether a match ever appears -- that scenario is now covered by */
/* test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded */
/* instead. This test's job is changed to JOB_STAT_RUN so its OWN real */
/* point survives: a RUN job with no matching recorded decision stays */
/* uncaptured (job->accounts stays NULL) after one call, remains fully */
/* eligible, and succeeds once a matching entry appears on a later round -- */
/* still the remaining, genuine reconciliation case (already RUN, */
/* allocation unknown, recorded decision not yet found). Mirrors */
/* test_reconcile_is_safe_to_recheck_on_a_later_round()'s "safe to retry" */
/* idiom above, but (unlike that sibling test, whose second round also has */
/* no match) proves the retry can actually succeed on a later one. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded_round1;
replay_session_t* decoded_round2;
int residx;
fixture_init(&sim);
/* bug-1836 fix: the capacity gate needs a REAL registered "workerB". */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "workerB", residx, 1000.0f);
/* bug-1838: RUN-state, no known allocation, no recorded decision
* available yet -- the realistic in-flight case reconciliation's
* RUN-only trigger still genuinely needs to cover. */
job = add_job(&(sim.core), 7041ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
TEST_ASSERT_NULL(job->allocs);
decoded_round1 = make_decoded_session(4100ULL, NULL);
sim_reconcile_unaccounted_jobs(&sim, decoded_round1, NULL);
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"no matching recorded decision this round -- a RUN job with an "
"unknown allocation must stay uncaptured (usage unavailable), "
"never fabricated");
TEST_ASSERT_NULL_MESSAGE(job->allocs, "job->allocs likewise stays NULL");
/* Round 2: the job's recorded DISPATCH decision now shows up. The job
* remains eligible (was never given up on) -- capture succeeds this
* time, proving the previous failed attempt left no "give up" state
* behind. */
jalloc = make_test_job_alloc_with_host(7041ULL, "workerB", "slots", 1.0f);
decoded_round2 = make_decoded_session(
4200ULL, make_dispatch_entry_from_alloc(jalloc));
sim_reconcile_unaccounted_jobs(&sim, decoded_round2, NULL);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc, job->allocs,
"a job that stayed uncaptured with no match must remain fully "
"eligible on a later round -- capture succeeds as soon as a "
"matching recorded decision appears");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "job->accounts must be built once the retry finds a "
"match");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, decoded_round2->n_decisions,
"the now-matched entry must be consumed exactly like any other "
"capture");
printf("test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 15 (scheduler-sim-eager-reconciliation task 1.2, Requirement 1.1, */
/* 3.1 under concurrent load): several simultaneously outstanding jobs, */
/* each with a DIFFERENT jobid, spread across three scenarios a single */
/* reconcile call must resolve with zero cross-job interference: */
/* - job_matched: RUN, its own DISPATCH entry IS this round. */
/* - job_no_match: RUN, no recorded entry at all this round. */
/* - job_already: captured on a hypothetical EARLIER round already */
/* (job->accounts/job->allocs preset directly via add_job()'s own */
/* "already accounted" shape, same technique */
/* test_reconcile_skips_job_that_already_has_accounts() and Test 13 */
/* above use) -- AND, to make the "no accidental overwrite" claim */
/* non-trivial, a matching-jobid DISPATCH entry for it is ALSO */
/* present this round. */
/* Asserts each job's own job->accounts reflects ONLY that job's own */
/* recorded decision, with no cross-job data or overwrite (the single-job */
/* tests above cannot distinguish "correct" from "correct by coincidence, */
/* only one candidate available", per design.md's own Testing Strategy). */
/* */
/* bug-1838: job_matched and job_no_match are both RUN (not PEND) here -- */
/* reconciliation's trigger is RUN-only again, so a PEND job is now always */
/* skipped regardless of whether a matching entry exists (that claim is */
/* covered on its own by */
/* test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded). */
/* Keeping job_no_match RUN here preserves this test's original, distinct */
/* claim: among multiple RUN candidates, reconciliation matches strictly */
/* by jobid rather than capturing the first/any available entry. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_multiple_jobs_captured_independently_across_different_rounds(
void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job_matched;
xce_sched_job_t* job_no_match;
xce_sched_job_t* job_already;
xce_job_account_t* already_accounts_before;
xce_job_alloc_t* already_allocs_before;
xce_job_alloc_t* jalloc_matched;
xce_job_alloc_t* jalloc_already_new;
replay_session_t* decoded;
int residx;
fixture_init(&sim);
/* bug-1836 fix: the capacity gate needs a REAL registered "workerA" --
* "worker-should-not-apply" (job_already's entry) is deliberately never
* registered: that entry is never even searched for (job_already's
* accounts != NULL guard skips it), so it must never reach the
* capacity gate at all -- if it does, this test's own assertions below
* already catch that regression independently. */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "workerA", residx, 1000.0f);
job_matched = add_job(&(sim.core), 7050ULL, JOB_STAT_RUN, NULL);
job_no_match = add_job(&(sim.core), 7051ULL, JOB_STAT_RUN, NULL);
job_already = add_job(&(sim.core), 7052ULL, JOB_STAT_RUN,
make_job_alloc(7052ULL, 0, NULL));
TEST_ASSERT_NOT_NULL_MESSAGE(
job_already->accounts,
"fixture setup: job_already must already have accounts, "
"simulating capture on an earlier round");
already_accounts_before = job_already->accounts;
already_allocs_before = job_already->allocs;
jalloc_matched
= make_test_job_alloc_with_host(7050ULL, "workerA", "slots", 5.0f);
jalloc_already_new = make_test_job_alloc_with_host(
7052ULL, "worker-should-not-apply", "slots", 99.0f);
decoded = (replay_session_t*)calloc(1, sizeof(*decoded));
decoded->begin_round = 6000ULL;
decoded->summary = NULL;
decoded->decisions
= (replay_decision_t**)calloc(2, sizeof(*(decoded->decisions)));
decoded->decisions[0] = make_dispatch_entry_from_alloc(jalloc_matched);
decoded->decisions[1]
= make_dispatch_entry_from_alloc(jalloc_already_new);
decoded->n_decisions = 2;
/* job_no_match (7051) intentionally has no entry in this round's
* decoded stream at all. */
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
/* job_matched: captured, and reflects EXACTLY its own jobid's data. */
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc_matched, job_matched->allocs,
"job_matched must capture its own recorded DISPATCH entry");
TEST_ASSERT_NOT_NULL(job_matched->accounts);
TEST_ASSERT_EQUAL_STRING_MESSAGE(
"workerA", job_matched->accounts->accounts[0]->host,
"job_matched's captured account must reflect its OWN entry's "
"host, not another job's");
TEST_ASSERT_EQUAL_FLOAT_MESSAGE(
5.0f, job_matched->accounts->accounts[0]->accounts[0]->reserved,
"job_matched's captured account must reflect its OWN entry's "
"reserved value, not another job's");
/* job_no_match: stays uncaptured -- no entry existed for it. */
TEST_ASSERT_NULL_MESSAGE(
job_no_match->accounts,
"job_no_match has no matching recorded entry this round -- must "
"stay uncaptured");
TEST_ASSERT_NULL(job_no_match->allocs);
/* job_already: completely unchanged, despite a matching-jobid DISPATCH
* entry for it being present this round -- proving no accidental
* overwrite/cross-contamination from the wider per-round scan. */
TEST_ASSERT_EQUAL_PTR_MESSAGE(
already_accounts_before, job_already->accounts,
"job_already's accounts must be the IDENTICAL pointer from before "
"this call -- never overwritten, even with a matching entry "
"present this round");
TEST_ASSERT_EQUAL_PTR_MESSAGE(
already_allocs_before, job_already->allocs,
"job_already's allocs must likewise be the IDENTICAL pointer from "
"before this call");
/* job_already's own matching entry was never even searched for (its
* job->accounts != NULL guard skips it before the search begins), so
* it must remain fully intact and unconsumed in decoded_session -- only
* job_matched's entry was consumed. */
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, decoded->n_decisions,
"exactly one entry (job_matched's) must be consumed -- "
"job_already's entry is left untouched since that job was never "
"eligible to be searched for at all");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
7052ULL, decoded->decisions[0]->body.dispatch->alloc->jobid,
"the remaining entry must be job_already's own, untouched, entry");
TEST_ASSERT_EQUAL_PTR_MESSAGE(
jalloc_already_new, decoded->decisions[0]->body.dispatch->alloc,
"job_already's entry's alloc must be the SAME pointer it started "
"with -- never taken over, since that job was already captured");
printf("test_reconcile_multiple_jobs_captured_independently_across_"
"different_rounds: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 16 (scheduler-sim-repl-verbosity task 4.2 remediation, post-review: */
/* review found this file's own second production call site to */
/* sim_print_decision() -- inside sim_reconcile_unaccounted_jobs() itself */
/* -- was never gated by sim_print_decision_gate() (sim_loop.c), the ONE */
/* other call site task 4.2 originally wired. Since this reconciliation */
/* path is a PRIMARY source of DISPATCH decisions surfacing in replay mode */
/* (most live-drain-loop activity is PEND), a round where it fires used to */
/* show BOTH this bare "[sim] DISPATCH jobid=..." line AND the Round */
/* Rollup's summary counting the same dispatch -- the exact bug-1610 */
/* duplication this whole task exists to eliminate, just via this second */
/* call site. This test proves the fix: with sim->interactive == true AND */
/* a properly-allocated sim->repl_state in XCE_REPL_MODE_CONTINUE, the bare */
/* line from THIS mechanism must no longer appear -- the Round Rollup's own */
/* summary becomes the sole source of decision output here too, matching */
/* design.md's Requirement 2.3 ("the round summary is the sole source of */
/* decision output") for real this time. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_continue_mode_with_repl_state_suppresses_bare_decision_line(
void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
sched_record_session_ctx_t* record_ctx;
char tmp[] = "/tmp/xce_simreconcile_gate_continue_XXXXXX";
int fd = mkstemp(tmp);
int saved_stdout;
off_t len;
char* captured;
int residx;
fixture_init(&sim);
/* bug-1836 fix: the capacity gate needs a REAL registered "worker9". */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker9", residx, 1000.0f);
job = add_job(&(sim.core), 7060ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
jalloc = make_test_job_alloc_with_host(7060ULL, "worker9", "slots", 3.0f);
decoded
= make_decoded_session(1000ULL, make_dispatch_entry_from_alloc(jalloc));
record_ctx = sched_record_session_ctx_new(1000ULL);
TEST_ASSERT_NOT_NULL(record_ctx);
/* The gate-relevant fixture shape (unlike test_reconcile_reports_
* reconstructed_decision_through_existing_channel above, which
* deliberately leaves repl_state NULL to prove the OTHER half of the
* gate's NULL-safety contract): a genuinely allocated repl_state in
* XCE_REPL_MODE_CONTINUE, the same repl_state_ensure() technique
* test_repl_round_rollup.c's own fixtures use for a direct-call (no
* real REPL command loop) test. */
sim.interactive = true;
sim.repl_state = repl_state_ensure(&sim);
TEST_ASSERT_NOT_NULL(sim.repl_state);
sim.repl_state->mode = XCE_REPL_MODE_CONTINUE;
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
"succeed");
fflush(stdout);
saved_stdout = dup(STDOUT_FILENO);
TEST_ASSERT_TRUE(saved_stdout >= 0);
TEST_ASSERT_TRUE(dup2(fd, STDOUT_FILENO) >= 0);
sim_reconcile_unaccounted_jobs(&sim, decoded, record_ctx);
fflush(stdout);
dup2(saved_stdout, STDOUT_FILENO);
close(saved_stdout);
len = lseek(fd, 0, SEEK_CUR);
TEST_ASSERT_TRUE(len >= 0);
captured = (char*)calloc(1, (size_t)len + 1);
TEST_ASSERT_NOT_NULL(captured);
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
if (len > 0) {
ssize_t nread = read(fd, captured, (size_t)len);
TEST_ASSERT_TRUE(nread == len);
}
close(fd);
/* The reconciliation itself (task 2.3) still happened -- this gate only
* ever affects the console print, nothing else in the reporting chain. */
TEST_ASSERT_EQUAL_PTR(jalloc, job->allocs);
TEST_ASSERT_NOT_NULL(job->accounts);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, record_ctx->n,
"the recording channel (sched_record_session_add_decision()) must "
"still report the reconstructed decision -- only the console print "
"is gated, not the recording");
/* THE ASSERTION THIS TEST EXISTS TO PROVE: the bare per-decision line
* must NOT appear via this call site while sim->interactive &&
* sim->repl_state->mode == XCE_REPL_MODE_CONTINUE (bug-1610, this
* remediation's own scope). */
TEST_ASSERT_NULL_MESSAGE(
strstr(captured, "[sim] DISPATCH jobid="),
"under continue mode with a genuinely allocated repl_state, "
"sim_reconcile_unaccounted_jobs()'s own reconstructed-decision "
"report must NOT print the bare per-decision line -- the Round "
"Rollup's own summary is the sole source of decision output here "
"too, closing bug-1610's duplication gap at this second call site");
free(captured);
free(sim.repl_state);
printf(
"test_reconcile_continue_mode_with_repl_state_suppresses_bare_"
"decision_line: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 17 (regression-safety half of Test 16): under `step` mode with a */
/* genuinely allocated repl_state, the bare per-decision line from THIS */
/* same call site must still appear exactly as before -- only `continue` */
/* mode is gated (Requirement 5.1's "step mode ... unaffected" guarantee, */
/* now proven for this second call site too, not just the drain loop's). */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_step_mode_with_repl_state_still_prints_bare_decision_line(
void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
char tmp[] = "/tmp/xce_simreconcile_gate_step_XXXXXX";
int fd = mkstemp(tmp);
int saved_stdout;
off_t len;
char* captured;
int residx;
fixture_init(&sim);
/* bug-1836 fix: the capacity gate needs a REAL registered "worker9". */
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker9", residx, 1000.0f);
job = add_job(&(sim.core), 7061ULL, JOB_STAT_RUN, NULL);
jalloc = make_test_job_alloc_with_host(7061ULL, "worker9", "slots", 3.0f);
decoded
= make_decoded_session(1000ULL, make_dispatch_entry_from_alloc(jalloc));
sim.interactive = true;
sim.repl_state = repl_state_ensure(&sim);
TEST_ASSERT_NOT_NULL(sim.repl_state);
/* XCE_REPL_MODE_STEP == 0, already repl_state_ensure()'s calloc()
* default -- set explicitly regardless, so this test's intent reads
* clearly and does not silently depend on the enum's numeric value. */
sim.repl_state->mode = XCE_REPL_MODE_STEP;
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
"succeed");
fflush(stdout);
saved_stdout = dup(STDOUT_FILENO);
TEST_ASSERT_TRUE(saved_stdout >= 0);
TEST_ASSERT_TRUE(dup2(fd, STDOUT_FILENO) >= 0);
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
fflush(stdout);
dup2(saved_stdout, STDOUT_FILENO);
close(saved_stdout);
len = lseek(fd, 0, SEEK_CUR);
TEST_ASSERT_TRUE(len >= 0);
captured = (char*)calloc(1, (size_t)len + 1);
TEST_ASSERT_NOT_NULL(captured);
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
if (len > 0) {
ssize_t nread = read(fd, captured, (size_t)len);
TEST_ASSERT_TRUE(nread == len);
}
close(fd);
TEST_ASSERT_EQUAL_PTR(jalloc, job->allocs);
TEST_ASSERT_NOT_NULL(job->accounts);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, "[sim] DISPATCH jobid="),
"while single-stepping (repl_state allocated, mode == "
"XCE_REPL_MODE_STEP), sim_reconcile_unaccounted_jobs()'s own "
"reconstructed-decision report must still print the bare "
"per-decision line exactly as before -- only continue mode is "
"gated");
free(captured);
free(sim.repl_state);
printf(
"test_reconcile_step_mode_with_repl_state_still_prints_bare_"
"decision_line: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 18 (scheduler-sim-reconcile-terminal-guard task 1.1, RED phase, */
/* Requirement 1.1, 1.3; bug-1764): a job driven all the way to */
/* JOB_STAT_DONE via xce_core_push_events() -- whose normal end-of-life */
/* cleanup (core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE case) correctly */
/* frees job->allocs/job->accounts, leaving job->accounts == NULL -- must */
/* NEVER have its allocation resurrected by sim_reconcile_unaccounted_ */
/* jobs(), even when this round's decoded_session carries a stray/ */
/* duplicate recorded DISPATCH entry for that same jobid (the live-observed */
/* bug-1764 shape: an array job's retried/leftover dispatch attempt still */
/* present in the replay stream after the job has already completed). */
/* Today's trigger check (sim_loop.c:690) is `job->accounts != NULL` alone, */
/* with NO check on job->state -- it cannot distinguish "not yet allocated" */
/* from "already completed and released", so THIS TEST IS EXPECTED TO FAIL */
/* until task 1.2 adds the terminal-state exclusion */
/* (JOB_IS_DONE()/JOB_IS_EXIT()). */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_never_reallocates_a_done_job_despite_stray_dispatch_entry(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* stray_alloc;
replay_session_t* decoded;
xce_core_event_t ev;
int push_rc;
fixture_init(&sim);
/* RUN-state job that already has a real allocation -- the "actively
* holding resources" shape a terminal transition must release, same
* "already accounted" fixture shape add_job()'s own header comment and
* test_reconcile_skips_job_that_already_has_accounts() above establish. */
job = add_job(&(sim.core), 7070ULL, JOB_STAT_RUN,
make_job_alloc(7070ULL, 0, NULL));
TEST_ASSERT_NOT_NULL_MESSAGE(
job->allocs, "fixture setup: job must start with a real allocation");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "fixture setup: job must start with real accounts");
/* Drive the job to JOB_STAT_DONE through the SAME production event path
* bug-1764's root cause depends on -- xce_core_push_events()'s
* XCE_CORE_EVENT_JOB_STATE_CHANGE case (core_events.c), same technique
* test_reconcile_then_release_via_job_state_change_is_asan_clean() above
* already establishes. */
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 7070ULL;
ev.u.job_state_change.old_state = JOB_STAT_RUN;
ev.u.job_state_change.new_state = JOB_STAT_DONE;
push_rc = xce_core_push_events(&(sim.core), &ev, 1);
TEST_ASSERT_EQUAL_INT(0, push_rc);
/* Precondition check (NOT this test's main assertion): confirm the
* existing end-of-life cleanup genuinely nulled allocs/accounts before
* blaming reconciliation for anything below. */
TEST_ASSERT_TRUE_MESSAGE(JOB_IS_DONE(job->state),
"precondition: job must genuinely be DONE "
"before this test's real assertion means "
"anything");
TEST_ASSERT_NULL_MESSAGE(
job->allocs, "precondition: xce_core_push_events()'s normal "
"end-of-life cleanup must null job->allocs on a "
"terminal transition");
TEST_ASSERT_NULL_MESSAGE(
job->accounts, "precondition: xce_core_push_events()'s normal "
"end-of-life cleanup must null job->accounts on a "
"terminal transition");
/* A later replay round's decoded_session still carries a stray/
* duplicate DISPATCH entry for this now-terminal job's jobid -- e.g. a
* retried dispatch attempt for one element of an array job (bug-1764's
* live-observed shape). */
stray_alloc = make_test_job_alloc_with_host(7070ULL, "worker-stray",
"slots", 9.0f);
decoded = make_decoded_session(
2000ULL, make_dispatch_entry_from_alloc(stray_alloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
/* THE ASSERTION THIS TEST EXISTS TO PROVE (expected to FAIL today): a
* DONE job's allocation must never be resurrected, no matter what the
* replay stream still contains for it. */
TEST_ASSERT_NULL_MESSAGE(
job->allocs,
"expected allocs NULL after reconciling a terminal (DONE) job, but "
"allocs was populated -- today's trigger (job->accounts == NULL "
"alone) cannot distinguish 'not yet allocated' from 'already "
"completed and released' (bug-1764)");
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"expected accounts NULL after reconciling a terminal (DONE) job, "
"but accounts was populated -- same root cause as the allocs "
"assertion above");
printf("test_reconcile_never_reallocates_a_done_job_despite_stray_"
"dispatch_entry: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 19 (scheduler-sim-reconcile-terminal-guard task 1.1, RED phase, */
/* Requirement 1.1, 1.3; bug-1764): identical scenario to Test 18 above but */
/* with JOB_STAT_EXIT instead of JOB_STAT_DONE -- both terminal states must */
/* be covered, not just one (Requirement 1.3's "treat a job's terminal */
/* state as permanent" applies to EXIT exactly as much as DONE). Also */
/* EXPECTED TO FAIL today, for the identical reason as Test 18. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry(
void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* stray_alloc;
replay_session_t* decoded;
xce_core_event_t ev;
int push_rc;
fixture_init(&sim);
job = add_job(&(sim.core), 7071ULL, JOB_STAT_RUN,
make_job_alloc(7071ULL, 0, NULL));
TEST_ASSERT_NOT_NULL_MESSAGE(
job->allocs, "fixture setup: job must start with a real allocation");
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "fixture setup: job must start with real accounts");
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 7071ULL;
ev.u.job_state_change.old_state = JOB_STAT_RUN;
ev.u.job_state_change.new_state = JOB_STAT_EXIT;
push_rc = xce_core_push_events(&(sim.core), &ev, 1);
TEST_ASSERT_EQUAL_INT(0, push_rc);
TEST_ASSERT_TRUE_MESSAGE(JOB_IS_EXIT(job->state),
"precondition: job must genuinely be EXIT "
"before this test's real assertion means "
"anything");
TEST_ASSERT_NULL_MESSAGE(
job->allocs, "precondition: xce_core_push_events()'s normal "
"end-of-life cleanup must null job->allocs on a "
"terminal transition");
TEST_ASSERT_NULL_MESSAGE(
job->accounts, "precondition: xce_core_push_events()'s normal "
"end-of-life cleanup must null job->accounts on a "
"terminal transition");
stray_alloc = make_test_job_alloc_with_host(7071ULL, "worker-stray-exit",
"slots", 9.0f);
decoded = make_decoded_session(
2100ULL, make_dispatch_entry_from_alloc(stray_alloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NULL_MESSAGE(
job->allocs,
"expected allocs NULL after reconciling a terminal (EXIT) job, but "
"allocs was populated -- today's trigger (job->accounts == NULL "
"alone) cannot distinguish 'not yet allocated' from 'already "
"completed and released' (bug-1764)");
TEST_ASSERT_NULL_MESSAGE(
job->accounts,
"expected accounts NULL after reconciling a terminal (EXIT) job, "
"but accounts was populated -- same root cause as the allocs "
"assertion above");
printf("test_reconcile_never_reallocates_an_exit_job_despite_stray_"
"dispatch_entry: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* REPL-capture helper (scheduler-sim-reconcile-terminal-guard task 2.2): */
/* duplicated from test_repl_resource_table.c's/test_repl_breakpoints_ */
/* inspection.c's own identical capture_dispatch()/capture_stdout_around() */
/* helpers, per this file family's established "duplicate rather than share */
/* via a new header" convention (design.md's File Structure Plan; those two */
/* files' own header comments document the same choice). This file already */
/* white-box #includes the full repl.c/repl_fmt.c/repl_rollup.c/repl_break. */
/* c/repl_info.c/repl_params.c stack every OTHER test here needs for */
/* sim_reconcile_unaccounted_jobs()'s own xce_repl_check_decision() call -- */
/* only the stdout-redirect capture helpers themselves are new here, not a */
/* new #include. */
/* ---------------------------------------------------------------------- */
static char* capture_stdout_around(void (*fn)(void))
{
char tmp[] = "/tmp/xce_simreconcile_replcap_XXXXXX";
int fd = mkstemp(tmp);
int saved_stdout;
off_t len;
char* buf;
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
"succeed");
fflush(stdout);
saved_stdout = dup(STDOUT_FILENO);
TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must "
"succeed");
TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0,
"dup2 redirect of STDOUT_FILENO must succeed");
fn();
fflush(stdout);
dup2(saved_stdout, STDOUT_FILENO);
close(saved_stdout);
len = lseek(fd, 0, SEEK_CUR);
TEST_ASSERT_TRUE(len >= 0);
buf = (char*)calloc(1, (size_t)len + 1);
TEST_ASSERT_NOT_NULL(buf);
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
if (len > 0) {
ssize_t nread = read(fd, buf, (size_t)len);
TEST_ASSERT_TRUE(nread == len);
}
close(fd);
unlink(tmp);
return buf;
}
static xce_sched_sim_driver_t* g_dispatch_sim;
static const char* g_dispatch_line;
static void run_dispatch_for_capture(void)
{
(void)repl_dispatch(g_dispatch_sim, g_dispatch_line);
}
static char* capture_dispatch(xce_sched_sim_driver_t* sim, const char* line)
{
g_dispatch_sim = sim;
g_dispatch_line = line;
return capture_stdout_around(run_dispatch_for_capture);
}
/* ---------------------------------------------------------------------- */
/* Test 20 (scheduler-sim-reconcile-terminal-guard task 2.2, Requirement */
/* 2.1, 2.2 -- FINAL task of this spec): REPL-visible confirmation of tasks */
/* 1.1/1.2's fix, mirroring bug-1764's live-observed shape -- an array job */
/* where some elements are DONE/EXIT while others remain running -- at the */
/* `info worker`/`info summary` display level Requirement 2 actually cares */
/* about, not just at the job->allocs/job->accounts level Tests 18/19 above */
/* already pin. */
/* */
/* Three sibling jobs, all dispatched onto the SAME worker ("workerarr"): */
/* - jobid 7080: driven to JOB_STAT_DONE via xce_core_push_events() */
/* (XCE_CORE_EVENT_JOB_STATE_CHANGE, the exact production path Test 18 */
/* uses), which correctly frees its real starting allocation. */
/* - jobids 7081/7082: remain genuinely JOB_STAT_RUN with their own real, */
/* distinct allocations to the same worker throughout. */
/* A later round's decoded_session then carries a stray/duplicate DISPATCH */
/* entry for jobid 7080 -- bug-1764's own live-observed cause (a retried/ */
/* leftover dispatch attempt for one array element still present in the */
/* replay stream after that element already completed) -- built with the */
/* exact same make_test_job_alloc_with_host()/make_dispatch_entry_from_ */
/* alloc()/make_decoded_session() helpers Tests 18/19 use. */
/* sim_reconcile_unaccounted_jobs() is called directly, the same convention */
/* design.md's Testing Strategy establishes and every other test in this */
/* file follows (the function is `static`, not driven through a full */
/* round-boundary path by any test in this suite). */
/* */
/* "info worker workerarr" and "info summary" are then captured through */
/* capture_dispatch() (above). Requirement 2.1: the completed job's id must */
/* be ABSENT from the assigned-jobs listing while both still-running jobs' */
/* ids remain present. Requirement 2.2 (display consistency): info */
/* summary's own running count -- computed purely from job->state via */
/* JOB_IS_RUN(), entirely independent of job->allocs/job->accounts -- must */
/* agree with exactly those two still-running jobs, proving the assigned- */
/* jobs listing and the running count tell the SAME story once the guard is */
/* in place (bug-1764's actual symptom was these two views disagreeing: */
/* the assigned-jobs listing showed 3 while running was correctly only 2). */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch(
void)
{
xce_sched_sim_driver_t sim;
xce_sched_worker_t* w;
xce_sched_job_t* job_done;
xce_sched_job_t* job_run1;
xce_sched_job_t* job_run2;
xce_job_alloc_t* alloc_done_initial;
xce_job_alloc_t* alloc_run1;
xce_job_alloc_t* alloc_run2;
xce_job_alloc_t* stray_alloc;
replay_session_t* decoded;
xce_core_event_t ev;
int residx;
int push_rc;
char* captured_worker;
char* captured_summary;
char done_jobid_str[64];
char run1_jobid_str[64];
char run2_jobid_str[64];
fixture_init(&sim);
residx = register_slots_resource(&(sim.core));
w = add_worker(&(sim.core), "workerarr", residx, 20.0f);
TEST_ASSERT_NOT_NULL(w);
/* jobid 7080 starts genuinely RUN with a real allocation -- the
* "actively holding resources" shape a terminal transition must
* release, same fixture shape Test 18/19 above establish. */
alloc_done_initial
= make_test_job_alloc_with_host(7080ULL, "workerarr", "slots", 3.0f);
job_done = add_job(&(sim.core), 7080ULL, JOB_STAT_RUN, alloc_done_initial);
TEST_ASSERT_NOT_NULL_MESSAGE(
job_done->allocs,
"fixture setup: job_done must start with a real allocation");
TEST_ASSERT_NOT_NULL_MESSAGE(
job_done->accounts,
"fixture setup: job_done must start with real accounts");
/* jobids 7081/7082: the "some elements remain running" half of the
* array-job scenario -- genuinely RUN, each with its own real, distinct
* allocation to the SAME worker, untouched for the rest of this test. */
alloc_run1 = make_test_job_alloc_with_host(7081ULL, "workerarr", "slots",
2.0f);
job_run1 = add_job(&(sim.core), 7081ULL, JOB_STAT_RUN, alloc_run1);
TEST_ASSERT_NOT_NULL(job_run1->allocs);
alloc_run2 = make_test_job_alloc_with_host(7082ULL, "workerarr", "slots",
4.0f);
job_run2 = add_job(&(sim.core), 7082ULL, JOB_STAT_RUN, alloc_run2);
TEST_ASSERT_NOT_NULL(job_run2->allocs);
/* Drive jobid 7080 to JOB_STAT_DONE through the SAME production event
* path bug-1764's root cause depends on (Test 18's own technique). */
memset(&ev, 0, sizeof(ev));
ev.type = XCE_CORE_EVENT_JOB_STATE_CHANGE;
ev.u.job_state_change.jobid = 7080ULL;
ev.u.job_state_change.old_state = JOB_STAT_RUN;
ev.u.job_state_change.new_state = JOB_STAT_DONE;
push_rc = xce_core_push_events(&(sim.core), &ev, 1);
TEST_ASSERT_EQUAL_INT(0, push_rc);
TEST_ASSERT_TRUE_MESSAGE(JOB_IS_DONE(job_done->state),
"precondition: job_done must genuinely be DONE "
"before this test's real assertions mean "
"anything");
TEST_ASSERT_NULL_MESSAGE(
job_done->allocs, "precondition: normal end-of-life cleanup must "
"null job_done->allocs on its terminal "
"transition");
TEST_ASSERT_NULL_MESSAGE(
job_done->accounts, "precondition: normal end-of-life cleanup must "
"null job_done->accounts on its terminal "
"transition");
/* A later replay round's decoded_session still carries a stray/
* duplicate DISPATCH entry for jobid 7080 -- bug-1764's live-observed
* shape (a retried dispatch attempt for one array element still present
* in the replay stream after that element already completed). */
stray_alloc = make_test_job_alloc_with_host(7080ULL, "workerarr",
"slots", 9.0f);
decoded = make_decoded_session(
9000ULL, make_dispatch_entry_from_alloc(stray_alloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
/* Sanity re-check (the accounting-level half of this task's own
* Observable, already covered by Tests 18/19 -- repeated here only to
* localize a failure before trusting the REPL-level assertions below). */
TEST_ASSERT_NULL_MESSAGE(
job_done->allocs,
"job_done's allocation must still be NULL after reconciling this "
"round's stray DISPATCH entry for its own jobid");
TEST_ASSERT_NULL_MESSAGE(job_done->accounts,
"job_done's accounts must likewise still be "
"NULL");
/* ---- The REPL-visible assertions this task actually exists for. ---- */
fmt_jobid(done_jobid_str, sizeof(done_jobid_str), 7080ULL);
fmt_jobid(run1_jobid_str, sizeof(run1_jobid_str), 7081ULL);
fmt_jobid(run2_jobid_str, sizeof(run2_jobid_str), 7082ULL);
captured_worker = capture_dispatch(&sim, "info worker workerarr");
TEST_ASSERT_NULL_MESSAGE(
strstr(captured_worker, done_jobid_str),
"Requirement 2.1: the completed job must be ABSENT from 'info "
"worker's assigned-jobs listing, even though this round's "
"decoded_session carried a stray DISPATCH entry for its jobid "
"(bug-1764)");
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured_worker, run1_jobid_str),
"the first still-running job must be present in the assigned-jobs "
"listing");
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured_worker, run2_jobid_str),
"the second still-running job must be present in the assigned-jobs "
"listing");
free(captured_worker);
captured_summary = capture_dispatch(&sim, "info summary");
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured_summary, "3 jobs (0 pending, 2 running)"),
"Requirement 2.2: 'info summary's own running count (computed "
"purely from job->state, independent of job->allocs/accounts) must "
"agree with the assigned-jobs listing above -- exactly the two "
"still-running jobs, never inflated by the completed one");
free(captured_summary);
printf(
"test_reconcile_info_worker_and_summary_hide_completed_job_after_"
"stray_dispatch: PASS\n");
}
/*
* bug-1759/bug-1779: a reconciled (replay-reconstructed) dispatch must
* remove the job from the picker's OWN candidate bookkeeping (fcfs_queue_t.
* pendJobs), exactly like a LIVE dispatch does -- sched_allocator()
* (scheduler.c) always calls job_picker->picker_iface->on_dispatch() before
* building the decision, but sim_reconcile_unaccounted_jobs() (this file's
* subject) never did. Left unfixed, sim_picker_thread_fn() (sim_loop.c) --
* a separate, always-running thread that walks next_job() every round and
* unconditionally PENDs whatever it is still offered (it predates this
* reconciliation mechanism and was never taught about it) -- walks right
* past the "already reconciled" job on the very next round and emits a
* bogus PEND decision for a job that has already been dispatched. This is
* the live-observed root cause of both bug-1759 (50-job batch, 22
* already-dispatched jobs also received a PEND) and bug-1779 (10-element
* array job, 5 already-dispatched elements also received a PEND).
*
* Fixture: sim_bootstrap_picker() (sim_picker.c, bug-1567) is the same
* retroactive registration a real replay's initial snapshot bootstrap
* performs -- register builtin_fcfs, then replay the existing queue/job as
* picker candidates, so `job` genuinely sits in the picker's pendJobs
* exactly as it would after a live JOBS_PUSH, without needing a full
* xce_core_push_events() event round trip.
*
* bug-1838: reconciliation's trigger is RUN-only again, so `job` must
* already be RUN by the time sim_reconcile_unaccounted_jobs() looks at it.
* But builtin_fcfs_on_job_push() (the picker hook sim_bootstrap_picker()
* calls) only ever inserts a PEND job into pendJobs -- a job constructed
* directly as RUN would never become a picker candidate in the first
* place, which would make this test's own "fixture sanity" check below
* fail for an unrelated reason (never a candidate to begin with) instead
* of proving the thing this test exists to prove.
*
* So the fixture builds the job PEND (so sim_bootstrap_picker() legitimately
* seats it in pendJobs, exactly like a live JOBS_PUSH would), then forces
* job->state to RUN directly afterward -- WITHOUT going through
* on_job_state_change(), unlike every real production path that reaches
* RUN (XCE_CORE_EVENT_JOB_STATE_CHANGE, core_events.c, always calls
* on_job_state_change() in the same breath it sets job->state, which
* already retires a no-longer-pending job from pendJobs on its own; the
* live sched_allocator() path, sched_election.c, always calls on_dispatch()
* BEFORE sched_commit_decision() sets job->state = RUN). This deliberately
* reproduces the pathological state those two invariants are supposed to
* make unreachable in production -- a RUN job the picker was never told
* about -- so this test still proves sim_reconcile_unaccounted_jobs()'s own
* on_dispatch() call (bug-1759/1779) is a genuine, working safety net, not
* dead code, even though bug-1838 closed off the one call path
* (PEND-state reconciliation) that used to reach it in practice.
*/
static void
test_reconcile_removes_reconciled_job_from_picker_candidates(void)
{
xce_sched_sim_driver_t sim;
replay_decision_t* dispatch_entry;
replay_session_t* decoded_session;
xce_sched_job_t* job;
xce_sched_plugin_t* picker = NULL;
xce_sched_job_t* offered;
int rc;
fixture_init(&sim);
job = add_job(&(sim.core), 9101ULL, JOB_STAT_PEND, NULL);
TEST_ASSERT_NOT_NULL(job);
sim_bootstrap_picker(&sim);
rc = sched_resolve_job_picker(&(sim.core), &picker);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(picker);
/* Fixture sanity: the job is genuinely a live picker candidate before
* reconciliation runs. */
picker->picker_iface->begin_session(picker);
offered = picker->picker_iface->next_job(picker);
picker->picker_iface->end_session(picker);
TEST_ASSERT_NOT_NULL_MESSAGE(
offered, "fixture sanity: job must be a picker candidate before "
"reconciliation runs");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
9101ULL, offered->jobid,
"fixture sanity: the offered candidate must be this test's own job");
/* bug-1838: force RUN directly (bypassing on_job_state_change) -- see
* this test's own header comment for why this, and not constructing
* the job as RUN from the start, is required to both (a) make it a
* genuine reconciliation candidate under the new RUN-only trigger and
* (b) keep it (pathologically) still present in pendJobs going in. */
job->state = JOB_STAT_RUN;
dispatch_entry = make_dispatch_entry(9101ULL);
decoded_session = make_decoded_session(0, dispatch_entry);
sim_reconcile_unaccounted_jobs(&sim, decoded_session, NULL);
TEST_ASSERT_NOT_NULL_MESSAGE(
job->allocs,
"reconciliation must still populate job->allocs exactly as before "
"this fix");
picker->picker_iface->begin_session(picker);
offered = picker->picker_iface->next_job(picker);
picker->picker_iface->end_session(picker);
TEST_ASSERT_NULL_MESSAGE(
offered,
"bug-1759/bug-1779: a job reconciled (dispatched) via replayed "
"decision data must be removed from the picker's candidate set, "
"exactly like a live dispatch (sched_allocator()'s on_dispatch() "
"call, scheduler.c) -- otherwise sim_picker_thread_fn()'s "
"unconditional-PEND thread re-offers and re-PENDs it on a later "
"round even though it has already been dispatched");
printf("test_reconcile_removes_reconciled_job_from_picker_candidates: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test (bug-1837, TDD): plugin.h documents on_dispatch()/on_pend() as */
/* "Post-allocation hooks (MUST) -- allocator thread, after each allocation */
/* attempt". Recorded-Decision Reconciliation used to call on_dispatch() */
/* directly from ITS OWN caller thread (sim_reconcile_unaccounted_jobs()'s */
/* thread -- normally sched_sim_driver_run()'s driver thread in production), */
/* violating that contract by construction. Dormant in production only */
/* because builtin_fcfs.c happens to guard every hook with its own internal */
/* lock -- a defensive measure plugin.h's contract text does not actually */
/* require of picker plugins in general ("the picker thread holds no lock */
/* the allocator thread needs"). */
/* */
/* Fix: once a real allocator thread is live for a core (xce_sched_sim_ */
/* driver_t.allocator_thread_live, sim.h), sim_reconcile_invoke_on_dispatch()*/
/* (sim_loop.c) routes the call through it via core->reconcile_req/ */
/* core->jbuf's XCE_SCHED_RECONCILE_WAKE sentinel/core->reconcile_ack */
/* (core.h), and sched_election.c's sched_allocator() is what actually */
/* invokes on_dispatch() on that thread. */
/* */
/* This test proves it directly, independent of builtin_fcfs's own internal */
/* locking: a minimal test-owned picker plugin records pthread_self() from */
/* inside its on_dispatch() implementation, and a one-shot thread standing */
/* in for sched_allocator() (relay_one_reconcile_request(), below) services */
/* the relay protocol. RED before the bug-1837 fix (on_dispatch() ran on */
/* this test's OWN thread -- g_dispatch_calling_tid would equal */
/* pthread_self(), and g_seen_jbuf_wake_value would never be observed since */
/* nothing was ever pushed onto jbuf at all), GREEN after. */
/* ---------------------------------------------------------------------- */
static pthread_t g_dispatch_calling_tid;
static int g_dispatch_call_count;
static uint64_t g_seen_jbuf_wake_value;
static int thread_recording_on_dispatch(void* picker, uint64_t jobid)
{
(void)picker;
(void)jobid;
g_dispatch_calling_tid = pthread_self();
g_dispatch_call_count++;
return 0;
}
static int noop_picker_session_hook(void* picker)
{
(void)picker;
return 0;
}
static xce_sched_job_t* noop_picker_next_job_hook(void* picker)
{
(void)picker;
return NULL;
}
static int noop_picker_jobid_hook(void* picker, uint64_t jobid)
{
(void)picker;
(void)jobid;
return 0;
}
static xce_job_picker_iface_t g_thread_recording_picker_iface = {
.init = NULL,
.fini = NULL,
.begin_session = noop_picker_session_hook,
.next_job = noop_picker_next_job_hook,
.end_session = noop_picker_session_hook,
.on_dispatch = thread_recording_on_dispatch,
.on_pend = noop_picker_jobid_hook,
.on_queue_update = NULL,
.on_job_push = NULL,
.on_job_state_change = NULL,
.on_job_update = NULL,
.on_config = NULL,
.on_job_rusage = NULL,
};
static xce_sched_plugin_t g_thread_recording_plugin
= { .name = (char*)"test_reconcile_thread_picker",
.picker_iface = &g_thread_recording_picker_iface };
/* One-shot stand-in for sched_election.c's real sched_allocator() --
* services exactly one Recorded-Decision Reconciliation relay request
* (core->jbuf's XCE_SCHED_RECONCILE_WAKE sentinel, then core->reconcile_req,
* then acks via core->reconcile_ack) and returns, mirroring that function's
* own SIM_RECONCILE_JBUF_WAKE branch exactly (see its own comment,
* sched_election.c, for the full protocol this mirrors). Deliberately not a
* forever loop: this test only ever sends one request. No TEST_ASSERT_*
* calls in here -- Unity's assertion macros longjmp against the MAIN test
* thread's context, unsafe to invoke from a spawned thread; this function
* only records observations into statics, asserted on after pthread_join()
* back on the main test thread. */
static void* relay_one_reconcile_request(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
uint64_t wake_val = 0;
uint64_t reconcile_jobid = 0;
xce_sched_plugin_t* job_picker = NULL;
int reconcile_rc = 0;
cbuf_get(&(core->jbuf), &wake_val);
g_seen_jbuf_wake_value = wake_val;
cbuf_get(&(core->reconcile_req), &reconcile_jobid);
(void)sched_resolve_job_picker(core, &job_picker);
if (job_picker != NULL) {
reconcile_rc = job_picker->picker_iface->on_dispatch(job_picker,
reconcile_jobid);
}
cbuf_put(&(core->reconcile_ack), &reconcile_rc);
return NULL;
}
static void
test_reconcile_invokes_on_dispatch_on_allocator_thread_not_caller(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
int residx;
pthread_t relay_tid;
int prc;
fixture_init(&sim);
g_dispatch_calling_tid = pthread_self(); /* sentinel: only meaningful if
* overwritten below */
g_dispatch_call_count = 0;
g_seen_jbuf_wake_value = 0;
TEST_ASSERT_EQUAL_INT(
0, plugin_register_core(&(sim.core), &g_thread_recording_plugin));
sim.core.job_pick_plugin = g_thread_recording_plugin.name;
residx = register_slots_resource(&(sim.core));
add_worker(&(sim.core), "worker1", residx, 1000.0f);
job = add_job(&(sim.core), 7200ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
jalloc = make_test_job_alloc_with_host(7200ULL, "worker1", "slots", 2.0f);
decoded = make_decoded_session(1000ULL, make_dispatch_entry_from_alloc(jalloc));
/* bug-1837 fix: mark a real allocator thread as live for this core, and
* actually spawn its stand-in BEFORE triggering reconciliation -- cbuf's
* own mutex+condvar makes the exact spawn/relay-vs-post ordering
* irrelevant (see relay_one_reconcile_request()'s own comment). */
sim.allocator_thread_live = true;
prc = pthread_create(&relay_tid, NULL, relay_one_reconcile_request,
&(sim.core));
TEST_ASSERT_EQUAL_INT(0, prc);
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
pthread_join(relay_tid, NULL);
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts, "fixture sanity: reconciliation must still have found "
"and captured the matching recorded decision");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_dispatch_call_count,
"on_dispatch() must be invoked exactly "
"once");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
XCE_SCHED_RECONCILE_WAKE, g_seen_jbuf_wake_value,
"bug-1837: reconciliation must wake the allocator thread via the "
"reserved jbuf sentinel, never by pushing the real jobid onto jbuf "
"directly (that would incorrectly route it through the live "
"allocation chain)");
TEST_ASSERT_FALSE_MESSAGE(
pthread_equal(g_dispatch_calling_tid, pthread_self()),
"bug-1837: on_dispatch() must run on the allocator thread, never on "
"sim_reconcile_unaccounted_jobs()'s own caller thread, once a real "
"allocator thread is live for this core -- plugin.h's documented "
"Post-allocation hooks contract (\"allocator thread, after each "
"allocation attempt\")");
TEST_ASSERT_TRUE_MESSAGE(
pthread_equal(g_dispatch_calling_tid, relay_tid),
"on_dispatch() must run specifically on the spawned allocator-relay "
"thread, not some other thread");
printf("test_reconcile_invokes_on_dispatch_on_allocator_thread_not_"
"caller: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Task 4.2 fixtures (scheduler-sim-live-allocation spec, "Sim-Mode */
/* Reconciliation Narrowing" component, Requirement 3.4/3.5): register a */
/* real, core-safe limit plugin via the SAME production bootstrap path */
/* sched_sim_driver_run() itself uses (sim_bootstrap_limit_plugin(), */
/* sim_limit.c, bug-1825), mirroring test_repl_limit_table.c's own */
/* test_limit_job_election_matches_global_limit_via_sim_registered_ */
/* plugin() fixture technique -- the same resource/limit-config shape, */
/* duplicated here per this file family's own "duplicate, don't share" */
/* convention (this file's header comment) rather than sharing statics */
/* across translation units. */
/* ---------------------------------------------------------------------- */
/* Registers a plain, non-host-scoped "mem" resource directly into `core`'s
* real resmgr -- is_host_based_limit() (src/lib/xcelimit.c) key_find()s a
* limit's val.resname directly against core->resmgr, so this fixture must
* exist for a GLOBAL (non-host-scoped) limit to resolve and match at all;
* same rationale/shape test_repl_limit_table.c's own
* register_per_host_resource() documents (that name describes the
* resource's own per-host consumption shape, not the limit's scope --
* flags carry no host-scope indicator, so is_host_based_limit() falls back
* to GLOBAL for a limit against it, exactly what these two tests need). */
static xce_resource_t* register_mem_resource(xce_sched_core_t* core)
{
xce_resource_t wire_resource;
xce_resource_t* r;
memset(&wire_resource, 0, sizeof(wire_resource));
wire_resource.name = (char*)"mem";
wire_resource.valtype = XCE_VAL_NUMERIC;
wire_resource.flags = XCE_RESF_CONSUMABLE;
r = makeResource(core->resmgr, &wire_resource);
TEST_ASSERT_NOT_NULL(r);
return r;
}
/* A minimal, heap-allocated xce_limit_config_t against the "mem" resource
* registered above -- identical shape test_repl_limit_table.c's own
* make_limit_config() documents (no per-consumer groups, so build_limit()'s
* build_consumer() falls back to get_group(mgr, "all") for every
* consumer). `name` is expected to be a string literal (build_limit()
* strdup()s it). */
static xce_limit_config_t* make_global_limit_config(const char* name)
{
xce_limit_config_t* cfg = (xce_limit_config_t*)calloc(1, sizeof(*cfg));
xce_limit_val_t* val = (xce_limit_val_t*)calloc(1, sizeof(*val));
TEST_ASSERT_NOT_NULL(cfg);
TEST_ASSERT_NOT_NULL(val);
val->resname = "mem";
val->percent = 0;
val->val = 10.0;
cfg->name = (char*)name;
cfg->val = val;
return cfg;
}
/* Bootstraps a real, core-safe limit plugin (production path:
* plugin_register_core() + limit_seed_config(), via sim_bootstrap_limit_
* plugin() -- sim_limit.c, bug-1825) against `sim`, seeded with one GLOBAL
* limit named `limit_name`, and bumps core->configver past the fresh
* xce_sched_obj_t default of 0 -- limit_job_election()'s own "already
* evaluated against this config" guard (obj->configver >= core->configver)
* would otherwise skip matching entirely, since both default to 0. Same
* ordering/rationale test_repl_limit_table.c's own
* test_limit_job_election_matches_global_limit_via_sim_registered_plugin()
* establishes. Returns the registered plugin (for callers that want to
* assert on it directly; both tests below only need the side effects). */
static xce_sched_plugin_t*
bootstrap_global_limit_plugin(xce_sched_sim_driver_t* sim,
const char* limit_name)
{
xce_sched_plugin_t* plugin = NULL;
xce_limit_config_t* limits[1];
xce_conf4sched_t config;
int rc;
TEST_ASSERT_NOT_NULL(register_mem_resource(&(sim->core)));
limits[0] = make_global_limit_config(limit_name);
memset(&config, 0, sizeof(config));
config.nlimits = 1;
config.limits = limits;
sim->core.config = &config;
sim_bootstrap_limit_plugin(sim);
rc = key_find(sim->core.plugins, "xce_sched_limit", (void**)&plugin);
TEST_ASSERT_TRUE(rc >= 0);
TEST_ASSERT_NOT_NULL(plugin);
sim->core.configver = 1;
return plugin;
}
/* ---------------------------------------------------------------------- */
/* Test (task 4.2's own PRIMARY Observable, Requirement 3.4): a job whose */
/* dispatch is found only via reconciliation still gets its GLOBAL (non- */
/* host-scoped) limit usage registered, through the same core-safe */
/* plugin_job_election() dispatch normal mode's own sched_allocator() calls */
/* (task 1.2) -- design.md's "Sim-Mode Reconciliation Narrowing" component. */
/* Host-scoped matching is deliberately OUT of scope for this call (design */
/* .md's own text: it needs a resolved worker context this function does */
/* not independently construct) -- this test only proves the GLOBAL path. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_registers_global_limit_usage_for_reconciled_job(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_alloc_t* jalloc;
replay_session_t* decoded;
var instances_handle;
iterator_t it;
void* item = NULL;
int rc;
xce_resource_t* mem_res;
fixture_init(&sim);
bootstrap_global_limit_plugin(&sim, "sim_reconcile_global_limit");
/* bug-1836 fix: the capacity gate needs a REAL registered "worker1"
* owning the "mem" resource. bootstrap_global_limit_plugin() above
* already registered "mem" itself (via its own register_mem_resource()
* call, for limit-matching purposes) -- calling register_mem_resource()
* a second time here would fail (makeResource() rejects a duplicate
* name), so look the already-registered resource up instead; only the
* WORKER instance owning it is still missing (the capacity gate's
* per-host check needs sched_get_worker("worker1") to resolve, which
* bootstrap_global_limit_plugin() never adds). */
TEST_ASSERT_TRUE(
key_find(sim.core.resmgr, (void*)"mem", (void**)&mem_res) >= 0);
TEST_ASSERT_NOT_NULL(mem_res);
add_worker(&(sim.core), "worker1", mem_res->idx, 1000.0f);
job = add_job(&(sim.core), 7090ULL, JOB_STAT_RUN, NULL);
TEST_ASSERT_NULL(job->accounts);
/* match_job_global_limit_() (plugins/limit.c) matches against
* job->schedobj->user/queue/project directly -- add_job()'s own
* core_build_job() path leaves ->project NULL (submit.project_name is
* never set by that shared fixture helper, and makeString(NULL)
* returns NULL, src/lib/xstring.c), which match_limit() cannot safely
* take. Set it directly here, scoped to this test only, rather than
* widening add_job() itself for every other test in this file that has
* no need of it. */
job->schedobj->project = (char*)"";
jalloc = make_test_job_alloc_with_host(7090ULL, "worker1", "mem", 2.0f);
decoded = make_decoded_session(1000ULL,
make_dispatch_entry_from_alloc(jalloc));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts,
"sanity: the job's dispatch must still be reconstructed exactly as "
"before this task");
instances_handle = limit_get_usage_instances(&(sim.core));
TEST_ASSERT_NOT_NULL_MESSAGE(
instances_handle,
"task 4.2: a job whose dispatch is found only via reconciliation "
"must still have its GLOBAL limit usage registered, via the same "
"core-safe plugin_job_election() dispatch normal mode uses");
rc = iter_make(&it, instances_handle);
TEST_ASSERT_TRUE(rc >= 0);
rc = iter_next(&it, &item);
TEST_ASSERT_TRUE(rc >= 0);
TEST_ASSERT_NOT_EQUAL(RTTI_ITER_NO_DATA, rc);
TEST_ASSERT_NOT_NULL_MESSAGE(
item, "a real usage instance must exist for the reconciled job's "
"global limit match, visible through the existing "
"limit_get_usage_instances() accessor -- the same one `info "
"limit` reads");
printf("test_reconcile_registers_global_limit_usage_for_reconciled_job: "
"PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test (task 4.2, Requirement 3.5's supporting guard): a job the LIVE path */
/* has already decided (job->accounts != NULL, as task 4.1's live picker/ */
/* allocator would leave it) is never reprocessed by this mechanism -- the */
/* existing trigger guard (job->accounts == NULL) already excludes it */
/* BEFORE this task's new plugin_job_election() call is ever reached, so no */
/* limit usage is (re-)registered and the job's own schedobj->configver */
/* stays untouched. Extends test_reconcile_skips_job_that_already_has_ */
/* accounts (Test 3 above), which already proves job->accounts/n_decisions */
/* are untouched for this shape -- this test proves the NEW call site */
/* respects that same exclusion. */
/* ---------------------------------------------------------------------- */
static void
test_reconcile_does_not_reregister_limit_for_already_decided_job(void)
{
xce_sched_sim_driver_t sim;
xce_sched_job_t* job;
xce_job_account_t* accounts_before;
replay_session_t* decoded;
var instances_handle;
iterator_t it;
void* item = NULL;
int rc;
fixture_init(&sim);
bootstrap_global_limit_plugin(&sim, "sim_reconcile_already_decided_limit");
/* Job already decided by the LIVE path (task 4.1) -- job->accounts !=
* NULL, the same "already accounted" fixture shape Test 3
* (test_reconcile_skips_job_that_already_has_accounts) above
* establishes. */
job = add_job(&(sim.core), 7091ULL, JOB_STAT_RUN,
make_job_alloc(7091ULL, 0, NULL));
TEST_ASSERT_NOT_NULL_MESSAGE(
job->accounts,
"fixture setup: job must already have accounts, as if the live "
"path already decided it");
accounts_before = job->accounts;
job->schedobj->project = (char*)"";
decoded = make_decoded_session(1000ULL, make_dispatch_entry(7091ULL));
sim_reconcile_unaccounted_jobs(&sim, decoded, NULL);
TEST_ASSERT_EQUAL_PTR_MESSAGE(
accounts_before, job->accounts,
"an already-decided job must never be re-touched by reconciliation "
"(unchanged, pre-existing guard)");
TEST_ASSERT_EQUAL_INT_MESSAGE(
0, job->schedobj->configver,
"task 4.2: the existing trigger guard already excludes an "
"already-decided job before the new plugin_job_election() call is "
"reached -- its schedobj->configver (bumped by a real election "
"call) must stay at its untouched default");
instances_handle = limit_get_usage_instances(&(sim.core));
TEST_ASSERT_NOT_NULL(instances_handle);
rc = iter_make(&it, instances_handle);
TEST_ASSERT_TRUE(rc >= 0);
rc = iter_next(&it, &item);
TEST_ASSERT_EQUAL_INT_MESSAGE(
RTTI_ITER_NO_DATA, rc,
"no limit usage instance may be created for an already-decided "
"job via this mechanism -- plugin_job_election() must never reach "
"it");
printf("test_reconcile_does_not_reregister_limit_for_already_decided_"
"job: PASS\n");
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(
test_reconcile_leaves_run_job_unaccounted_when_no_recorded_decision);
RUN_TEST(test_reconcile_is_safe_to_recheck_on_a_later_round);
RUN_TEST(test_reconcile_skips_job_that_already_has_accounts);
RUN_TEST(test_reconcile_ignores_non_dispatch_entry_with_matching_jobid);
RUN_TEST(
test_reconcile_ignores_pend_state_job_even_when_dispatch_recorded);
RUN_TEST(
test_reconcile_reconstructs_allocs_and_accounts_from_matching_dispatch);
RUN_TEST(test_reconcile_compacts_via_shift_down_preserving_order);
RUN_TEST(test_reconcile_then_release_via_job_state_change_is_asan_clean);
RUN_TEST(
test_reconcile_reports_reconstructed_decision_through_existing_channel);
RUN_TEST(test_reconcile_null_args_are_a_safe_noop);
RUN_TEST(test_captured_job_included_in_resinuse_same_round_as_capture);
RUN_TEST(
test_reconcile_still_skips_already_captured_job_regardless_of_state);
RUN_TEST(test_reconcile_no_match_leaves_pend_job_uncaptured_and_retries);
RUN_TEST(
test_reconcile_multiple_jobs_captured_independently_across_different_rounds);
RUN_TEST(
test_reconcile_continue_mode_with_repl_state_suppresses_bare_decision_line);
RUN_TEST(
test_reconcile_step_mode_with_repl_state_still_prints_bare_decision_line);
RUN_TEST(
test_reconcile_never_reallocates_a_done_job_despite_stray_dispatch_entry);
RUN_TEST(
test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry);
RUN_TEST(
test_reconcile_info_worker_and_summary_hide_completed_job_after_stray_dispatch);
RUN_TEST(test_reconcile_removes_reconciled_job_from_picker_candidates);
RUN_TEST(
test_reconcile_invokes_on_dispatch_on_allocator_thread_not_caller);
RUN_TEST(test_reconcile_registers_global_limit_usage_for_reconciled_job);
RUN_TEST(
test_reconcile_does_not_reregister_limit_for_already_decided_job);
return UNITY_END();
}
Reference in a new issue View git blame Copy permalink