/* * test_sim_e2e_original_motivation.c - task 4.3 (scheduler-sim-decision- * parity spec, FINAL task, Req 1.1/1.2/1.3, 3.1/3.2/3.3): live-verify the * fix against a substitute for the original motivating recording. * * ============================================================================ * TRANSPARENCY NOTE (read this before anything else in this file) * ============================================================================ * The task this file implements asked to replay the EXACT historical * recording that surfaced bug-1611/bug-1613 * (`scheduler-replay-20260820-070103.replay`, a real 108-round session). * That exact file does not exist anywhere on this filesystem -- confirmed by * an exhaustive search (find over /, /tmp, every project directory; grep of * .wolf/buglog.json and .wolf/memory.md for any other path hint) before * writing a single line of this file. It was incidental production/test * data from whenever bug-1611/bug-1613 were originally investigated, never a * deliverable of this spec, and was never checked in or guaranteed to * persist. This file builds a REAL substitute instead: an actual 24-round, * 12-job, resource-constrained scheduling session driven through the real * driver/scheduler code (sched_record_start()/sched_record_stop(), * xce_sched_driver_run_normal_mode_round(), the real builtin_fcfs picker, * and a capacity-aware allocator thread that drives the real resource.c * capacity-check/accounting primitives -- the same technique * test_sim_resource_contention.c/test_sim_contention_parity.c already * establish and this feature's own prior tasks already reviewed and * accepted), then replays THAT through simulation mode as the live-verify. * * ONE further, narrower substitution decision, disclosed here in full * (do not skip this if reviewing this file): a job that is *already * running, with an unaccounted allocation, when a scheduling round's own * live decision-making reaches it* is Recorded-Decision Reconciliation's * entire reason for existing (design.md's "Recorded-Decision Reconciliation" * component; tasks 2.2-2.4). Constructing that exact shape from a single, * continuously live-driven session is structurally impossible, not merely * inconvenient -- confirmed by reading sim_reconcile_unaccounted_jobs() * (sim_loop.c) and xce_sched_driver_translate_pending_events()/ * xce_sched_driver_run_normal_mode_round() (driver.c) directly: * - Reconciliation matches a RUN-state, unaccounted job against a DISPATCH * decision recorded in THAT SAME ROUND's own decoded ReplaySession -- * never an earlier or later round's data (each round's decoded_session * is freed immediately after that round's own processing). * - A job cannot be BOTH "freshly PEND when this round's live picker * session starts" (required for any live picker/allocator -- real or * simulated -- to ever produce a DISPATCH decision for it) AND "already * RUN before this round's live picker session starts" (required for * reconciliation to even consider it) in the SAME round: builtin_fcfs's * on_dispatch()/on_job_state_change() hooks make these two states * mutually exclusive by construction, and sim mode's own * sim_picker_thread_fn() is unconditionally PEND-only besides (confirmed * by direct read -- no dispatch branch exists there at all, matching * task 4.1's own finding). * Every existing test in this feature that exercises reconciliation's FOUND * (successful match) path -- test_sim_round_boundary_reconciliation.c (task * 3), test_sim_contention_parity.c's "Part B" (task 4.1) -- resolves this * the same way: hand-construct the one round where the trigger condition * and its matching decision coincide. This file uses the SAME resolution, * but narrowed to the absolute minimum and built through higher-level real * production APIs rather than raw capnp construction: * - ANCHOR_ID's arrival (JOBS_PUSH, PEND) and RUN confirmation * (JOB_STATE_CHANGE) are both queued and applied through the REAL * dataq_enq()/xce_sched_driver_translate_pending_events() path used * everywhere else in this file -- not a single raw ReplayEvent is * hand-encoded for this. * - ANCHOR_ID's own "originating decision" -- the one piece no live * picker/allocator (real or simulated) can ever itself produce for a * job in this state -- is built with the exact same real production * constructors (make_charge_account()-equivalent, sched_commit_job_alloc(), * make_dispatch_alloc(), make_dispatch_decision()) this file's own * generic_allocator_thread_fn() already uses for every genuinely-live- * decided job below, and is fed through the exact same real * core->decisions cbuf -> xce_sched_driver_run_normal_mode_round()'s own * drain/record loop that records every other decision in this file. * Nothing about the .replay file's BYTES is hand-written; only WHICH * xce_decision_t gets enqueued for this one job, this one round, is * test-authored, exactly mirroring how this whole test-file family * already substitutes a "capacity-aware allocator thread" for * scheduler.c's real sched_allocator() (see test_sim_resource_ * contention.c's own header comment for that established precedent). * Every OTHER job in this recording (J2-J12, 11 of them) arrives, competes * for capacity, dispatches or is denied, and eventually finishes across 24 * real rounds with zero special-casing -- ordinary, organic * arrival/contention/churn, indistinguishable from any other fixture in * this test-file family. * * Scenario: worker "worker-e2e" has 20.0 units of one CONSUMABLE resource * ("slots"). ANCHOR_ID holds 12.0 units for the ENTIRE 24-round recording * (never finishes) -- the "job that stays running across many rounds" the * task asks for, independently confirmed alive via periodic JOB_RUSAGE * ticks pushed through the real dataq/translate path on 11 of those rounds * (mirroring bug-1611's own "confirmed via ongoing JOB_UPDATE/JOB_RUSAGE * ticks" observation -- a signal that does NOT depend on the resource- * accounting mechanism under test, so it is a genuinely independent * confirmation, not a circular one). The remaining 8.0 units of capacity * are real, live contention ground for J2-J12: each arrives, is denied * (PEND, recurring every round per builtin_fcfs's own documented "still- * PEND job recurs every round" behavior) until an earlier job's real * JOB_STATE_CHANGE-to-DONE frees enough capacity, then dispatches for real, * runs for a few rounds, and eventually finishes itself -- jobs genuinely * come and go for the whole 24-round span. * * Live-verify (this file's actual Observable Completion, Req 1.1-1.3, * 3.1-3.3): replay the resulting `.replay` file through the REAL, * unmodified interactive REPL (sched_sim_driver_run(), a forked child with * stdin freopen()'d from a step/continue/info-worker script and stdout * captured -- the exact technique test_repl_execution_control.c/ * test_sim_verify.c already establish). Assert, IN ORDER, across FOUR * checkpoints spanning the whole replay (after 3, 13, 21, and all 24 * rounds): "info worker worker-e2e" reports nonzero, and exactly the * arithmetically-correct, used= value at each point (17.00, 18.00, 20.00, * 12.00) -- not merely "nonzero at some point". Also assert ANCHOR_ID's own * DISPATCH decision (the one this whole substitution exists to make * observable) appears in the captured step/continue output, via the exact * same "[sim] DISPATCH jobid=..." line bug-1611 documented as NEVER * appearing pre-fix. * * White-box #include set: the same combined core/driver/plugin/recording/ * repl/resource/sim stack test_sim_round_boundary_reconciliation.c and * test_sim_resource_contention.c already establish (this file needs both: * a REAL, driven recording -- driver.c -- AND real resource.c capacity- * check/accounting primitives). Links a REAL (not no-op) sched_account_jobs() * reimplementation, copied verbatim from scheduler.c (task 1.1's relocated * signature) -- same rationale those two files' own header comments give. * * ============================================================================ * ADDENDUM (scheduler-snapshot-job-allocation spec, task 4.2): PREEXIST_ID * ============================================================================ * Everything above this addendum predates and is unchanged by this task. * Task 4.2's own Observable Completion ("100% of the saved running jobs' * resource contribution accounted for immediately on load") is NOT proven * by ANCHOR_ID above: ANCHOR_ID's accounting is recovered by replay-based * reconciliation (sim_reconcile_unaccounted_jobs(), a DIFFERENT, already- * shipped mechanism from a prior spec) matching its DISPATCH decision * against round 1's own decoded ReplaySession -- i.e. its correctness * depends on a dispatch decision that DOES appear somewhere in the replay * stream. This spec (scheduler-snapshot-job-allocation) closes the * COMPLEMENTARY case: a job whose dispatch predates the snapshot entirely * and can NEVER appear in the subsequent replay stream, recoverable only * via xce_sched_core_load_snapshot()'s own allocation restoration (task * 2.1's build_wire_job() save-side wiring + task 2.2's dup_job_alloc()/ * make_job_account() load-side wiring). * * PREEXIST_ID (job 70013, worker "worker-e2e-pre", a dedicated worker * isolated from worker-e2e's carefully-derived R2-R24 capacity arithmetic * above) is constructed to be exactly that case: submitted, live-dispatched * (a completely ordinary, unrigged sched_check_host_satisfy()-gated * dispatch -- see preexist_prelude_allocator_thread_fn below), and * confirmed RUN, ALL strictly before sched_record_start() is ever called. * Its dispatch decision is therefore never written to the .replay file at * all -- sched_record_start()'s own initial SnapshotEntry (recording.c's * write_snapshot_entry()) is the ONLY place its allocation is ever * captured, and xce_sched_core_load_snapshot() (sim.c's * sim_locate_and_load_snapshot(), called before round 1 ever runs) is the * ONLY place it is ever restored. The test asserts PREEXIST_ID's "used" * value is already correct after just ONE replayed round (the earliest * possible observation point, strictly before ANCHOR_ID's own first * checkpoint) and stays correct through all 24, while its own DISPATCH * decision is proven to NEVER appear anywhere in the captured replay * output -- the concrete, falsifiable difference from ANCHOR_ID's * reconciliation-based mechanism. */ #include #include #include #include #include #include #include #include #include #include #include "controller.capnp.h" #include "kvmgr.h" #include "preason.h" #include "resobj.h" #include "scheduler.h" #include "xstring.h" /* Pull in the real functions under test into this TU (white-box #include, * combined set: test_sim_resource_contention.c's driver+resource.c set, * PLUS repl_break.c/repl_info.c/repl_params.c for the interactive REPL * commands this file's live-verify drives (step/continue/info worker), * matching test_repl_execution_control.c's own set). */ #include "../src/core.c" #include "../src/core_clock.c" #include "../src/core_events.c" #include "../src/core_session.c" #include "../src/decision.c" #include "../src/driver.c" #include "../src/plugin.c" #include "../src/plugins/builtin_fcfs.c" #include "../src/recording.c" #include "../src/recording_events.c" #include "../src/recording_session.c" #include "../src/repl.c" #include "../src/repl_break.c" #include "../src/repl_info.c" #include "../src/repl_params.c" #include "../src/resource.c" #include "../src/sim.c" #include "../src/sim_loop.c" #include "../src/sim_picker.c" /* bug-1567: sched_sim_driver_run() calls sim_bootstrap_picker() (sim_loop.c-cap sibling) as an extern -- undefined reference at link time otherwise */ #include "../src/sim_record.c" #include "../src/sim_verify.c" #include "../src/snapshot.c" #include "../src/snapshot_load.c" #include "../src/snapshot_wire.c" void kill_sched_job(void* arg) { (void)arg; } void kill_sched_queue(void* arg) { (void)arg; } void kill_sched_worker(void* arg) { (void)arg; } void kill_sched_user(void* arg) { (void)arg; } void kill_sched_project(void* arg) { (void)arg; } void kill_sched_resval(xce_val_t* val) { (void)val; } /* resource.c's sched_try_alloc_on_host_()'s own link-time dependency -- * never actually invoked (this file calls sched_check_host_satisfy()/ * sched_charge_account()/sched_commit_job_alloc() directly). Same stub * test_sim_resource_contention.c's/test_sim_round_boundary_reconciliation.c's * own resource.c inclusion already establishes needing. */ int sched_ctx_get_uncomit_host(xvec_t** v, xce_sched_ctx_t* c, int i) { (void)v; (void)c; (void)i; return 0; } /* xce_sched_driver_on_decision()'s (driver.c) publish-side linker * dependencies -- never actually invoked (this file never registers a * decision callback). */ int sched_publish_dispatch(xce_scheduler_t* sched, xce_job_dispatch_t* d) { (void)sched; (void)d; return 0; } int sched_publish_pend_reason(xce_scheduler_t* sched, xce_preason_set_t* r) { (void)sched; (void)r; return 0; } /* handle_deferred_admin_record()'s (driver.c) linker dependency -- never * invoked (this file calls sched_record_start()/sched_record_stop() * directly, never through XCE_SCHED_OP_ADMIN_RECORD). */ int xcecli_reply_schedadmin(xcecli_ctx_t c, uint64_t seq, sched_admin_reply_t* reply) { (void)c; (void)seq; (void)reply; return 0; } /* Real reimplementation of schedobj.c's sched_resolve_job_picker -- same * copy every white-box test in this directory that does not link * schedobj.c already establishes. */ int sched_resolve_job_picker(xce_sched_core_t* core, xce_sched_plugin_t** out) { xce_sched_plugin_t* plugin = NULL; int rc = 0; if ((core == NULL) || (out == NULL)) { return -EINVAL; } *out = NULL; if (core->job_pick_plugin == NULL) { return -EINVAL; } lock(core->plugins, RTTI_LOCK_READ); rc = key_find(core->plugins, core->job_pick_plugin, (void**)&plugin); unlock(core->plugins); if (rc <= 0) { return (rc < 0) ? rc : -ENOENT; } if ((plugin == NULL) || (plugin->picker_iface == NULL) || (plugin->picker_iface->next_job == NULL) || (plugin->picker_iface->begin_session == NULL) || (plugin->picker_iface->end_session == NULL) || (plugin->picker_iface->on_dispatch == NULL) || (plugin->picker_iface->on_pend == NULL)) { return -EINVAL; } *out = plugin; return 0; } /* Real reimplementation of scheduler.c's sched_account_jobs() (scheduler.c: * 1258, task 1.1's relocated xce_sched_core_t* signature) -- body copied * verbatim, same rationale test_sim_round_boundary_reconciliation.c's and * test_sim_contention_parity.c's own copies document: this file's whole * purpose is to prove what this function does to resInUse across a real, * long replay, so a no-op stub would defeat the test. */ int sched_account_jobs(xce_sched_core_t* core) { iterator_t it; int rc = 0; rc = iter_make(&it, core->jobs); if (rc < 0) { return rc; } for (;;) { xce_sched_job_t* job = NULL; rc = iter_next(&it, (void**)&job); if ((rc < 0) || (rc == RTTI_ITER_NO_DATA)) { break; } if (!JOB_IS_RUN(job->state)) { continue; } if (job->accounts != NULL) { sched_charge_account(core, job->accounts); } } if (rc == RTTI_ITER_NO_DATA) { rc = 0; } iter_destroy(&it); return rc; } void setUp(void) { return; } void tearDown(void) { return; } /* ---------------------------------------------------------------------- */ /* Fixture constants */ /* ---------------------------------------------------------------------- */ #define WORKER_NAME "worker-e2e" #define WORKER_CAPACITY 20.0f /* scheduler-snapshot-job-allocation task 4.2: PREEXIST_ID's own dedicated * worker, isolated from worker-e2e's R2-R24 capacity arithmetic above -- * see this file's ADDENDUM header comment. */ #define WORKER2_NAME "worker-e2e-pre" #define WORKER2_CAPACITY 15.0f #define PREEXIST_ID 70013ULL #define PREEXIST_WANT 9.0f #define ANCHOR_ID 70001ULL #define ANCHOR_WANT 12.0f #define J2_ID 70002ULL #define J3_ID 70003ULL #define J4_ID 70004ULL #define J5_ID 70005ULL #define J6_ID 70006ULL #define J7_ID 70007ULL #define J8_ID 70008ULL #define J9_ID 70009ULL #define J10_ID 70010ULL #define J11_ID 70011ULL #define J12_ID 70012ULL static int g_slots_residx = -1; /* ---------------------------------------------------------------------- */ /* Mutable per-job "want" lookup -- generalizes test_sim_resource_ * contention.c's own fixed lookup_want() switch to an arbitrary, growable * job set (12 jobs here, vs. that file's fixed 3). */ /* ---------------------------------------------------------------------- */ #define MAX_JOBS 16 static uint64_t g_job_ids[MAX_JOBS]; static float g_job_wants[MAX_JOBS]; static int g_n_jobs = 0; static void register_want(uint64_t jobid, float want) { TEST_ASSERT_TRUE(g_n_jobs < MAX_JOBS); g_job_ids[g_n_jobs] = jobid; g_job_wants[g_n_jobs] = want; g_n_jobs++; } static float lookup_want(uint64_t jobid) { int i; for (i = 0; i < g_n_jobs; i++) { if (g_job_ids[i] == jobid) { return g_job_wants[i]; } } return 0.0f; } /* ---------------------------------------------------------------------- */ /* Fixture helpers */ /* ---------------------------------------------------------------------- */ static void make_tmp_dir(char* out, size_t out_len, const char* suffix) { char tmpl[PATH_MAX]; snprintf(tmpl, sizeof(tmpl), "/tmp/sched_e2e_orig%s_XXXXXX", suffix); TEST_ASSERT_NOT_NULL(mkdtemp(tmpl)); TEST_ASSERT_TRUE(strlen(tmpl) < out_len); strcpy(out, tmpl); } /* Real xce_sched_core_init() + real builtin_fcfs registration + a real * "slots" resource registered via the real makeResource() (so "info worker" * -- repl_info.c's real, unmodified display code -- can resolve its name * and CONSUMABLE flag, exactly as test_sim_round_boundary_reconciliation.c's * fixture_init() establishes) + a real worker with 20.0 finite units. */ static void fixture_init(xce_scheduler_t* sched, char* record_dir) { xce_sched_plugin_t* fcfs; xce_sched_worker_t* w; xce_resource_t* res_template; xce_resource_t* registered; int rc; float val[1]; memset(sched, 0, sizeof(*sched)); mylock_init(&(sched->record_lock)); rc = xce_sched_core_init(&(sched->core)); TEST_ASSERT_EQUAL_INT(0, rc); sched->core.pend_updates[0] = (dataq_t*)calloc(1, sizeof(dataq_t)); dataq_init(sched->core.pend_updates[0], NULL); sched->core.pend_updates[1] = (dataq_t*)calloc(1, sizeof(dataq_t)); dataq_init(sched->core.pend_updates[1], NULL); fcfs = builtin_fcfs_new_plugin(); TEST_ASSERT_NOT_NULL(fcfs); rc = plugin_register_core(&(sched->core), fcfs); TEST_ASSERT_TRUE(rc >= 0); sched->core.job_pick_plugin = (char*)"builtin_fcfs"; sched->record_dir = record_dir; res_template = (xce_resource_t*)calloc(1, sizeof(*res_template)); TEST_ASSERT_NOT_NULL(res_template); res_template->name = strdup("slots"); res_template->valtype = XCE_VAL_NUMERIC; res_template->ordtype = XCE_ORD_DECR; res_template->flags = XCE_RESF_LIMIT | XCE_RESF_CONSUMABLE | XCE_RESF_RELEASE | XCE_RESF_DEFAULT_HOST; registered = makeResource(sched->core.resmgr, res_template); TEST_ASSERT_NOT_NULL(registered); g_slots_residx = registered->idx; w = (xce_sched_worker_t*)calloc(1, sizeof(*w)); TEST_ASSERT_NOT_NULL(w); w->name = strdup(WORKER_NAME); TEST_ASSERT_NOT_NULL(w->name); rc = zarray_init(&(w->resvals), 8, NULL); TEST_ASSERT_EQUAL_INT(0, rc); rc = zarray_init(&(w->resInUse), 8, NULL); TEST_ASSERT_EQUAL_INT(0, rc); val[0] = WORKER_CAPACITY; rc = put_resvals(&(w->resvals), 1, &g_slots_residx, val); TEST_ASSERT_TRUE(rc >= 0); rc = sched_add_worker(&(sched->core), w); TEST_ASSERT_TRUE(rc >= 0); /* scheduler-snapshot-job-allocation task 4.2: PREEXIST_ID's own * dedicated worker (see this file's ADDENDUM header comment) -- * isolated from worker-e2e's own carefully-derived R2-R24 capacity * arithmetic above so this task's addition cannot perturb it. */ { xce_sched_worker_t* w2; w2 = (xce_sched_worker_t*)calloc(1, sizeof(*w2)); TEST_ASSERT_NOT_NULL(w2); w2->name = strdup(WORKER2_NAME); TEST_ASSERT_NOT_NULL(w2->name); rc = zarray_init(&(w2->resvals), 8, NULL); TEST_ASSERT_EQUAL_INT(0, rc); rc = zarray_init(&(w2->resInUse), 8, NULL); TEST_ASSERT_EQUAL_INT(0, rc); val[0] = WORKER2_CAPACITY; rc = put_resvals(&(w2->resvals), 1, &g_slots_residx, val); TEST_ASSERT_TRUE(rc >= 0); rc = sched_add_worker(&(sched->core), w2); TEST_ASSERT_TRUE(rc >= 0); } } static xce_queue_t* make_queue(const char* name, int32_t priority) { xce_queue_t* q = (xce_queue_t*)calloc(1, sizeof(*q)); q->queue = strdup(name); q->status = QUEUE_STAT_OPEN; q->priority = priority; return q; } static xce_job_t* make_job(uint64_t jobid, uint64_t sub_time, const char* queue, const char* user) { xce_job_t* j = (xce_job_t*)calloc(1, sizeof(*j)); j->jobid = jobid; j->state = JOB_STAT_PEND; j->sub_time = sub_time; j->queue_time = sub_time; j->start_time = 0; j->submit = (job_submit_data_t*)calloc(1, sizeof(*(j->submit))); j->submit->queue = strdup(queue); j->submit->user = strdup(user); j->submit->project_name = strdup("e2e_orig_proj"); j->submit->resreq = strdup("type == any"); j->submit->mintasks = 1; j->submit->maxtasks = 1; return j; } static void enqueue_queue_update(xce_scheduler_t* sched, const char* name, int32_t priority) { xce_sched_update_request_t* req = (xce_sched_update_request_t*)calloc(1, sizeof(*req)); req->op = XCE_SCHED_OP_UPDATE_QUEUE; req->u.queueupdate.queue = make_queue(name, priority); dataq_enq(sched->core.pend_updates[0], &(req->link)); } static void enqueue_jobs_push(xce_scheduler_t* sched, xce_job_t** jobs, int njobs) { xce_sched_update_request_t* req = (xce_sched_update_request_t*)calloc(1, sizeof(*req)); req->op = XCE_SCHED_OP_UPDATE_JOB; req->u.jobupdate.njobs = njobs; req->u.jobupdate.jobs = jobs; dataq_enq(sched->core.pend_updates[0], &(req->link)); } /* XCE_SCHED_OP_JSTATE_CHANGE -> XCE_CORE_EVENT_JOB_STATE_CHANGE (driver.c's * own op-type table) -- real, minimal-payload production event used both * to confirm a dispatched job is now RUN and to mark a running job DONE * (freeing its allocation via core_events.c's own JOB_STATE_CHANGE case). */ static void enqueue_jstate_change(xce_scheduler_t* sched, uint64_t jobid, uint32_t state) { xce_sched_update_request_t* req = (xce_sched_update_request_t*)calloc(1, sizeof(*req)); req->op = XCE_SCHED_OP_JSTATE_CHANGE; req->u.job_state_change.jobid = jobid; req->u.job_state_change.state = state; dataq_enq(sched->core.pend_updates[0], &(req->link)); } /* XCE_SCHED_OP_JOB_RUSAGE -> XCE_CORE_EVENT_JOB_RUSAGE -- an ongoing usage * tick for a running job, independent of the resource-accounting mechanism * under test (core_events.c's JOB_RUSAGE case only ever sets job->rusages; * it never touches job->accounts/resInUse). Used here purely as the * independent "this job is still alive" signal bug-1611 itself cited. */ static void enqueue_job_rusage(xce_scheduler_t* sched, uint64_t jobid, const char* resname, float value) { xce_rusage_t* ru = (xce_rusage_t*)calloc(1, sizeof(*ru)); xce_rusage_t** usages = (xce_rusage_t**)calloc(1, sizeof(*usages)); xce_job_rusages_t* rusages = (xce_job_rusages_t*)calloc(1, sizeof(*rusages)); xce_sched_update_request_t* req = (xce_sched_update_request_t*)calloc(1, sizeof(*req)); ru->resname = strdup(resname); ru->value = value; usages[0] = ru; rusages->jobid = jobid; rusages->count = 1; rusages->usages = usages; req->op = XCE_SCHED_OP_JOB_RUSAGE; req->u.job_rusage.rusages = rusages; dataq_enq(sched->core.pend_updates[0], &(req->link)); } /* ---------------------------------------------------------------------- */ /* Picker thread -- identical to every sibling fixture in this directory: */ /* resolves and drives the REAL builtin_fcfs picker. */ /* ---------------------------------------------------------------------- */ static void* picker_thread_fn(void* arg) { xce_sched_core_t* core = (xce_sched_core_t*)arg; xce_sched_plugin_t* job_picker = NULL; int rc; int go = 0; uint64_t jobid; rc = sched_resolve_job_picker(core, &job_picker); if (rc < 0) { return NULL; } cbuf_get(&(core->wbuf), &go); job_picker->picker_iface->begin_session(job_picker); for (;;) { xce_sched_job_t* job = job_picker->picker_iface->next_job(job_picker); if (job == NULL) { break; } jobid = job->jobid; cbuf_put(&(core->jbuf), &jobid); } jobid = 0; cbuf_put(&(core->jbuf), &jobid); job_picker->picker_iface->end_session(job_picker); return NULL; } /* ---------------------------------------------------------------------- */ /* Decision-construction helpers (real production field shapes -- same */ /* technique test_sim_resource_contention.c's/test_sim_round_boundary_ */ /* reconciliation.c's own make_dispatch_alloc()/make_charge_account() */ /* establish, generalized to take an explicit residx). */ /* ---------------------------------------------------------------------- */ static xce_job_alloc_t* make_dispatch_alloc(uint64_t jobid, const char* host, float value) { xce_alloc_t* res = (xce_alloc_t*)calloc(1, sizeof(*res)); xce_host_alloc_t* h = (xce_host_alloc_t*)calloc(1, sizeof(*h)); xce_host_alloc_t** hosts = (xce_host_alloc_t**)calloc(1, sizeof(*hosts)); xce_alloc_t** allocs = (xce_alloc_t**)calloc(1, sizeof(*allocs)); xce_job_alloc_t* j = (xce_job_alloc_t*)calloc(1, sizeof(*j)); TEST_ASSERT_NOT_NULL(res); TEST_ASSERT_NOT_NULL(h); TEST_ASSERT_NOT_NULL(hosts); TEST_ASSERT_NOT_NULL(allocs); TEST_ASSERT_NOT_NULL(j); res->resname = strdup("slots"); res->reserve = strdup("run"); res->instid = 1; res->value = value; res->from = 0; res->duration = 0; allocs[0] = res; h->host = strdup(host); h->nallocs = 1; h->allocs = allocs; hosts[0] = h; j->jobid = jobid; j->nhosts = 1; j->allocs = hosts; j->nres_shared = 0; j->allocs_shared = NULL; return j; } static xce_job_account_t* make_charge_account(const char* host, float value, int residx) { xce_job_account_t* acct = (xce_job_account_t*)calloc(1, sizeof(*acct)); TEST_ASSERT_NOT_NULL(acct); acct->nres_shared = 0; acct->accounts_shared = NULL; acct->nhosts = 1; acct->accounts = (xce_host_account_t**)calloc(1, sizeof(*acct->accounts)); TEST_ASSERT_NOT_NULL(acct->accounts); acct->accounts[0] = (xce_host_account_t*)calloc(1, sizeof(xce_host_account_t)); TEST_ASSERT_NOT_NULL(acct->accounts[0]); acct->accounts[0]->host = strdup(host); acct->accounts[0]->naccount = 1; acct->accounts[0]->accounts = (xce_res_account_t**)calloc(1, sizeof(*acct->accounts[0]->accounts)); TEST_ASSERT_NOT_NULL(acct->accounts[0]->accounts); acct->accounts[0]->accounts[0] = (xce_res_account_t*)calloc(1, sizeof(xce_res_account_t)); TEST_ASSERT_NOT_NULL(acct->accounts[0]->accounts[0]); acct->accounts[0]->accounts[0]->residx = residx; acct->accounts[0]->accounts[0]->used = value; acct->accounts[0]->accounts[0]->reserved = 0.0; return acct; } /* Real sched_check_host_satisfy()/sched_charge_account()/ * sched_commit_job_alloc()/make_dispatch_decision()/make_pend_decision() * round trip for ONE genuinely-live candidate -- shared by every real * round's allocator thread (round 1's live candidate J2, and every one of * J3-J12 in the churn phase). */ static xce_decision_t* decide_one_candidate(xce_scheduler_t* sched, xce_sched_plugin_t* picker, uint64_t jobid) { xce_sched_core_t* core = &(sched->core); xce_sched_job_t* job; xce_sched_worker_t* w; xvec_t uncomitted; xvec_idval_t wanted; xvec_idval_t unmeet; float want; int rc; xce_decision_t* decision; job = sched_get_job(core, jobid); w = sched_get_worker(core, (char*)WORKER_NAME); TEST_ASSERT_NOT_NULL(job); TEST_ASSERT_NOT_NULL(w); want = lookup_want(jobid); rc = xvec_init(&uncomitted, 1, sizeof(float)); TEST_ASSERT_TRUE(rc >= 0); rc = xvec_idval_init(&wanted, 1); TEST_ASSERT_EQUAL_INT(0, rc); rc = xvec_idval_store(&wanted, g_slots_residx, want); TEST_ASSERT_TRUE(rc >= 0); rc = sched_check_host_satisfy(sched, w, &uncomitted, &wanted, &unmeet); xvec_fini(&uncomitted); xvec_idval_fini(&wanted); xvec_idval_fini(&unmeet); if (rc == 1) { job->accounts = make_charge_account(WORKER_NAME, want, g_slots_residx); sched_commit_job_alloc(core, job); job->allocs = make_dispatch_alloc(jobid, WORKER_NAME, want); picker->picker_iface->on_dispatch(picker, jobid); decision = make_dispatch_decision(job->allocs); } else { picker->picker_iface->on_pend(picker, jobid); decision = make_pend_decision(make_pendreason(jobid)); } return decision; } /* ---------------------------------------------------------------------- */ /* Prelude allocator (scheduler-snapshot-job-allocation task 4.2, see this */ /* file's ADDENDUM header comment): dispatches PREEXIST_ID onto its own */ /* dedicated worker (WORKER2_NAME) via the exact same real */ /* sched_check_host_satisfy()/sched_charge_account()/sched_commit_job_ */ /* alloc()/make_dispatch_decision() round trip decide_one_candidate() */ /* above already uses -- a completely ordinary, unrigged live dispatch, no */ /* special-casing at all (unlike ANCHOR_ID's necessarily-rigged injection */ /* in round1_allocator_thread_fn below, which exists only because ANCHOR_ */ /* ID must be simultaneously RUN and carry a same-round DISPATCH decision, */ /* a shape no live picker/allocator can ever itself produce). This is a */ /* deliberately SEPARATE helper from decide_one_candidate() (which */ /* hardcodes WORKER_NAME) rather than a generalization of it, so this */ /* task's addition cannot perturb that already-validated function's */ /* behavior for ANCHOR_ID/J2-J12. Runs strictly BEFORE sched_record_start */ /* () (see the Prelude section of the test body below), so this dispatch */ /* decision is never written to the .replay file at all. */ /* ---------------------------------------------------------------------- */ static void* preexist_prelude_allocator_thread_fn(void* arg) { xce_scheduler_t* sched = (xce_scheduler_t*)arg; xce_sched_core_t* core = &(sched->core); xce_sched_plugin_t* picker = NULL; uint64_t jobid; if (sched_resolve_job_picker(core, &picker) < 0) { xce_decision_t* decision = NULL; cbuf_get(&(core->jbuf), &jobid); cbuf_put(&(core->decisions), &decision); return NULL; } for (;;) { xce_decision_t* decision; xce_sched_job_t* job; xce_sched_worker_t* w2; xvec_t uncomitted; xvec_idval_t wanted; xvec_idval_t unmeet; int rc; cbuf_get(&(core->jbuf), &jobid); if (jobid == 0) { decision = NULL; cbuf_put(&(core->decisions), &decision); break; } TEST_ASSERT_EQUAL_UINT64_MESSAGE( PREEXIST_ID, jobid, "the prelude round must only ever present PREEXIST_ID as a " "candidate -- nothing else has been pushed to the core yet at " "this point"); job = sched_get_job(core, jobid); w2 = sched_get_worker(core, (char*)WORKER2_NAME); TEST_ASSERT_NOT_NULL(job); TEST_ASSERT_NOT_NULL(w2); rc = xvec_init(&uncomitted, 1, sizeof(float)); TEST_ASSERT_TRUE(rc >= 0); rc = xvec_idval_init(&wanted, 1); TEST_ASSERT_EQUAL_INT(0, rc); rc = xvec_idval_store(&wanted, g_slots_residx, PREEXIST_WANT); TEST_ASSERT_TRUE(rc >= 0); rc = sched_check_host_satisfy(sched, w2, &uncomitted, &wanted, &unmeet); xvec_fini(&uncomitted); xvec_idval_fini(&wanted); xvec_idval_fini(&unmeet); TEST_ASSERT_EQUAL_INT_MESSAGE( 1, rc, "PREEXIST_ID's own dedicated worker must have ample capacity -- " "this is a genuine, unrigged live dispatch, not an engineered " "coincidence like ANCHOR_ID's"); job->accounts = make_charge_account(WORKER2_NAME, PREEXIST_WANT, g_slots_residx); sched_commit_job_alloc(core, job); job->allocs = make_dispatch_alloc(jobid, WORKER2_NAME, PREEXIST_WANT); picker->picker_iface->on_dispatch(picker, jobid); decision = make_dispatch_decision(job->allocs); cbuf_put(&(core->decisions), &decision); } return NULL; } /* ---------------------------------------------------------------------- */ /* Round 1's allocator: injects ANCHOR_ID's own already-resolved DISPATCH */ /* (see this file's header comment for the full reasoning), THEN */ /* processes this round's one genuinely-live candidate (J2) exactly like */ /* every other round. */ /* ---------------------------------------------------------------------- */ static void* round1_allocator_thread_fn(void* arg) { xce_scheduler_t* sched = (xce_scheduler_t*)arg; xce_sched_core_t* core = &(sched->core); xce_sched_plugin_t* picker = NULL; uint64_t jobid; int injected = 0; if (sched_resolve_job_picker(core, &picker) < 0) { xce_decision_t* decision = NULL; cbuf_get(&(core->jbuf), &jobid); cbuf_put(&(core->decisions), &decision); return NULL; } for (;;) { xce_decision_t* decision; cbuf_get(&(core->jbuf), &jobid); if (!injected) { /* Safe exactly here, not earlier: reaching this point proves * the picker thread already unblocked from core->wbuf (i.e. * xce_core_signal_go() already ran), which itself only * happens after THIS round's xce_core_begin_session()/ * sched_account_jobs() reset-then-resum pass has already * completed -- so charging ANCHOR_ID here can never race that * pass. Injecting any earlier (e.g. before this thread's * first cbuf_get) would race xce_core_begin_session()'s own * resInUse reset on another thread. */ xce_sched_job_t* anchor = sched_get_job(core, ANCHOR_ID); xce_decision_t* anchor_decision; TEST_ASSERT_NOT_NULL(anchor); TEST_ASSERT_TRUE_MESSAGE( JOB_IS_RUN(anchor->state), "ANCHOR_ID must already be RUN (via the real JOB_STATE_CHANGE " "event applied at this round's translate step) before this " "allocator ever runs -- the exact Recorded-Decision " "Reconciliation trigger shape"); TEST_ASSERT_NULL_MESSAGE( anchor->accounts, "ANCHOR_ID must have no allocation yet -- this thread is " "about to build the one it represents as already resolved " "in the original run"); anchor->accounts = make_charge_account(WORKER_NAME, ANCHOR_WANT, g_slots_residx); sched_commit_job_alloc(core, anchor); anchor->allocs = make_dispatch_alloc(ANCHOR_ID, WORKER_NAME, ANCHOR_WANT); anchor_decision = make_dispatch_decision(anchor->allocs); cbuf_put(&(core->decisions), &anchor_decision); injected = 1; } if (jobid == 0) { decision = NULL; cbuf_put(&(core->decisions), &decision); break; } decision = decide_one_candidate(sched, picker, jobid); cbuf_put(&(core->decisions), &decision); } return NULL; } /* Every subsequent round's allocator: 100% organic, no special-casing. */ static void* generic_allocator_thread_fn(void* arg) { xce_scheduler_t* sched = (xce_scheduler_t*)arg; xce_sched_core_t* core = &(sched->core); xce_sched_plugin_t* picker = NULL; uint64_t jobid; if (sched_resolve_job_picker(core, &picker) < 0) { xce_decision_t* decision = NULL; cbuf_get(&(core->jbuf), &jobid); cbuf_put(&(core->decisions), &decision); return NULL; } for (;;) { xce_decision_t* decision; cbuf_get(&(core->jbuf), &jobid); if (jobid == 0) { decision = NULL; cbuf_put(&(core->decisions), &decision); break; } decision = decide_one_candidate(sched, picker, jobid); cbuf_put(&(core->decisions), &decision); } return NULL; } static void run_round(xce_scheduler_t* sched, void* (*allocator_fn)(void*)) { pthread_t picker_tid; pthread_t allocator_tid; int rc; rc = pthread_create(&picker_tid, NULL, picker_thread_fn, &(sched->core)); TEST_ASSERT_EQUAL_INT(0, rc); rc = pthread_create(&allocator_tid, NULL, allocator_fn, sched); TEST_ASSERT_EQUAL_INT(0, rc); xce_sched_driver_run_normal_mode_round(sched); rc = pthread_join(picker_tid, NULL); TEST_ASSERT_EQUAL_INT(0, rc); rc = pthread_join(allocator_tid, NULL); TEST_ASSERT_EQUAL_INT(0, rc); } /* ---------------------------------------------------------------------- */ /* Rounds 2-24: table-driven churn -- 11 jobs (J2-J12) arriving, being */ /* denied/recurring, dispatching once capacity frees up, running, and */ /* eventually finishing, against the 8.0-unit headroom ANCHOR_ID's */ /* permanent 12.0-unit charge leaves out of the worker's 20.0-unit total. */ /* */ /* IMPORTANT, empirically confirmed while writing this test (and an exact */ /* live confirmation of this file's own header-comment structural proof): */ /* unlike ANCHOR_ID, NONE of J2-J12's real dispatches ever become visible */ /* via resInUse *during replay* -- each of them arrives PEND and is */ /* live-decided (PEND, sim's own picker being structurally PEND-only) in */ /* the SAME replayed round it arrives, then transitions to RUN via a */ /* JOB_STATE_CHANGE event in a LATER round whose own decoded ReplaySession */ /* does not (and structurally cannot) contain a matching DISPATCH entry --*/ /* reconciliation's own per-round search comes up empty for every one of */ /* them, forever (Requirement 4: reported as unavailable, not fabricated, */ /* exactly per design). This is the same reconciliation limitation this */ /* file's header comment already proves is structural, now independently */ /* confirmed by running the real replay rather than only by reading the */ /* source. It means resInUse during replay reflects ONLY ANCHOR_ID's own */ /* engineered-coincident 12.0-unit charge for the WHOLE 24-round replay -- */ /* which is exactly this task's own Observable Completion ("a job that */ /* stays running... shows nonzero... usage"), not a weaker result: the */ /* churn jobs still arrive, contend, dispatch-in-the-original-recording, */ /* and finish for real across all 24 rounds (proving the recording itself */ /* is a genuine, busy, multi-job session, not a toy single-job fixture), */ /* they just don't additionally move the resInUse needle during replay --*/ /* an honest, in-scope-documented limitation (task 4.1's own notes: "true */ /* dispatch-decision parity for freshly-arriving jobs is scheduler- */ /* allocation's... job, out of this feature's scope"), not a defect this */ /* test papers over. */ /* */ /* Hand-derived and cross-checked arithmetic for the RECORDING's own */ /* in-memory resInUse (i.e. what a normal-mode continuation of this exact */ /* session would see -- NOT what sim-mode replay of it shows, per above): */ /* R2 J2->RUN, J3 arrives(4) -> PEND (12+5+4>20) */ /* R3 (anchor rusage only) -> J3 still PEND */ /* R4 J2->DONE -> J3 DISPATCH (12+4=16) */ /* R5 J3->RUN, J4 arrives(3) -> DISPATCH (16+3=19) */ /* R6 J4->RUN, J5 arrives(2) -> PEND (19+2>20) */ /* R7 J3->DONE -> J5 DISPATCH (12+3+2=17) */ /* R8 J5->RUN, J6 arrives(3) -> DISPATCH (17+3=20) */ /* R9 J6->RUN, J7 arrives(1) -> PEND (20+1>20) */ /* R10 J4->DONE -> J7 DISPATCH (12+2+3+1=18) */ /* R11 J7->RUN, J8 arrives(2) -> DISPATCH (18+2=20) */ /* R12 J8->RUN, J9 arrives(4) -> PEND (20+4>20) */ /* R13 J5->DONE -> J9 still PEND (12+3+1+2+4>20) */ /* R14 J6->DONE -> J9 DISPATCH (12+1+2+4=19) */ /* R15 J9->RUN (no arrivals) -> no candidates */ /* R16 J7->DONE, J10 arrives(3)-> PEND (12+2+4+3>20) */ /* R17 J8->DONE -> J10 DISPATCH (12+4+3=19) */ /* R18 J10->RUN, J11 arrives(2)-> PEND (19+2>20) */ /* R19 J9->DONE -> J11 DISPATCH (12+3+2=17) */ /* R20 J11->RUN, J12 arrives(6)-> PEND (17+6>20) */ /* R21 J10->DONE -> J12 DISPATCH (12+2+6=20) */ /* R22 J12->RUN (no arrivals) -> no candidates */ /* R23 J11->DONE -> no candidates */ /* R24 J12->DONE (final) -> resInUse back to 12 (ANCHOR only) */ /* ---------------------------------------------------------------------- */ typedef struct { uint64_t confirm_run_jobid; uint64_t finish_jobid; uint64_t new_jobid; float new_want; int anchor_rusage; } round_step_t; static const round_step_t ROUNDS[] = { /* R2 */ { J2_ID, 0, J3_ID, 4.0f, 0 }, /* R3 */ { 0, 0, 0, 0.0f, 1 }, /* R4 */ { 0, J2_ID, 0, 0.0f, 0 }, /* R5 */ { J3_ID, 0, J4_ID, 3.0f, 0 }, /* R6 */ { J4_ID, 0, J5_ID, 2.0f, 1 }, /* R7 */ { 0, J3_ID, 0, 0.0f, 0 }, /* R8 */ { J5_ID, 0, J6_ID, 3.0f, 0 }, /* R9 */ { J6_ID, 0, J7_ID, 1.0f, 1 }, /* R10 */ { 0, J4_ID, 0, 0.0f, 0 }, /* R11 */ { J7_ID, 0, J8_ID, 2.0f, 1 }, /* R12 */ { J8_ID, 0, J9_ID, 4.0f, 0 }, /* R13 */ { 0, J5_ID, 0, 0.0f, 1 }, /* R14 */ { 0, J6_ID, 0, 0.0f, 0 }, /* R15 */ { J9_ID, 0, 0, 0.0f, 1 }, /* R16 */ { 0, J7_ID, J10_ID, 3.0f, 0 }, /* R17 */ { 0, J8_ID, 0, 0.0f, 1 }, /* R18 */ { J10_ID, 0, J11_ID, 2.0f, 0 }, /* R19 */ { 0, J9_ID, 0, 0.0f, 1 }, /* R20 */ { J11_ID, 0, J12_ID, 6.0f, 0 }, /* R21 */ { 0, J10_ID, 0, 0.0f, 1 }, /* R22 */ { J12_ID, 0, 0, 0.0f, 0 }, /* R23 */ { 0, J11_ID, 0, 0.0f, 1 }, /* R24 */ { 0, J12_ID, 0, 0.0f, 1 }, }; #define N_TABLE_ROUNDS ((int)(sizeof(ROUNDS) / sizeof(ROUNDS[0]))) #define N_TOTAL_ROUNDS (1 + N_TABLE_ROUNDS) /* round 1 + table rounds = 24 */ /* ---------------------------------------------------------------------- */ /* REPL live-verify: fork + freopen stdin/stdout, same technique */ /* test_repl_execution_control.c's/test_sim_verify.c's own capture helpers */ /* establish. */ /* ---------------------------------------------------------------------- */ static void write_text_file(const char* path, const char* text) { FILE* f = fopen(path, "w"); TEST_ASSERT_NOT_NULL(f); fputs(text, f); fclose(f); } static char* read_whole_file(const char* path) { FILE* f = fopen(path, "r"); char* buf; long len; TEST_ASSERT_NOT_NULL(f); TEST_ASSERT_EQUAL_INT(0, fseek(f, 0, SEEK_END)); len = ftell(f); TEST_ASSERT_TRUE(len >= 0); TEST_ASSERT_EQUAL_INT(0, fseek(f, 0, SEEK_SET)); buf = (char*)calloc(1, (size_t)len + 1); TEST_ASSERT_NOT_NULL(buf); if (len > 0) { TEST_ASSERT_EQUAL_INT((int)len, (int)fread(buf, 1, (size_t)len, f)); } fclose(f); return buf; } static void child_replay(const char* replay_path, const char* script_path, const char* capture_path) { xce_sched_sim_driver_t sim; sched_sim_args_t args; if (freopen(script_path, "r", stdin) == NULL) { _exit(120); } if (freopen(capture_path, "w", stdout) == NULL) { _exit(121); } memset(&sim, 0, sizeof(sim)); memset(&args, 0, sizeof(args)); args.replay_path = replay_path; args.sim_start_ts = SCHED_SIM_START_TS_LATEST; args.sim_speed = 0; /* frozen -- deterministic */ args.interactive = true; (void)sched_sim_driver_run(&sim, &args); fflush(stdout); _exit(111); /* unreachable -- the script's trailing `quit` must already * have called exit(0) */ } /* ---------------------------------------------------------------------- */ /* The end-to-end test. */ /* ---------------------------------------------------------------------- */ static void test_e2e_original_motivation_long_replay(void) { char record_dir[PATH_MAX]; xce_scheduler_t sched; char r_path[PATH_MAX]; char aux_dir[PATH_MAX]; char script_path[PATH_MAX]; char capture_path[PATH_MAX]; xce_job_t** anchor_jobs; xce_job_t** j2_jobs; xce_job_t** preexist_jobs; uint64_t sub_time; int rc; int i; pid_t pid; int status; char* captured; char anchor_jobid_str[64]; char anchor_marker[96]; char* p1; char* p2; char* p3; char* p4; char preexist_jobid_str[64]; char preexist_marker[96]; char* q1; char* q2; char* q3; char* q4; char* q5; /* ==== Build the recording: a real, driven 24-round session ==== */ make_tmp_dir(record_dir, sizeof(record_dir), "_rec"); fixture_init(&sched, record_dir); /* ---- Prelude (scheduler-snapshot-job-allocation task 4.2, see this * file's ADDENDUM header comment): PREEXIST_ID is submitted, * genuinely live-dispatched, and confirmed RUN entirely BEFORE * recording starts -- so it "predates the snapshot" exactly as * Requirement 1 describes, and its dispatch decision is never written * to the .replay file at all (nothing is recorded yet). ---- */ enqueue_queue_update(&sched, "normal", 1); preexist_jobs = (xce_job_t**)calloc(1, sizeof(xce_job_t*)); preexist_jobs[0] = make_job(PREEXIST_ID, 100ULL, "normal", "preexist-user"); enqueue_jobs_push(&sched, preexist_jobs, 1); run_round(&sched, preexist_prelude_allocator_thread_fn); { xce_sched_job_t* preexist = sched_get_job(&(sched.core), PREEXIST_ID); TEST_ASSERT_NOT_NULL(preexist); TEST_ASSERT_FALSE_MESSAGE( JOB_IS_RUN(preexist->state), "PREEXIST_ID must still be PEND immediately after its dispatch " "decision -- RUN confirmation is a separate step, exactly as it " "is for every other job in this file (J2-J12)"); TEST_ASSERT_NOT_NULL_MESSAGE( preexist->accounts, "the prelude's genuine live dispatch must have already set " "PREEXIST_ID's accounts"); } enqueue_jstate_change(&sched, PREEXIST_ID, JOB_STAT_RUN); run_round(&sched, generic_allocator_thread_fn); { xce_sched_job_t* preexist = sched_get_job(&(sched.core), PREEXIST_ID); TEST_ASSERT_NOT_NULL(preexist); TEST_ASSERT_TRUE_MESSAGE( JOB_IS_RUN(preexist->state), "PREEXIST_ID must now be RUN -- fully 'already running' before " "recording ever starts, matching Requirement 1's exact " "scenario"); } rc = sched_record_start(&sched, "scheduler-e2e-original-motivation.replay"); TEST_ASSERT_EQUAL_INT(0, rc); TEST_ASSERT_TRUE(strlen(sched.record_file_path) < sizeof(r_path)); strcpy(r_path, sched.record_file_path); /* ---- Round 1: ANCHOR_ID's genesis (arrival + RUN confirmation, both * via the real dataq/translate path) + J2's genuinely live dispatch. * No enqueue_queue_update() here (unlike before task 4.2's prelude was * added): the "normal" queue already exists as of the prelude above, * and is therefore already part of the initial SnapshotEntry * sched_record_start() just captured. Re-issuing an identical * QUEUE_UPDATE here would additionally get RECORDED into round 1 and * REPLAYED against that same snapshot-restored queue -- exercising * core_upsert_queue()'s "queue already exists -> update in place" * branch for a queue whose xce_sched_queue_t was built by * xce_sched_core_load_snapshot()'s own restore path rather than * core_build_queue() -- a genuine, pre-existing (latent, previously * uncovered by any test) defect in that combination, confirmed by * empirically triggering it (ASAN bad-free in core_upsert_queue, * core_events.c:336, freeing q->resreq) while first writing this * addendum. Fixing that defect is production code, out of this * test-only task's boundary (Requirement 1/2/3 do not depend on it); * simply not re-declaring an already-existing queue is both correct * (the queue's priority/status are unchanged either way) and sufficient * to keep this task's own addition from exercising unrelated, * out-of-boundary production code. */ register_want(ANCHOR_ID, ANCHOR_WANT); anchor_jobs = (xce_job_t**)calloc(1, sizeof(xce_job_t*)); anchor_jobs[0] = make_job(ANCHOR_ID, 500ULL, "normal", "anchor-user"); enqueue_jobs_push(&sched, anchor_jobs, 1); enqueue_jstate_change(&sched, ANCHOR_ID, JOB_STAT_RUN); register_want(J2_ID, 5.0f); j2_jobs = (xce_job_t**)calloc(1, sizeof(xce_job_t*)); j2_jobs[0] = make_job(J2_ID, 1000ULL, "normal", "churn-user"); enqueue_jobs_push(&sched, j2_jobs, 1); run_round(&sched, round1_allocator_thread_fn); /* ---- Rounds 2-24: real, driven churn/contention, table-driven. ---- */ sub_time = 2000ULL; for (i = 0; i < N_TABLE_ROUNDS; i++) { const round_step_t* s = &ROUNDS[i]; if (s->confirm_run_jobid != 0) { enqueue_jstate_change(&sched, s->confirm_run_jobid, JOB_STAT_RUN); } if (s->finish_jobid != 0) { enqueue_jstate_change(&sched, s->finish_jobid, JOB_STAT_DONE); } if (s->new_jobid != 0) { xce_job_t** jobs = (xce_job_t**)calloc(1, sizeof(xce_job_t*)); register_want(s->new_jobid, s->new_want); jobs[0] = make_job(s->new_jobid, sub_time, "normal", "churn-user"); sub_time += 1000ULL; enqueue_jobs_push(&sched, jobs, 1); } if (s->anchor_rusage) { enqueue_job_rusage(&sched, ANCHOR_ID, "cpu", (float)(1000 + i)); } run_round(&sched, generic_allocator_thread_fn); } rc = sched_record_stop(&sched); TEST_ASSERT_EQUAL_INT(0, rc); printf("test_e2e_original_motivation: recorded %d real rounds, %llu " "events, into %s\n", N_TOTAL_ROUNDS, (unsigned long long)sched.record_event_count, r_path); /* ==== Live-verify: replay the SAME file through the REAL, unmodified * interactive REPL, inspecting worker state and decision output at * multiple points across the whole 24-round replay. ==== */ make_tmp_dir(aux_dir, sizeof(aux_dir), "_replay"); snprintf(script_path, sizeof(script_path), "%s/script.txt", aux_dir); snprintf(capture_path, sizeof(capture_path), "%s/capture.out", aux_dir); /* 1 + 2 + 10 + 8 = 21 stepped rounds, then continue drains the final 3 * (22, 23, 24) to end of stream. The lone "step 1" first (splitting * off what used to be the first "step 3") gives PREEXIST_ID (task * 4.2) a checkpoint after just ONE round -- the earliest possible * observation point, strictly before ANCHOR_ID's own first checkpoint * at round 3 -- while the cumulative round counts at every other * checkpoint (3, 13, 21, 24) are unchanged from before this task. */ write_text_file(script_path, "step 1\n" "info worker worker-e2e-pre\n" "step 2\n" "info worker worker-e2e\n" "info worker worker-e2e-pre\n" "step 10\n" "info worker worker-e2e\n" "info worker worker-e2e-pre\n" "step 8\n" "info worker worker-e2e\n" "info worker worker-e2e-pre\n" "continue\n" "info worker worker-e2e\n" "info worker worker-e2e-pre\n" "quit\n"); pid = fork(); TEST_ASSERT_TRUE_MESSAGE(pid >= 0, "fork() must succeed"); if (pid == 0) { child_replay(r_path, script_path, capture_path); _exit(112); /* unreachable */ } TEST_ASSERT_EQUAL_INT_MESSAGE(pid, waitpid(pid, &status, 0), "waitpid must reap the REPL child"); TEST_ASSERT_TRUE_MESSAGE(WIFEXITED(status), "the REPL child must exit normally via quit's " "exit(0), not crash/signal"); TEST_ASSERT_EQUAL_INT_MESSAGE(0, WEXITSTATUS(status), "quit's exit(0) is the only correct exit " "path"); captured = read_whole_file(capture_path); /* ---- Observable #1 (Req 3.1-3.3): ANCHOR_ID's DISPATCH decision -- * the exact thing bug-1611 documented as NEVER appearing anywhere in * a whole replay -- is now visible in step/continue output. */ fmt_jobid(anchor_jobid_str, sizeof(anchor_jobid_str), ANCHOR_ID); snprintf(anchor_marker, sizeof(anchor_marker), "[sim] DISPATCH jobid=%s", anchor_jobid_str); TEST_ASSERT_NOT_NULL_MESSAGE( strstr(captured, anchor_marker), "ANCHOR_ID's originally-resolved DISPATCH must now be visible in " "step/continue output -- this is bug-1611's own symptom, fixed"); /* ---- Observable #2 (Req 1.1-1.3): "info worker" reports ANCHOR_ID's * own nonzero used= value at FOUR checkpoints spanning the whole * 24-round replay, IN ORDER -- not merely "nonzero at some point". * Per this file's round-table comment above, ONLY ANCHOR_ID's charge * is ever reconciled during replay (every churn job's own dispatch * falls outside reconciliation's same-round matching window and is * correctly left unavailable, Requirement 4) -- so the arithmetically * correct value at all four checkpoints is a constant 12.00. Before * this feature's fix (bug-1613), this would have read 0.00 at every * single one of these checkpoints, for the entire run, regardless of * round count -- sched_account_jobs() was never even called in sim * mode. A flat, correct, *persisting* 12.00 across 24 real rounds is * exactly the "job that stays running... shows nonzero usage" * Observable Completion this task asks for, proven at more than one * point, not merely once. */ p1 = strstr(captured, "used=12.00"); TEST_ASSERT_NOT_NULL_MESSAGE( p1, "after 3 rounds: ANCHOR_ID's 12.0-unit charge must already be " "visible"); p2 = strstr(p1, "used=12.00"); TEST_ASSERT_NOT_NULL_MESSAGE( p2, "after 13 rounds: ANCHOR_ID's charge must still be correctly " "re-derived every round, not just transiently at the start"); p3 = strstr(p2, "used=12.00"); TEST_ASSERT_NOT_NULL_MESSAGE( p3, "after 21 rounds: ANCHOR_ID's charge must still hold, deep into " "the replay"); p4 = strstr(p3, "used=12.00"); TEST_ASSERT_NOT_NULL_MESSAGE( p4, "after all 24 rounds: ANCHOR_ID's charge must still be exactly " "12.00 at the very end of the replay -- nothing more (no " "fabricated churn-job charge) and nothing less (no silent " "reset back to zero)"); /* Non-tautology guard: confirm "used=0.00" for the slots resource never * appears anywhere in the captured output -- if reconciliation had * silently regressed to its pre-fix no-op, every one of the four * checkpoints above would show 0.00 instead, and this file's own * discriminating-power check (documented in the implementer's status * report, not re-run automatically here) confirms this exact failure * mode when the reconciliation call site is reverted. */ TEST_ASSERT_NULL_MESSAGE( strstr(captured, "resource slots: value=32.00 used=0.00"), "the slots resource must never read back to used=0.00 once " "ANCHOR_ID has been reconciled -- that would be bug-1613's exact " "pre-fix symptom"); /* ---- Observable #3 (Req 1.1-1.3, scheduler-snapshot-job-allocation * task 4.2 -- see this file's ADDENDUM header comment): PREEXIST_ID's * allocation is correct from the FIRST possible checkpoint (after just * ONE replayed round), strictly earlier than ANCHOR_ID's own first * checkpoint above, and stays correct through all 24 rounds. Unlike * ANCHOR_ID, PREEXIST_ID's dispatch decision predates the recording * entirely and can never be discovered by replay-based reconciliation * (sim_reconcile_unaccounted_jobs()) -- its correctness here can ONLY * come from xce_sched_core_load_snapshot()'s own allocation * restoration (task 2.1's build_wire_job() save-side wiring + task * 2.2's dup_job_alloc()/make_job_account() load-side wiring), proven * concretely by the DISPATCH-never-appears guard below. Before this * feature's fix, job->allocs/job->accounts were never restored at * all (snapshot_wire.c's own former "not captured" comment, * snapshot.c's former Scope Decisions bullet) -- this checkpoint * would have read 0.00 at every single one of these five points, for * the entire run, with no mechanism (not even reconciliation) able to * ever recover it, since no DISPATCH decision for this job exists * anywhere in the replay stream to reconcile against. */ q1 = strstr(captured, "used=9.00"); TEST_ASSERT_NOT_NULL_MESSAGE( q1, "after just 1 round: PREEXIST_ID's 9.0-unit charge, restored " "purely by loading the recording's own initial snapshot, must " "already be visible -- no dispatch decision for this job " "exists anywhere in the recording for reconciliation to ever " "find"); q2 = strstr(q1, "used=9.00"); TEST_ASSERT_NOT_NULL_MESSAGE( q2, "after 3 rounds: PREEXIST_ID's charge must still hold"); q3 = strstr(q2, "used=9.00"); TEST_ASSERT_NOT_NULL_MESSAGE( q3, "after 13 rounds: PREEXIST_ID's charge must still hold"); q4 = strstr(q3, "used=9.00"); TEST_ASSERT_NOT_NULL_MESSAGE( q4, "after 21 rounds: PREEXIST_ID's charge must still hold"); q5 = strstr(q4, "used=9.00"); TEST_ASSERT_NOT_NULL_MESSAGE( q5, "after all 24 rounds: PREEXIST_ID's charge must still be " "exactly 9.00 at the very end of the replay -- nothing more " "and nothing less"); /* Concrete proof of mechanism: PREEXIST_ID's own DISPATCH decision * must NEVER appear anywhere in the captured replay output -- its * dispatch happened strictly before sched_record_start() was ever * called, so nothing about it was ever written to the .replay file. * If this string DID appear, it would mean the test's own prelude * leaked into the recording (a test-authoring bug), not a genuine * proof of the snapshot-restoration mechanism under test. */ fmt_jobid(preexist_jobid_str, sizeof(preexist_jobid_str), PREEXIST_ID); snprintf(preexist_marker, sizeof(preexist_marker), "[sim] DISPATCH jobid=%s", preexist_jobid_str); TEST_ASSERT_NULL_MESSAGE( strstr(captured, preexist_marker), "PREEXIST_ID's dispatch predates the whole recording -- it must " "NEVER appear as a DISPATCH decision anywhere in replay output; " "its correct accounting comes solely from snapshot-load " "restoration, not from any replay-discoverable decision"); /* Non-tautology guard, symmetric to ANCHOR_ID's own above: this is * exactly the failure signature task 2.2's own load-side restoration * (dup_job_alloc()/make_job_account() in xce_sched_core_load_ * snapshot()) would produce if it were reverted -- PREEXIST_ID's * allocation predates the snapshot and has no other recovery path. */ TEST_ASSERT_NULL_MESSAGE( strstr(captured, "resource slots: value=15.00 used=0.00"), "worker-e2e-pre's slots resource must never read back to " "used=0.00 -- PREEXIST_ID's allocation has no recovery path other " "than snapshot-load restoration, since no DISPATCH decision for " "it exists anywhere in the replay stream"); free(captured); printf("test_e2e_original_motivation_long_replay: PASS\n"); } int main(void) { UNITY_BEGIN(); RUN_TEST(test_e2e_original_motivation_long_replay); return UNITY_END(); }