lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): sim mode picker never learns about the initial snapshot's own jobs/queues (bug-1567)

Browse source 
xce_scheduler -i -s <replay>: WARN "failed to resolve job picker
plugin (rc=-2)" on load, and every job already present in the
replay file's INITIAL snapshot stayed PEND forever -- never even
reached the picker, let alone got dispatched.

Root cause: sched_sim_driver_start() (sim.c) loads the initial
snapshot via xce_sched_core_load_snapshot() BEFORE any picker
plugin is registered (sim.h's own documented "Known scope
boundary" -- only the picker's NAME is threaded through from the
ReplayHeader). sched_resolve_job_picker() therefore always returns
-ENOENT there, and xce_sched_core_load_snapshot()'s own
picker-rebuild walk (on_queue_update/on_job_push per queue/job) is
silently skipped. The real builtin_fcfs picker only gets registered
moments later in sched_sim_driver_run() -- but nothing then replayed
that rebuild walk, so any job/queue from the INITIAL snapshot (as
opposed to one arriving later via a ReplayEvent, which correctly
reaches the picker through xce_core_push_events()'s own picker!=NULL
check) was permanently invisible to next_job().

Fixed by a new file, sim_picker.c: sim_bootstrap_picker() (replacing
sim_loop.c's sim_register_default_picker(), moved out because
sim_loop.c was already at its own documented 800-line "zero
line-budget headroom" cap) registers the picker AND then replays the
missed rebuild walk against it. Wired into scheduler_srcs,
sched_sim_isolation_lib, and all 9 white-box test TUs that #include
sim_loop.c directly.

New regression test (test_sim_event_loop.c): a job baked into a
fixture's INITIAL snapshot (not a ReplayEvent) now gets a live PEND
decision. RED confirmed by temporarily no-op'ing the rebuild walk
(test failed as expected), GREEN with the real fix. Full scheduler
suite green (41/41, excluding the pre-existing unrelated bug-1560).

Note: sim mode still never DISPATCHes (only PENDs) -- that is a
separate, intentionally out-of-scope gap (real resource-matching
allocation belongs to the not-yet-designed scheduler-allocation
spec), not addressed here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-17 10:21:11 +00:00
parent 0f1df4998e
commit 0c8e417bda
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
14 changed files with 395 additions and 93 deletions
Download patch file Download diff file Expand all files Collapse all files

177
src/scheduler/testsuite/test_sim_event_loop.c
View file

@ -82,6 +82,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <unity/unity.h>
@ -107,6 +108,7 @@
#include "../src/repl_params.c"
#include "../src/sim.c"
#include "../src/sim_loop.c"
#include "../src/sim_picker.c" /* bug-1567: sched_sim_driver_run() now calls sim_bootstrap_picker() (sim_loop.c-cap sibling) as an extern -- undefined reference at link time otherwise */
#include "../src/sim_record.c"
#include "../src/sim_verify.c"
#include "../src/snapshot.c"
@ -551,6 +553,179 @@ test_run_processes_multiple_sessions_in_order_and_no_longer_returns_enosys(void)
"returns_enosys: PASS\n");
}
/* ---------------------------------------------------------------------- */
/* Test 3 (bug-1567 regression): a job/queue already present in the */
/* INITIAL snapshot -- as opposed to arriving later via a ReplayEvent -- */
/* must still reach the real picker and get a live decision. */
/* ---------------------------------------------------------------------- */
/* Same fixture technique as write_header_and_snapshot() above, but (a)
* records schedPolicy="builtin_fcfs" (test_sim_verify.c's own
* build_fixture_scheduler() precedent -- this file's own copy deliberately
* uses a non-matching stub name instead, so it cannot be reused here
* unmodified) so sim_bootstrap_picker() (sim_picker.c) registers the REAL
* builtin_fcfs plugin, not a no-op stub, and (b) adds one real queue and
* one real PEND job directly to core->queues/core->jobs (core_build_queue()/
* core_build_job() + sched_add_queue()/sched_add_job(), core_events.c,
* already white-box #included above) BEFORE calling sched_record_start() --
* so xce_sched_core_save_snapshot() (snapshot.c) captures both into the
* INITIAL snapshot itself, not as a later ReplayEvent. */
static FILE* write_header_and_snapshot_with_pending_job(char* out_path,
size_t out_path_len,
const char* suffix,
const char* queue_name,
uint64_t jobid)
{
char record_dir[PATH_MAX];
xce_scheduler_t sched;
xce_queue_t wire_queue;
xce_sched_queue_t* q;
job_submit_data_t submit;
xce_job_t wire_job;
xce_sched_job_t* job;
int rc;
make_tmp_dir(record_dir, sizeof(record_dir), suffix);
memset(&sched, 0, sizeof(sched));
mylock_init(&(sched.record_lock));
rc = xce_sched_core_init(&(sched.core));
TEST_ASSERT_EQUAL_INT(0, rc);
sched.core.job_pick_plugin = (char*)"builtin_fcfs";
sched.record_dir = record_dir;
memset(&wire_queue, 0, sizeof(wire_queue));
wire_queue.queue = (char*)queue_name;
wire_queue.status = QUEUE_STAT_OPEN;
wire_queue.priority = 1;
q = core_build_queue(&wire_queue);
TEST_ASSERT_NOT_NULL(q);
TEST_ASSERT_GREATER_OR_EQUAL(0, sched_add_queue(&(sched.core), q));
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)queue_name;
submit.user = (char*)"alice";
submit.mintasks = 1;
submit.maxtasks = 1;
memset(&wire_job, 0, sizeof(wire_job));
wire_job.jobid = jobid;
wire_job.state = JOB_STAT_PEND;
wire_job.submit = &submit;
job = core_build_job(&(sched.core), &wire_job);
TEST_ASSERT_NOT_NULL(job);
TEST_ASSERT_GREATER_OR_EQUAL(0, sched_add_job(&(sched.core), job));
rc = sched_record_start(&sched, "fixture.replay");
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_TRUE(strlen(sched.record_file_path) < out_path_len);
strcpy(out_path, sched.record_file_path);
return sched.record_file;
}
/* Same dup()/dup2() STDOUT_FILENO capture technique test_sim_verify.c's own
* run_sim_and_capture_stdout() establishes (this file's white-box TU does
* not #include that file, so a plain copy is needed here rather than a
* cross-file static call). */
static char* run_sim_and_capture_stdout(const char* path, int* out_rc)
{
sched_sim_args_t args;
xce_sched_sim_driver_t sim;
char tmp[] = "/tmp/xce_simloop_run_XXXXXX";
int fd = mkstemp(tmp);
int saved_stdout;
off_t len;
char* buf;
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "mkstemp for stdout capture must "
"succeed");
fflush(stdout);
saved_stdout = dup(STDOUT_FILENO);
TEST_ASSERT_TRUE_MESSAGE(saved_stdout >= 0, "dup(STDOUT_FILENO) must "
"succeed");
TEST_ASSERT_TRUE_MESSAGE(dup2(fd, STDOUT_FILENO) >= 0,
"dup2 redirect of STDOUT_FILENO must succeed");
memset(&args, 0, sizeof(args));
args.replay_path = path;
args.sim_start_ts = SCHED_SIM_START_TS_LATEST;
args.sim_speed = 0; /* frozen -- deterministic, no wall-clock wait */
*out_rc = sched_sim_driver_run(&sim, &args);
fflush(stdout);
dup2(saved_stdout, STDOUT_FILENO);
close(saved_stdout);
len = lseek(fd, 0, SEEK_CUR);
TEST_ASSERT_TRUE(len >= 0);
buf = (char*)calloc(1, (size_t)len + 1);
TEST_ASSERT_NOT_NULL(buf);
TEST_ASSERT_TRUE(lseek(fd, 0, SEEK_SET) == 0);
if (len > 0) {
ssize_t nread = read(fd, buf, (size_t)len);
TEST_ASSERT_TRUE(nread == len);
}
close(fd);
unlink(tmp);
return buf;
}
/*
* bug-1567: before this fix, sched_sim_driver_start() (sim.c) loaded the
* initial snapshot while no picker plugin was registered yet (sim.h's own
* "Known scope boundary"), so xce_sched_core_load_snapshot()'s
* picker-rebuild walk (on_queue_update/on_job_push) was skipped for the
* job/queue this fixture bakes into the INITIAL snapshot -- and nothing
* ever replayed that walk once sim_bootstrap_picker() registered the real
* picker moments later. The job was therefore invisible to next_job()
* forever: no PEND, no DISPATCH, nothing -- exactly the user report this
* bug tracks ("no job will be scheduled and all jobs in pend state").
* After the fix, sim_bootstrap_picker() (sim_picker.c) replays that walk
* itself, so this job -- despite never appearing in a single ReplayEvent
* -- still reaches the picker and gets a live PEND decision (this driver
* never DISPATCHes at all, a separate, intentionally out-of-scope gap --
* see sim_picker_thread_fn()'s own doc comment, sim_loop.c).
*/
static void
test_job_present_only_in_initial_snapshot_still_reaches_the_picker(void)
{
char path[PATH_MAX];
FILE* f;
int rc;
char* captured;
f = write_header_and_snapshot_with_pending_job(path, sizeof(path),
"_initsnap", "normal",
4242ULL);
TEST_ASSERT_NOT_NULL(f);
/* One session boundary is enough to trigger one scheduling round --
* the job under test arrives via the INITIAL SNAPSHOT, not via any
* ReplayEvent appended here (deliberately: zero jobs/queues events in
* this stream is the whole point of this regression). */
append_replay_session(f, 1000ULL);
append_end_of_stream_marker(f);
fclose(f);
captured = run_sim_and_capture_stdout(path, &rc);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL_MESSAGE(
strstr(captured, "PEND jobid=4242"),
"a job present only in the replay file's INITIAL snapshot must "
"still reach the real picker and receive a live PEND decision "
"(bug-1567)");
free(captured);
printf("test_job_present_only_in_initial_snapshot_still_reaches_the_"
"picker: PASS\n");
}
int main(void)
{
UNITY_BEGIN();
@ -559,6 +734,8 @@ int main(void)
test_process_replay_event_advances_virtual_clock_to_exact_event_ts_in_file_order);
RUN_TEST(
test_run_processes_multiple_sessions_in_order_and_no_longer_returns_enosys);
RUN_TEST(
test_job_present_only_in_initial_snapshot_still_reaches_the_picker);
return UNITY_END();
}