fix(scheduler): cache hits keep the full pend reason; sim uses the cache (bug-1932)

The session-level unsatisfiable-resreq cache stored only the exhausted
attempt's main reason (an int). The allocators record shortfalls as
per-host/per-resource entries and leave main_reason at 0, so every cache
hit in the live scheduler pended its job with an empty reason. The
simulator registered no cache at all (its table lived on xce_scheduler_t),
so it searched every identical pending job again and its traces differed
from production.

- The cache table moves to xce_sched_core_t; cache_lookup/insert/clear and
  the hook adapters take the core, with no cast to a driver struct.
  sched_init() builds it, cache_clear() rebuilds it per session,
  xce_sched_core_fini() destroys it.
- A cache entry is a deep copy of the attempt's full xce_job_preason_t
  plus the job that proved the request unsatisfiable (core->alloc_jobid,
  set by sched_allocator()). A hit moves a copy into ctx->reason; a miss
  leaves ctx->reason untouched.
- main.c passes the same three adapters to the simulator through new
  sched_sim_args_t fields; sched_sim_driver_start() sets them on the sim
  core (plain field writes: white-box tests include sim.c alone).
- Decision trace: a hit renders "no plan generated: skipped, same request
  as <job> proved unsatisfiable earlier this session" (new
  XCE_TRACE_NOPLAN_KNOWN_UNSAT via sched_trace_known_unsat()); the outcome
  shows the restored reasons.
- sim_verify keeps its main-only PEND compare (old recordings carry
  main-only hits); comments and the scheduler-sim-live-allocation design
  note updated.

Tests: full-reason assertions in the cache dispatch tests (RED before:
"a cache hit must restore the original attempt's FULL pend reason"),
test_alloc_cache.c ported to the new API (deep copy, clear frees, -EEXIST
frees), a sim-registered clear test, a renderer test, and a sim e2e
scenario with two identical unsatisfiable jobs (204, 205) where 205 is
skipped with 204's full reasons (RED before: no skip line).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-09 08:02:18 +00:00
parent cff3304f5d
commit 0317c7af98
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
27 changed files with 982 additions and 737 deletions

View file

@ -386,6 +386,9 @@ static uint64_t array_elem(uint32_t k)
* 203 selects mem>20000 (w01 mismatches), pended by the
* cluster-wide limit (6 of 6 in use)
* 3000 after 201 and both elements finish: 202 and 203 are tried again
* 4000 after 202 and 203 finish: 204 and 205 (identical requests, more
* mem than any host has; bug-1932) pend in every session -- 204
* by a full search, 205 by the unsatisfiable-request cache
*/
void write_extra_replay(const char* dest)
{
@ -393,7 +396,7 @@ void write_extra_replay(const char* dest)
char path[PATH_MAX];
FILE* f;
xce_worker_t* workers[2];
xce_job_t* jobs[5];
xce_job_t* jobs[7];
snprintf(record_dir, sizeof(record_dir), "/tmp/sched_tre2e_gen_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(record_dir));
@ -420,7 +423,9 @@ void write_extra_replay(const char* dest)
1, 1);
jobs[4] = make_push_job(203, 1104, "select[mem>20000] rusage[mem=512]",
"alice", 1, 1);
append_jobs(f, 1003ULL, jobs, 5);
jobs[5] = make_push_job(204, 1105, "rusage[mem=999999]", "carol", 1, 1);
jobs[6] = make_push_job(205, 1106, "rusage[mem=999999]", "carol", 1, 1);
append_jobs(f, 1003ULL, jobs, 7);
append_session(f, 2000ULL);
append_state_change(f, 2500ULL, 201, JOB_STAT_RUN, JOB_STAT_DONE);
@ -429,6 +434,9 @@ void write_extra_replay(const char* dest)
append_state_change(f, 2700ULL, array_elem(2), JOB_STAT_RUN,
JOB_STAT_DONE);
append_session(f, 3000ULL);
append_state_change(f, 3500ULL, 202, JOB_STAT_RUN, JOB_STAT_DONE);
append_state_change(f, 3600ULL, 203, JOB_STAT_RUN, JOB_STAT_DONE);
append_session(f, 4000ULL);
append_end_of_stream(f);
TEST_ASSERT_EQUAL_INT(0, fclose(f));