feat(scheduler-sim-live-allocation): wire simulation mode onto shared picker/allocator (task 4.1)

Replaces sim_picker_thread_fn() (the always-PEND placeholder) with the
real sched_job_picker()/sched_allocator() threads (task 3.1), spawned
against &(sim->core) in sched_sim_driver_run(). Simulation mode now
genuinely uses the jbuf capacity-1 rendezvous for the first time
(confirmed still exactly one producer/one consumer). Adds
sched_election.c to sched_sim_isolation_lib.

Fixes bug-1829 (blocking, only reachable once sim_allocator() actually
called sched_make_allocation() for the first time): core->seqjob_
allocator/parjob_allocator were never bootstrapped for simulation mode
at all. Fixed via new sim_alloc.c (sim_bootstrap_seqjob_allocator()/
sim_bootstrap_parjob_allocator()), mirroring the sim_picker.c/
sim_limit.c precedent.

That fix's own workaround -- a throwaway xce_scheduler_t wrapper
satisfying seqjob/parjob's init_fn -- has a documented, narrow side
effect: prepare_allocation_hook fan-out during a live seqjob/parjob
dispatch attempt reaches the wrapper's own separate plugin registry,
not sim->core.plugins, so a plugin registered only against the real
sim core (the limit plugin) isn't reached for host-scoped matching
during a live dispatch yet. Documented in design.md's new "Known Seam
Gap" section; task 4.3 added to tasks.md to close it before task 5.2's
acceptance test can pass for the host-scoped case.

TDD: new end-to-end test drives a real .replay with an empty decisions
array (ruling out reconciliation) through the real thread pair,
proving a genuine DISPATCH from live resource availability. 25
pre-existing white-box sim/repl test files given link-satisfying
stubs reproducing the OLD always-PEND contract unchanged -- their own
assertions are untouched (task 5.3's job to reconcile deeper).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-19 04:50:40 +00:00
parent 4195331b47
commit 08df64930b
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
33 changed files with 3809 additions and 92 deletions

View file

@ -22509,6 +22509,40 @@
],
"occurrences": 1,
"last_seen": "2026-09-18T15:14:00Z"
},
{
"id": "bug-1829",
"timestamp": "2026-09-19T04:00:00Z",
"error_message": "AddressSanitizer: SEGV (NULL pointer read) in sched_make_seqjob_allocation (src/scheduler/src/scheduler.c:59, `plugin->job_alloc_fn` where `plugin = core->seqjob_allocator` is NULL), called from sched_make_primary_alloc_ -> sched_make_primary_allocation -> sched_make_allocation -> sched_allocator (sched_election.c) -> the real pthread spawned by sched_sim_driver_run() (sim_loop.c). Discovered while writing the mandated TDD test for scheduler-sim-live-allocation task 4.1 (its own Observable: 'a simulation-mode session ... produces a genuine dispatch or pend decision for a job based on its actual resource availability, instead of unconditionally pending it') -- the crash is 100% reproducible for ANY job reaching primary resource allocation in simulation mode, regardless of resreq shape or dispatch/pend outcome (sched_make_parjob_allocation has the identical NULL-core->parjob_allocator crash for maxTasks>1 jobs).",
"root_cause": "core->seqjob_allocator/core->parjob_allocator (xce_sched_core_t fields, core.h) are never populated for the simulation driver, anywhere. Normal mode populates them via sched_init()'s sched_register_plugins() -> register_seqjob_allocator()/register_parjob_allocator() (plugins/seqjob_allocator.c, plugins/parjob_allocator.c), whose init_fn does `((xce_scheduler_t*)plugin->sched)->core.seqjob_allocator = plugin` -- REQUIRING a genuine, live xce_scheduler_t*. sched_sim_driver_start() (sim.c) registers no plugin at all (sim.h's own documented 'Known scope boundary'), and xce_sched_sim_driver_t deliberately has no xce_scheduler_t*/xce_scheduler_t field of its own (Req 5 AC 17, simulation isolation). This is NOT limited to the init_fn: every one of these two plugins' RUNTIME hooks (seqjob_prepare_allocation()/my_alloc(), parjob_prepare_allocation()/my_alloc(), and several internal static helpers taking xce_scheduler_t* sched as a parameter) ALSO read `plugin->sched->core.*` (resmgr for the 'slots'/rusage lookups, workers for host_iter, and sched_make_allocs()/sched_try_alloc_on_host_() in scheduler.c/resource.c which ALSO take xce_scheduler_t*/need `sched->core.resmgr`) on every single allocation attempt, not just at registration. This gap was already discovered and explicitly deferred by scheduler-sim-live-allocation task 3.1 (see test_sched_election_sim_fixture.c's own header comment: 'a pre-existing, tracked, OUT-OF-SCOPE gap ... NOT one of task 3.1's five permitted changes -- fixing it would be scope creep into a different, not-yet-assigned task') -- task 3.1's own fixture never reached this crash only because IT registers seqjob_allocator directly against a real, throwaway xce_scheduler_t wrapper whose OWN .core IS the core under test (single-core fixture), a shape simulation mode itself can never have (its core lives inside xce_sched_sim_driver_t, never xce_scheduler_t). Task 4.1 is the first task whose OWN production change (spawning the real sched_job_picker()/sched_allocator() thread pair against simulation mode's own core, replacing the old always-PEND sim_picker_thread_fn() placeholder that never called sched_make_allocation() at all) makes this code path reachable in simulation mode for the first time ever.",
"fix": "PARTIALLY FIXED (2026-09-19, scheduler-sim-live-allocation task 4.1, within that task's own file boundary -- a full fix is tracked as NOT YET DONE, see below). New file src/scheduler/src/sim_alloc.c (added to scheduler_srcs and sched_sim_isolation_lib, meson.build) provides sim_bootstrap_seqjob_allocator()/sim_bootstrap_parjob_allocator(), called from sched_sim_driver_run() (sim_loop.c) right after the existing sim_bootstrap_picker()/sim_bootstrap_limit_plugin() calls (same ordering constraint, same 'extern declared where consumed' idiom). Each builds a long-lived (heap-allocated, never freed -- same accepted-leak convention this file family already uses for its own never-joined threads), throwaway xce_scheduler_t wrapper, points its .core.resmgr/.core.workers fields (both `var`, core.h -- plain assignable rtti handles, not embedded values) at simulation mode's OWN resmgr/workers objects BEFORE calling the real, unmodified register_seqjob_allocator()/register_parjob_allocator() against it, then copies the resulting plugin pointer onto sim->core.seqjob_allocator/parjob_allocator. Because resmgr/workers are shared BY REFERENCE, any resource/worker registered later against the real sim->core (via the genuine event-ingest path) is immediately visible through the wrapper's aliased fields too, so seqjob_prepare_allocation()/my_alloc() see simulation mode's REAL, live resource/worker state -- not an empty, disconnected copy. Verified live: test_sim_live_allocation.c (new dedicated fixture, real scheduler_srcs link, real WORKERS_PUSH/RESOURCES_PUSH/JOBS_PUSH replay events, real sched_sim_driver_run() call, static-storage xce_sched_sim_driver_t) drives a job with a satisfiable resource want through the full real chain and observes a genuine `[sim] DISPATCH jobid=...` -- previously impossible (would have crashed) before this fix, PEND-only after reverting to sim_picker_thread_fn() at HEAD (confirmed RED). Full scheduler suite: 81/81, zero regressions (25 pre-existing white-box sim tests that #include sim_loop.c directly needed matching link-satisfying no-op/PEND-only stubs for the two new bootstrap functions and for the sched_job_picker()/sched_allocator() symbols sim_loop.c now calls directly instead of its own removed sim_picker_thread_fn() -- none of those 25 files test genuine resource-based allocation, so behavior-preserving stubs keep their existing assertions valid unchanged). NOT YET FIXED (tracked follow-up, explicitly out of task 4.1's file boundary): each wrapper's own core.plugins registry is separate from sim->core.plugins, so a plugin registered only against sim->core (e.g. the limit plugin via sim_bootstrap_limit_plugin()) is never reached by plugin_prepare_allocation()'s fan-out when it runs inside seqjob/parjob's own call chain rooted at the wrapper's core -- limit-plugin host-scoped matching under a REAL simulation-mode dispatch is not yet wired end-to-end (task 4.2/5.2's own territory, which specifically covers limit-usage parity under live sim-mode dispatch -- re-verify this exact scenario when implementing those). The FULL, root-cause fix -- retyping seqjob_allocator.c's/parjob_allocator.c's runtime hooks and internal helpers from `plugin->sched`/`xce_scheduler_t* sched` parameters to the already-available `plugin->core`/`xce_sched_core_t*` (task 1.2's own established pattern, mirroring task 2.3's limit.c migration) -- was NOT attempted: those hooks also call sched_make_allocs() (scheduler.c) and sched_try_alloc_on_host_() (resource.c) with the same xce_scheduler_t*, and scheduler.c/resource.c were explicitly out of task 4.1's file boundary (already migrated/committed by earlier tasks in this spec). A future task should do this properly rather than layering more wrapper-aliasing tricks.",
"files": [
"src/scheduler/src/sim_alloc.c (new file)",
"src/scheduler/src/sim_loop.c (sched_sim_driver_run() call sites)",
"src/scheduler/meson.build (scheduler_srcs, sched_sim_isolation_lib)",
"src/scheduler/testsuite/test_sim_live_allocation.c (new file, RED/GREEN proof)",
"src/scheduler/src/plugins/seqjob_allocator.c (root cause, not modified)",
"src/scheduler/src/plugins/parjob_allocator.c (root cause, not modified)"
],
"tags": [
"scheduler",
"sim-mode",
"asan",
"segv",
"null-deref",
"seqjob_allocator",
"parjob_allocator",
"plugin-sched-vs-plugin-core",
"scheduler-sim-live-allocation",
"task-4.1",
"partially-fixed",
"follow-up-tracked"
],
"related_bugs": [
"bug-1828"
],
"occurrences": 1,
"last_seen": "2026-09-19T04:00:00Z"
}
]
}