feat(scheduler-sim-live-allocation): close host-scoped limit fan-out seam gap (task 4.3)

Fully closes bug-1829's own tracked follow-up from task 4.1: retypes
seqjob_allocator.c/parjob_allocator.c's init_fn AND runtime hooks
(prepare_allocation_hook, job_alloc_fn) to read the plugin's own
honest plugin->core field (task 1.2) instead of deriving an
xce_scheduler_t* from plugin->sched -- mirroring task 2.3's limit.c
migration exactly. The gap was broader than initially scoped: not
just init_fn, but every runtime hook independently derived sched from
plugin->sched on each allocation attempt.

Adds seqjob_allocator_new_plugin()/parjob_allocator_new_plugin()
factories (plugin.h, mirroring limit_new_plugin()), letting
sim_alloc.c's bootstrap functions register both plugins DIRECTLY
against &(sim->core) via the existing plugin_register_core() -- the
same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already
use. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the
plugin-registry/fan-out path entirely; plugin_prepare_allocation()'s
fan-out during a live seqjob/parjob dispatch now genuinely reaches
sim->core.plugins, the same registry the limit plugin is registered
in.

One narrow, necessary exception remains, independently verified
confined: my_alloc() still needs a small xce_scheduler_t wrapper
(only .core.resmgr aliased to sim->core) to satisfy sched_make_allocs()
(scheduler.c, out of this task's boundary, unmodified) -- set on
plugin->sched only AFTER plugin_register_core() has run, never read by
the plugin registry or any prepare_allocation_hook fan-out.

TDD: new test_sim_alloc_limit_fanout.c drives a real live dispatch
(sched_job_picker()/sched_allocator() thread pair) with a host-scoped
limit registered directly against sim->core, proving a real usage
instance is created -- isolated from both reconciliation and
global-limit matching. RED: dispatch succeeded but the usage-instance
iterator was empty. GREEN: 82/82, zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-19 07:38:34 +00:00
parent 915782fc66
commit f6a2862110
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 778 additions and 193 deletions

View file

@ -22515,14 +22515,17 @@
"timestamp": "2026-09-19T04:00:00Z",
"error_message": "AddressSanitizer: SEGV (NULL pointer read) in sched_make_seqjob_allocation (src/scheduler/src/scheduler.c:59, `plugin->job_alloc_fn` where `plugin = core->seqjob_allocator` is NULL), called from sched_make_primary_alloc_ -> sched_make_primary_allocation -> sched_make_allocation -> sched_allocator (sched_election.c) -> the real pthread spawned by sched_sim_driver_run() (sim_loop.c). Discovered while writing the mandated TDD test for scheduler-sim-live-allocation task 4.1 (its own Observable: 'a simulation-mode session ... produces a genuine dispatch or pend decision for a job based on its actual resource availability, instead of unconditionally pending it') -- the crash is 100% reproducible for ANY job reaching primary resource allocation in simulation mode, regardless of resreq shape or dispatch/pend outcome (sched_make_parjob_allocation has the identical NULL-core->parjob_allocator crash for maxTasks>1 jobs).",
"root_cause": "core->seqjob_allocator/core->parjob_allocator (xce_sched_core_t fields, core.h) are never populated for the simulation driver, anywhere. Normal mode populates them via sched_init()'s sched_register_plugins() -> register_seqjob_allocator()/register_parjob_allocator() (plugins/seqjob_allocator.c, plugins/parjob_allocator.c), whose init_fn does `((xce_scheduler_t*)plugin->sched)->core.seqjob_allocator = plugin` -- REQUIRING a genuine, live xce_scheduler_t*. sched_sim_driver_start() (sim.c) registers no plugin at all (sim.h's own documented 'Known scope boundary'), and xce_sched_sim_driver_t deliberately has no xce_scheduler_t*/xce_scheduler_t field of its own (Req 5 AC 17, simulation isolation). This is NOT limited to the init_fn: every one of these two plugins' RUNTIME hooks (seqjob_prepare_allocation()/my_alloc(), parjob_prepare_allocation()/my_alloc(), and several internal static helpers taking xce_scheduler_t* sched as a parameter) ALSO read `plugin->sched->core.*` (resmgr for the 'slots'/rusage lookups, workers for host_iter, and sched_make_allocs()/sched_try_alloc_on_host_() in scheduler.c/resource.c which ALSO take xce_scheduler_t*/need `sched->core.resmgr`) on every single allocation attempt, not just at registration. This gap was already discovered and explicitly deferred by scheduler-sim-live-allocation task 3.1 (see test_sched_election_sim_fixture.c's own header comment: 'a pre-existing, tracked, OUT-OF-SCOPE gap ... NOT one of task 3.1's five permitted changes -- fixing it would be scope creep into a different, not-yet-assigned task') -- task 3.1's own fixture never reached this crash only because IT registers seqjob_allocator directly against a real, throwaway xce_scheduler_t wrapper whose OWN .core IS the core under test (single-core fixture), a shape simulation mode itself can never have (its core lives inside xce_sched_sim_driver_t, never xce_scheduler_t). Task 4.1 is the first task whose OWN production change (spawning the real sched_job_picker()/sched_allocator() thread pair against simulation mode's own core, replacing the old always-PEND sim_picker_thread_fn() placeholder that never called sched_make_allocation() at all) makes this code path reachable in simulation mode for the first time ever.",
"fix": "PARTIALLY FIXED (2026-09-19, scheduler-sim-live-allocation task 4.1, within that task's own file boundary -- a full fix is tracked as NOT YET DONE, see below). New file src/scheduler/src/sim_alloc.c (added to scheduler_srcs and sched_sim_isolation_lib, meson.build) provides sim_bootstrap_seqjob_allocator()/sim_bootstrap_parjob_allocator(), called from sched_sim_driver_run() (sim_loop.c) right after the existing sim_bootstrap_picker()/sim_bootstrap_limit_plugin() calls (same ordering constraint, same 'extern declared where consumed' idiom). Each builds a long-lived (heap-allocated, never freed -- same accepted-leak convention this file family already uses for its own never-joined threads), throwaway xce_scheduler_t wrapper, points its .core.resmgr/.core.workers fields (both `var`, core.h -- plain assignable rtti handles, not embedded values) at simulation mode's OWN resmgr/workers objects BEFORE calling the real, unmodified register_seqjob_allocator()/register_parjob_allocator() against it, then copies the resulting plugin pointer onto sim->core.seqjob_allocator/parjob_allocator. Because resmgr/workers are shared BY REFERENCE, any resource/worker registered later against the real sim->core (via the genuine event-ingest path) is immediately visible through the wrapper's aliased fields too, so seqjob_prepare_allocation()/my_alloc() see simulation mode's REAL, live resource/worker state -- not an empty, disconnected copy. Verified live: test_sim_live_allocation.c (new dedicated fixture, real scheduler_srcs link, real WORKERS_PUSH/RESOURCES_PUSH/JOBS_PUSH replay events, real sched_sim_driver_run() call, static-storage xce_sched_sim_driver_t) drives a job with a satisfiable resource want through the full real chain and observes a genuine `[sim] DISPATCH jobid=...` -- previously impossible (would have crashed) before this fix, PEND-only after reverting to sim_picker_thread_fn() at HEAD (confirmed RED). Full scheduler suite: 81/81, zero regressions (25 pre-existing white-box sim tests that #include sim_loop.c directly needed matching link-satisfying no-op/PEND-only stubs for the two new bootstrap functions and for the sched_job_picker()/sched_allocator() symbols sim_loop.c now calls directly instead of its own removed sim_picker_thread_fn() -- none of those 25 files test genuine resource-based allocation, so behavior-preserving stubs keep their existing assertions valid unchanged). NOT YET FIXED (tracked follow-up, explicitly out of task 4.1's file boundary): each wrapper's own core.plugins registry is separate from sim->core.plugins, so a plugin registered only against sim->core (e.g. the limit plugin via sim_bootstrap_limit_plugin()) is never reached by plugin_prepare_allocation()'s fan-out when it runs inside seqjob/parjob's own call chain rooted at the wrapper's core -- limit-plugin host-scoped matching under a REAL simulation-mode dispatch is not yet wired end-to-end (task 4.2/5.2's own territory, which specifically covers limit-usage parity under live sim-mode dispatch -- re-verify this exact scenario when implementing those). The FULL, root-cause fix -- retyping seqjob_allocator.c's/parjob_allocator.c's runtime hooks and internal helpers from `plugin->sched`/`xce_scheduler_t* sched` parameters to the already-available `plugin->core`/`xce_sched_core_t*` (task 1.2's own established pattern, mirroring task 2.3's limit.c migration) -- was NOT attempted: those hooks also call sched_make_allocs() (scheduler.c) and sched_try_alloc_on_host_() (resource.c) with the same xce_scheduler_t*, and scheduler.c/resource.c were explicitly out of task 4.1's file boundary (already migrated/committed by earlier tasks in this spec). A future task should do this properly rather than layering more wrapper-aliasing tricks.",
"fix": "FULLY FIXED (2026-09-19, scheduler-sim-live-allocation task 4.3, closing this bug's own tracked follow-up). Root cause: seqjob_allocator_init()/parjob_allocator_init() (plugins/seqjob_allocator.c, plugins/parjob_allocator.c) and every runtime hook these plugins register (seqjob_prepare_allocation()/my_alloc(), parjob_prepare_allocation()/my_alloc()/parjob_try_plan_()) now read the plugin's own honestly-typed `core` field (task 1.2's xce_sched_plugin_t.core) instead of deriving an xce_scheduler_t* from plugin->sched -- mirroring task 2.3's plugins/limit.c migration exactly. Two new factory functions, seqjob_allocator_new_plugin()/parjob_allocator_new_plugin() (plugin.h, mirroring limit_new_plugin()/builtin_fcfs_new_plugin()), let sim_alloc.c's sim_bootstrap_seqjob_allocator()/sim_bootstrap_parjob_allocator() register the allocator plugin DIRECTLY against &(sim->core) via the existing plugin_register_core() -- the same call sim_bootstrap_picker()/sim_bootstrap_limit_plugin() already use -- so plugin->core honestly points at sim->core and every later plugin_prepare_allocation() fan-out (my_alloc() -> sched_try_alloc_on_host_() -> plugin_prepare_allocation(), resource.c/plugin.c) reaches the SAME core->plugins registry the limit plugin (and picker) are registered in. Task 4.1's throwaway xce_scheduler_t wrapper is gone from the plugin-registry/fan-out path entirely -- no separate registry stands in for it anymore. One narrow, necessary exception remains: my_alloc() still keeps a small, long-lived, heap-allocated xce_scheduler_t wrapper (never freed, same accepted-leak convention) whose ONLY purpose is satisfying sched_make_allocs()'s (scheduler.c, out of this task's file boundary, unmodified) own real xce_scheduler_t* requirement for the FINAL allocation record of an already-decided winning host -- that function's body only ever reads sched->core.resmgr/lock/nextid, so the wrapper's .core.resmgr is ALIASED (a `var`, not copied) to sim->core's own; it is set on plugin->sched ONLY AFTER plugin_register_core() has already run (init_fn no longer reads plugin->sched at all) and is never used for the plugin registry or any prepare_allocation_hook fan-out, so it cannot reintroduce the separate-registry gap this fix closes. Verified live: new test_sim_alloc_limit_fanout.c (real scheduler_srcs link, bare xce_sched_sim_driver_t fixture, real sched_job_picker()/sched_allocator() thread pair) registers a host-scoped limit directly against sim->core (sim_bootstrap_limit_plugin()) alongside the seqjob allocator plugin (sim_bootstrap_seqjob_allocator()), drives a genuine live dispatch to a real DISPATCH decision (JOB_STAT_RUN), and confirms the limit's usage instance is created (limit_get_usage_instances()) -- RED before this fix (job dispatches correctly but zero usage instances; the fan-out never reached the limit plugin), GREEN after. Full scheduler suite: 82/82, zero regressions -- three white-box test files that construct an xce_sched_plugin_t directly without going through plugin_register()/plugin_register_core() (test_seqjob_allocator.c, test_seqjob_rewrap_equivalence.c, test_parjob_combinatorial.c) needed their fixture plugin instances' own `.core` field set alongside the pre-existing `.sched` field, plus parjob_try_plan_()'s one direct white-box call site (test_parjob_combinatorial.c) updated from &sched to &(sched.core) for its now-honest xce_sched_core_t* signature -- the same established ripple-fix pattern this feature's own tasks.md Implementation Notes already document for prior signature migrations.",
"files": [
"src/scheduler/src/sim_alloc.c (new file)",
"src/scheduler/src/sim_loop.c (sched_sim_driver_run() call sites)",
"src/scheduler/meson.build (scheduler_srcs, sched_sim_isolation_lib)",
"src/scheduler/testsuite/test_sim_live_allocation.c (new file, RED/GREEN proof)",
"src/scheduler/src/plugins/seqjob_allocator.c (root cause, not modified)",
"src/scheduler/src/plugins/parjob_allocator.c (root cause, not modified)"
"src/scheduler/include/plugin.h (seqjob_allocator_new_plugin()/parjob_allocator_new_plugin() factory declarations, task 4.3)",
"src/scheduler/src/plugins/seqjob_allocator.c (task 4.3: init_fn + prepare_allocation_hook + job_alloc_fn retyped to read plugin->core; new seqjob_allocator_new_plugin() factory -- genuinely modified, no longer just the root-cause file)",
"src/scheduler/src/plugins/parjob_allocator.c (task 4.3: same retyping + new parjob_allocator_new_plugin() factory)",
"src/scheduler/src/sim_alloc.c (task 4.1: new file, initial bootstrap via throwaway wrapper; task 4.3: rewritten to register directly against sim->core via plugin_register_core(), no wrapper in the registration/fan-out path)",
"src/scheduler/src/sim_loop.c (task 4.1: sched_sim_driver_run() call sites -- unmodified by task 4.3)",
"src/scheduler/meson.build (task 4.1: scheduler_srcs, sched_sim_isolation_lib -- unmodified by task 4.3)",
"src/scheduler/testsuite/test_sim_live_allocation.c (task 4.1: new file, live-dispatch RED/GREEN proof)",
"src/scheduler/testsuite/test_sim_alloc_limit_fanout.c (task 4.3: new file, host-scoped-limit-fan-out RED/GREEN proof)",
"src/scheduler/testsuite/test_seqjob_allocator.c, test_seqjob_rewrap_equivalence.c, test_parjob_combinatorial.c (task 4.3: white-box fixture ripple -- plugin instances built directly in these tests needed their own .core field set alongside .sched)"
],
"tags": [
"scheduler",
@ -22536,13 +22539,15 @@
"scheduler-sim-live-allocation",
"task-4.1",
"partially-fixed",
"follow-up-tracked"
"follow-up-tracked",
"task-4.3",
"fully-fixed"
],
"related_bugs": [
"bug-1828"
],
"occurrences": 1,
"last_seen": "2026-09-19T04:00:00Z"
"last_seen": "2026-09-19T08:00:00Z"
}
]
}