test(scheduler-allocation): end-to-end multi-host parallel job proof

Drive a real multi-slot parallel job through the actual production
dispatch path -- real sched_init(), real queue/worker/job ingest via
xce_core_push_events(), a real scheduling session (exercising task 5.2's
cache-clear wiring), the real sched_allocator() thread, cache lookup,
parjob_allocator's real my_alloc(), host_iter, and commit -- and confirm
the resulting job allocation genuinely spans 2 hosts with correct
per-host slot counts (Req 4.1, 4.2, 4.3).

Discovered and logged bug-1770: parse_resreq_primary() unconditionally
hardcodes primary->minTasks/maxTasks to 1 regardless of what a job
actually requests, and nothing in the real ingest pipeline ever copies
the job's own submitted task count (correctly threaded from the
apiserver's num_processors/maxnum_processors all the way to
xce_sched_job_t.mintasks/maxtasks) onto the nested xce_primary_resreq_t
fields the allocator dispatch actually reads. This predates the feature
by ~7 months and is out of this spec's scope to fix, but it means
parjob_allocator has zero live production path until a dedicated
follow-up lands. The test compensates with one clearly-labeled patch of
the real, already-ingested resreq's task-count fields immediately after
real parsing; everything else in the pipeline is unmodified production
code.

Also logged bug-1771 (xce_sched_fcfs's fcfs_del_job() dereferences
job->queue->name with no NULL guard) discovered while building this
task's fixture; worked around by registering the job's queue before
push, mirroring the apiserver's own pre-submission queue validation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg
This commit is contained in:
Rongsong Shen 2026-09-05 01:56:54 +00:00
parent e8c2087d5e
commit c009f108b7
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
4 changed files with 694 additions and 1 deletions

View file

@ -85,7 +85,7 @@
- _Requirements: 7.2_
- _Boundary: core_session.c_
- _Depends: 2.2_
- [ ] 5.3 End-to-end multi-slot parallel job allocation proof
- [x] 5.3 End-to-end multi-slot parallel job allocation proof
- Drive a real multi-host parallel job through the full dispatch path from submission to a committed multi-host allocation
- Observable: the resulting job allocation spans more than one host with correct per-host slot counts, confirmed through the same dispatch path production jobs use
- _Requirements: 4.1, 4.2, 4.3_
@ -124,3 +124,4 @@
- (5.1) Two significant, pre-existing, out-of-scope bugs were discovered and confirmed during this task's review, logged as bug-1768 (compound/alter-nested primaries never have real slot/rusage availability checked — `seqjob_prepare_allocation()`/`parjob_prepare_allocation()` both no-op when `ctx->schedobj->tag != XCE_RESREQ_PRIMARY`) and bug-1769 (`sched_make_alter_allocation()` type-confuses `xce_compound_schedobj_t*` as `xce_sched_obj_t*`, a ~112-byte heap over-read, currently completely untested — the existing test believed to cover ALTER dispatch actually parses as PRIMARY per the resreq grammar). Neither is in this feature's scope to fix; both need dedicated follow-up.
- (5.2) `core_begin_session_internal()` (`core_session.c`) is called from TWO structurally incompatible types sharing the same `xce_sched_core_t core` prefix: `driver.c` (`xce_scheduler_t*`) and `sim_loop.c` (`xce_sched_sim_driver_t*`, deliberately isolated, no cache field). Round 1's `__attribute__((weak))` cast-based approach caused every simulation session to silently corrupt `xce_sched_sim_driver_t.alt_events_file` (same byte offset as `xce_scheduler_t.unsatisfiable_cache`), invisible to ASAN (in-bounds) and to the whole test suite (no `test_sim_*.c` links `alloc_cache.c`). **Any future per-session hook that needs the real `xce_scheduler_t` (not just the shared `xce_sched_core_t` prefix) must use the driver-registered-callback pattern** (`xce_sched_core_t.cache_clear_cb`, set via `main.c` only in the driver-mode branch — `--sim` mode returns before reaching that registration — mirroring the existing `on_decision`/`xce_core_set_decision_cb()` precedent), never a cast of `core` back to `xce_scheduler_t*` inside code shared by both driver and sim paths.
- (5.2) The real `cache_clear(xce_scheduler_t*)`→`xce_alloc_cache_clear_cb(xce_sched_core_t*)` adapter lives in `alloc_cache.c`, not `driver.c`, specifically because ~11 test files white-box `#include "../src/driver.c"` without linking `alloc_cache.c` — placing the adapter in `driver.c` would break their link. Any future adapter bridging a driver-only hook to a subsystem those tests don't link should follow this same placement rule (adapter lives with the subsystem it calls into, not with `driver.c`).
- (5.3) **CRITICAL, feature-level finding**: `parjob_allocator` (tasks 4.1-4.3, 5.1-5.2, all already implemented/reviewed/committed) has ZERO live production path today. `parse_resreq_primary()` (`src/lib/resreq.c:431-432`) unconditionally hardcodes `primary->minTasks = 1; primary->maxTasks = 1` regardless of what the job actually requests — this predates the entire scheduler-allocation feature by ~7 months (commit `cd4f3953`, 2026-02-05), untouched by task 1.2's diff. The real, user-facing submission pipeline (`apiserver/models/jobs.go`'s `num_processors`/`maxnum_processors` → `controllers/jobs.go` → `backend/convert.go:286-292` → `controller.capnp` → `core_build_job()`/`make_sched_job()`) correctly threads the requested task count onto the TOP-LEVEL `xce_sched_job_t.mintasks/maxtasks` — but NOTHING ever copies that onto the NESTED `xce_primary_resreq_t.minTasks/maxTasks` that `sched_make_primary_alloc_()`'s seqjob-vs-parjob dispatch branch (`scheduler.c:290`) and `parjob_allocator.c`'s `my_alloc()` actually read. Separately, the resreq grammar's own `span[ptile=N]` syntax parses into `xce_resreq_data_t.spanString` and is then discarded (`primary->span` stays a stub empty struct) — a fix needs to reconcile BOTH the job-level submitted task count AND resreq's own span syntax, not just wire one through. Logged as bug-1770 (with bug-1771, a related fcfs NULL-queue-deref crash discovered while building task 5.3's fixture). **This must be a dedicated, high-priority follow-up task/spec — the scheduler-allocation feature has no practical effect on real clusters until it lands.**