myxce/.kiro/specs/scheduler-allocation/tasks.md

Implementation Plan

• 1. Foundation: allocation data types and extension points

• 1.1 Add the lazy plan-generation and host-candidate iterator contracts

  • Define the plan representation as a partition of a job's task count into per-host slot buckets, most-preferred ordering to be produced by whichever component generates plans
  • Define the plan-iterator and host-iterator contracts (initialize, produce next candidate, release resources) as reusable abstractions any allocator can adopt
  • Observable: the contracts compile and are referenced by at least a stub/no-op implementation with no allocator wired in yet
  • Requirements: 2.1, 3.1

• 1.2 Add resreq fingerprinting for the unsatisfiable-resreq cache

  • Compute a stable resource-usage fingerprint once when a resource request is parsed
  • Store that precomputed fingerprint on the job's scheduling object once at ingest, so every later cache lookup for that job reuses it instead of recomputing it (the full cache key, fingerprint plus user/queue/project identity, is still assembled fresh at each lookup)
  • Write a unit test proving the fingerprint is stable across repeated computation for an identical resource request and differs when any contributing field changes
  • Observable: a passing test demonstrates identical requests produce identical fingerprints and any differing field changes the fingerprint
  • Requirements: 7.1

• 1.3 Add the forward-compatible allocation-mode field

  • Add an allocation-mode field to the per-attempt allocation context, defaulting to normal-resources-only
  • Observable: the field exists, defaults correctly, and no existing allocation code path branches on any value other than the default
  • Requirements: 9.5

• 1.4 Add the per-host resource-debit reversal primitive

  • Add a helper that reverses exactly the provisional resource debit an earlier successful host-claim made, using the existing subtraction primitive
  • Write a unit test that debits a host's provisional usage, reverses it, and confirms the host's provisional usage returns to its pre-debit value
  • Observable: a passing test shows debit-then-reverse leaves provisional usage unchanged from baseline
  • Requirements: 4.2

• 2. Core: shared allocation infrastructure

• 2.1 (P) Implement and test the lazy host-candidate iterator

  • Implement the shared host-candidate iterator against the existing select-expression evaluator and resource-availability check, with no allocator-specific logic
  • Ensure each call advances a real cancellation point so a long-running scan can be safely interrupted
  • Write a unit test proving a host too small for one candidate search remains a valid candidate for a later, smaller search using a freshly-initialized iterator over the same underlying worker set
  • Observable: a passing test suite covers select-expression filtering, availability filtering, exhaustion signaling, and the too-small-then-fits-smaller-search case
  • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6
  • Boundary: host_iter.c

• 2.2 (P) Implement and test the unsatisfiable-resreq cache

  • Implement the cache's key type and its hash/comparison/copy/release operations, backed by the existing keyed-table infrastructure, storing a reason category (not a bare present/absent marker) as each entry's value
  • Implement lookup, insert, and clear operations against the cache
  • Write a unit test proving a lookup on an identical fingerprint-plus-identity is a hit that returns the same reason category given at insert time, a lookup on any differing field is a miss, and clear empties all entries
  • Observable: a passing test suite includes a same-key-twice hit assertion (not only an insert-then-miss-on-a-different-key round-trip) and a reason-category round-trip assertion
  • Requirements: 7.1, 7.3, 7.4, 7.5
  • Boundary: alloc_cache.c

• 3. Core: seqjob allocator rewrap

• 3.1 Rewrap seqjob's allocation onto the shared iterators

  • Restructure seqjob's allocation to use the shared plan-iterator and host-iterator contracts as a trivial single-bucket case, replacing its own inline worker-collection loop
  • Add a guard so seqjob's resource-request preparation only ever acts on single-task jobs, closing the fan-out gap this creates for the parallel allocator's later fix
  • Observable: seqjob's allocation still runs end to end using the shared iterators, with no allocator-specific worker-collection logic left in seqjob's own code
  • Requirements: 9.1, 4.6

• 3.2 Prove seqjob's rewrap preserves today's host-selection behavior

  • Write a test suite constructing fixture states with multiple candidate hosts, including at least one host failing the select filter, asserting identical host selection to today's pre-rewrap behavior given identical worker/resource state
  • Observable: a passing equivalence test suite demonstrates no behavior change across fixture states beyond the single-obvious-host trivial case
  • Requirements: 9.1
  • Depends: 3.1

• 4. Core: parjob combinatorial allocation

• 4.1 Implement parjob's lazy combinatorial plan generation

  • Generate candidate plans as a pure function of a job's task count: ascending host-count order, one balanced partition per host-count, descending bucket order within each plan
  • Ensure each call advances a real cancellation point and that plan generation never inspects live worker state (a pure function of the task count alone)
  • Write a unit test proving one-plan-per-call generation, most-preferred-first ordering (including an uneven split, e.g. splitting 7 tasks across 3 hosts), and a stable exhaustion signal
  • Observable: a passing test suite covers plan ordering, uneven-split partitioning, and exhaustion signaling
  • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 4.1

• 4.2 Implement parjob's per-bucket allocation loop

  • Replace the parallel allocator's stub with a real implementation that, for each candidate plan, claims a distinct host for every bucket using a freshly-initialized host search per bucket, excluding only hosts already claimed elsewhere in the same plan attempt
  • Fix the parallel allocator's resource-request preparation to compute each bucket's want from that bucket's own slot count rather than a fixed value
  • On a bucket failure, reverse the provisional debits already made for that plan's earlier successful buckets before advancing to the next candidate plan
  • Return the same not-satisfied signal as before when every candidate plan is exhausted, leaving pend-reason handling unaffected
  • Write a unit test proving a plan whose buckets are tried in descending size order succeeds where the same host pool would fail in ascending order, and a test proving a mid-plan bucket failure correctly reverses earlier successful claims before the next plan is tried
  • Observable: a passing test suite covers multi-bucket success, the descending-vs-ascending worked example, and partial-failure-then-reversal
  • Requirements: 4.2, 4.4, 4.5, 4.6, 4.7
  • Depends: 3.1

• 4.3 Implement parjob's multi-host allocation commit

  • Merge every claimed bucket's host allocation into a single job allocation record spanning all claimed hosts, leaving job-level shared-resource allocation computed once and unaffected by how many hosts were used
  • Observable: a successfully allocated multi-host job's allocation record correctly lists every claimed host with its own slot count
  • Requirements: 4.3

• 5. Integration: dispatch and cache wiring

• 5.1 (P) Wire the unsatisfiable-resreq cache around the allocator dispatch

  • Before running full allocation for a primary resource request (whether the job's own request is a primary, or one nested inside a compound or an alternative), look up its fingerprint in the cache; on a hit, pend the job immediately using the stored reason category without invoking either allocator
  • When an allocator exhausts every candidate plan, populate the cache with the job's fingerprint and the resulting reason category before pending the job
  • Observable: a job sharing an identical fingerprint and identity with a previously-exhausted job is pended immediately with the same reason category, without either allocator running
  • Requirements: 7.3, 7.4, 7.5
  • Boundary: scheduler.c dispatch
  • Depends: 2.2, 3.1, 4.2

• 5.2 (P) Clear the cache at the start of every scheduling session

  • Clear the cache as part of the existing per-session reset step, alongside the other state that step already resets
  • Observable: a job cached as unsatisfiable in one session is re-evaluated fresh (not immediately pended from the cache) in the next session
  • Requirements: 7.2
  • Boundary: core_session.c
  • Depends: 2.2

• 5.3 End-to-end multi-slot parallel job allocation proof

  • Drive a real multi-host parallel job through the full dispatch path from submission to a committed multi-host allocation
  • Observable: the resulting job allocation spans more than one host with correct per-host slot counts, confirmed through the same dispatch path production jobs use
  • Requirements: 4.1, 4.2, 4.3
  • Depends: 5.1, 5.2, 4.3

• 6. Validation: regression and soak

• 6.1 Full existing scheduler regression pass

  • Run the existing scheduler test suite, updating only the assertions that directly exercised the eager-one-shot or stub-return behavior this feature replaces
  • Confirm the compound and alternative resource-request dispatch paths remain exercised, unmodified, by their existing tests
  • Observable: the full existing scheduler suite passes with no unexpected assertion changes
  • Requirements: 9.2, 9.3
  • Depends: 5.3

• 6.2 HA soak regression

  • Run a full sequential and chaotic HA regression soak
  • Observable: the soak reports zero new job loss, divergence, or liveness stall attributable to this feature
  • Requirements: 9.4
  • Depends: 6.1

Implementation Notes

• (2.2) xce_scheduler_t.unsatisfiable_cache (constructed in sched_init()/destroyed in sched_fini(), matching xce_select.c's sel_node_table_init() construction convention exactly) lives on the outer xce_scheduler_t, per Req 7 AC1. Task 5.2's clear call site (core_begin_session_internal(), core_session.c) only receives xce_sched_core_t* core — since core is confirmed the first field of xce_scheduler_t, task 5.2 should recover the outer struct via (xce_scheduler_t*)core (valid, defined C behavior), not by widening the function's signature.
• (2.1) xce_alloc_plan_t gained a ctx field (xce_sched_ctx_t*) so host_iter_init() can reach ctx->sched/ctx->primary->select/ctx->alloc_req.host — host_iter_init() rejects a plan with ctx == NULL with -EINVAL rather than crashing. Tasks 3.1 and 4.1 must set plan->ctx = ctx on every plan their plan-iterator returns from next() (not just once) before constructing a host_iter from it — a plan-iterator that reuses one internal buffer across calls/jobs must refresh this field each time, since a stale-but-non-NULL pointer from a torn-down ctx is not caught by the NULL guard.
• (1.4) When a new scheduler test needs a real function from schedobj.c/core.c (e.g. sched_ctx_get_uncomit_host()), white-box #include the real source file rather than privately reimplementing its logic in the test — established precedent: test_picker_config_fields.c's #include "../src/core.c" + #include "../src/schedobj.c" pairing. A private reimplementation may pass today but silently drifts from the real function with nothing to catch it later.
• (1.1) xce_host_iter_t.next() returns xce_sched_worker_t* (sched/types.h, carries resvals/resInUse/host_eval_ctx) — never xce_worker_t* (xce/types.h, the wire/config type make_sched_worker() converts FROM). requirements.md/design.md briefly had this wrong; fixed in commit d1a3e158. Task 2.1 (host_iter.c's real implementation) and any later task touching worker candidates must use xce_sched_worker_t* from the start.
• (1.2) resreq_fp's ingest-time population lives in core_build_primary_schedobj() (src/scheduler/src/core_events.c), right after primary->resreq = resreq; — confirmed wired (min_tasks/max_tasks/select_node_id/rusage_hash all set from the parsed xce_primary_resreq_t) as of task 1.2 round 2, regression-covered by test_core_build_primary_schedobj_populates_resreq_fp_from_parsed_resreq in src/scheduler/testsuite/test_core_events.c. Round 1 added the type/field/hash machinery but never wired this call site, leaving resreq_fp permanently zeroed — do not rediscover this gap in a later task.
• (3.1) The per-host try-alloc leaf (sched_try_alloc_on_host_(xce_sched_worker_t* w, xce_scheduler_t* sched, xce_sched_ctx_t* ctx) — select check → sched_ctx_get_uncomit_host → plugin_prepare_allocation → sched_check_host_satisfy → xvec_idval(..., XVEC_ADD_FLOAT)) now lives in resource.c, exported (non-static, non-variadic) and declared in scheduler.h. Task 4.2 (parjob's per-bucket allocation loop) must call this exact shared function from its own bucket/host_iter loop, not write a second copy. Round 1 of this task wrote a private ~70-line duplicate (seqjob_try_alloc_on_host_) because the original was static/va_list-based; round 2 fixed this by exporting the real function and deleting both the duplicate and the now-dead sched_try_alloc_on_hosts() variadic wrapper. This is the same reimplementation-vs-share risk task 1.4 hit in test code — here it recurred in production code, one task later.
• (3.1) src/scheduler/meson.build's production scheduler_srcs list was missing host_iter.c since task 2.1 only wired that file into testsuite/meson.build — invisible until task 3.1 became the first production caller of host_iter.c's functions (a shared .c file with zero production callers builds/links fine without being in scheduler_srcs at all, so task 2.1's own green build/test run couldn't have caught this). Any future task adding a new shared scheduler .c file should double check it lands in scheduler_srcs, not only testsuite/meson.build, even if nothing calls it yet.
• (3.2) host_iter.c's pre-filter select check and sched_try_alloc_on_host_()'s (resource.c) own independent sched_meet_host_selection re-check are pre-existing, intentional defense-in-depth: both run the identical check against the same live state, so a regression in either one alone is invisible at the observable-pick level — only a regression in both simultaneously changes behavior. This is benign and predates this feature; task 4.2 (parjob) will inherit the same double-check when it reuses host_iter.c/sched_try_alloc_on_host_(), and should not "fix" the redundancy as part of that task.
• (3.2) sched_collect_workers()'s deleted (task 3.1) iteration and host_iter.c's iter_make()/iter_next() iteration are structurally identical, not coincidentally similar: both called the exact same third-party iter_make()/iter_next() (RTTI lib) over the literal same sched->core.workers var, and the old xlist_find() walk was forward/insertion-order — confirmed via git show b18c5f78^:src/scheduler/src/scheduler.c and xlist.c (unmodified across this feature). Useful precedent if task 4.1/4.2's parjob equivalence claims ever need the same grounding.
• (4.2) sched_host_iter_next()'s pre-filter reads ctx->alloc_req.host (which sched_check_host_satisfy() trivially passes when count==0), but nothing primes it per-bucket — a fresh per-bucket host_iter scan must reset ctx->alloc_req.host to empty at the top of each bucket's scan (parjob_allocator.c's parjob_try_plan_()) or it can inherit a stale want from a differently-sized prior bucket. This is SAFE, not an oversell risk: sched_try_alloc_on_host_() always calls plugin_prepare_allocation() (which repopulates .host from the bucket's real ctx->nslots) before its own authoritative sched_check_host_satisfy() — the reset only makes the pre-filter a one-time-per-bucket no-op, never wrong. Task 5.x (cache/dispatch wiring) doesn't touch this path, but any future allocator built on host_iter.c with a multi-bucket loop should reset ctx->alloc_req.host the same way.
• (4.2) parjob_allocator.c's my_alloc() guards primary->maxTasks <= 1 → return 0 (matching Req 4 AC1's literal maxTasks > 1 precondition) — required because a snapshot-restored job with an unpopulated maxTasks==0 (bug-1767's pre-existing, out-of-scope gap in snapshot_load.c) fails seqjob's maxTasks != 1 guard (task 3.1) and falls through to parjob; without this guard parjob's now-real my_alloc() creates a ctx->reason where the old inert stub never did, corrupting unrelated snapshot-regression assertions. Causally verified (temporarily disabled, confirmed the exact predicted failure, restored).
• (4.2) The multi-host allocation merge (task 4.3's stated scope: "merge every claimed bucket's host allocation into a single job allocation record") is already implemented and verified as part of 4.2 — make_job_alloc() takes a plain runtime-sized xce_host_alloc_t** array, and my_alloc() already builds/passes it with nhosts == nclaimed == plan->nbuckets on every full-plan success. Task 4.3 may only need targeted verification of the job-level shared-resource-computed-once property, not new implementation — check before dispatching a full implementer round.
• (5.1) sched_make_primary_alloc_() (scheduler.c) is the single choke point for ALL primary-resreq dispatch shapes — standalone, compound-nested, and (transitively, via sched_make_alter_allocation()'s delegation to sched_make_compound_allocation()) alter-nested. Any future dispatch-level change (cache, mode-switching, etc.) belongs there, not in the three callers.
• (5.1) A nested primary (inside a COMPOUND/ALTER) has no real per-primary xce_sched_obj_t; sched_primary_cache_obj_() synthesizes one on the stack per dispatch for cache-key purposes. This is memory-safe (verified line-by-line): alloc_cache.c's build_key_() copies the fingerprint and identity fields by value synchronously during the call, before the stack frame unwinds — no pointer into the caller's stack locals is ever retained.
• (5.1) Req 7.5's "same reason category" is only exercisable in the degenerate all-zero case today: no code path anywhere in the scheduler sets xce_job_preason_t.main_reason to a nonzero value except this task's own new sched_pend_from_cached_reason_() — every existing allocator-exhaustion path only sets host/resource sub-reasons via add_job_preason(). If a future task wants to prove genuine (non-degenerate) category propagation, it needs a path that actually produces a nonzero main_reason first.
• (5.1) Two significant, pre-existing, out-of-scope bugs were discovered and confirmed during this task's review, logged as bug-1768 (compound/alter-nested primaries never have real slot/rusage availability checked — seqjob_prepare_allocation()/parjob_prepare_allocation() both no-op when ctx->schedobj->tag != XCE_RESREQ_PRIMARY) and bug-1769 (sched_make_alter_allocation() type-confuses xce_compound_schedobj_t* as xce_sched_obj_t*, a ~112-byte heap over-read, currently completely untested — the existing test believed to cover ALTER dispatch actually parses as PRIMARY per the resreq grammar). Neither is in this feature's scope to fix; both need dedicated follow-up.
• (5.2) core_begin_session_internal() (core_session.c) is called from TWO structurally incompatible types sharing the same xce_sched_core_t core prefix: driver.c (xce_scheduler_t*) and sim_loop.c (xce_sched_sim_driver_t*, deliberately isolated, no cache field). Round 1's __attribute__((weak)) cast-based approach caused every simulation session to silently corrupt xce_sched_sim_driver_t.alt_events_file (same byte offset as xce_scheduler_t.unsatisfiable_cache), invisible to ASAN (in-bounds) and to the whole test suite (no test_sim_*.c links alloc_cache.c). Any future per-session hook that needs the real xce_scheduler_t (not just the shared xce_sched_core_t prefix) must use the driver-registered-callback pattern (xce_sched_core_t.cache_clear_cb, set via main.c only in the driver-mode branch — --sim mode returns before reaching that registration — mirroring the existing on_decision/xce_core_set_decision_cb() precedent), never a cast of core back to xce_scheduler_t* inside code shared by both driver and sim paths.
• (5.2) The real cache_clear(xce_scheduler_t*)→xce_alloc_cache_clear_cb(xce_sched_core_t*) adapter lives in alloc_cache.c, not driver.c, specifically because ~11 test files white-box #include "../src/driver.c" without linking alloc_cache.c — placing the adapter in driver.c would break their link. Any future adapter bridging a driver-only hook to a subsystem those tests don't link should follow this same placement rule (adapter lives with the subsystem it calls into, not with driver.c).
• (6.2) Both sequential and chaotic HA soaks pass cleanly on the final commit (zero job loss/divergence/stall), and were independently re-run by the reviewer with matching results. The soak's worker-rejoin-resync coverage (kill primary mid-job, wait for new-primary election, confirm reconciliation) is live evidence that task 5.2's cache_clear_cb fix holds under real failover, since every new-primary election runs a fresh core_begin_session_internal(). Standing caveat for the whole feature: this soak — like everything else in this spec — can only exercise the real production surface bug-1770 leaves reachable (seqjob's rewrap + cache dispatch/clear wiring). parjob_allocator's entire multi-host allocation (tasks 4.1-4.3, 5.1-5.2) has zero live production path until bug-1770 is fixed as a dedicated follow-up; this feature is soak-validated but not yet "done" for real clusters.
• (6.1) Regression audit (base c4f2fec0 vs. HEAD c009f108, full feature range): only 4 pre-existing src/scheduler/testsuite/ files were touched (meson.build, test_core_events.c, test_job_update.c, test_resource_accounting.c), all pure additions or link-scaffolding replacement — zero existing assertions altered. All COMPOUND/ALTER-referencing existing tests (test_snapshot_load_allocator_regression.c, test_sim_record_output.c, test_sim_alt_events.c, test_snapshot_save_load.c, test_scheduler_backpressure.c) confirmed byte-identical to pre-feature state despite task 5.1 modifying their shared sched_make_primary_alloc_() funnel. One benign, out-of-formal-scope gap noted: src/lib/testsuite/test.c (a different testsuite tree) also got a 2-line pure-wiring addition for task 1.2's new fingerprint test, omitted from the audit's "4 files" framing but not an altered assertion.
• (5.3) CRITICAL, feature-level finding: parjob_allocator (tasks 4.1-4.3, 5.1-5.2, all already implemented/reviewed/committed) has ZERO live production path today. parse_resreq_primary() (src/lib/resreq.c:431-432) unconditionally hardcodes primary->minTasks = 1; primary->maxTasks = 1 regardless of what the job actually requests — this predates the entire scheduler-allocation feature by ~7 months (commit cd4f3953, 2026-02-05), untouched by task 1.2's diff. The real, user-facing submission pipeline (apiserver/models/jobs.go's num_processors/maxnum_processors → controllers/jobs.go → backend/convert.go:286-292 → controller.capnp → core_build_job()/make_sched_job()) correctly threads the requested task count onto the TOP-LEVEL xce_sched_job_t.mintasks/maxtasks — but NOTHING ever copies that onto the NESTED xce_primary_resreq_t.minTasks/maxTasks that sched_make_primary_alloc_()'s seqjob-vs-parjob dispatch branch (scheduler.c:290) and parjob_allocator.c's my_alloc() actually read. Separately, the resreq grammar's own span[ptile=N] syntax parses into xce_resreq_data_t.spanString and is then discarded (primary->span stays a stub empty struct) — a fix needs to reconcile BOTH the job-level submitted task count AND resreq's own span syntax, not just wire one through. Logged as bug-1770 (with bug-1771, a related fcfs NULL-queue-deref crash discovered while building task 5.3's fixture). This must be a dedicated, high-priority follow-up task/spec — the scheduler-allocation feature has no practical effect on real clusters until it lands.