myxce/.kiro/specs/scheduler-allocator-runner/requirements.md

Requirements Document

Project Description (Input)

Who has the problem: scheduler plugin authors and maintainers of the XCE scheduler allocation path.

Current situation: seqjob_allocator.c (my_alloc, ~215 lines) and parjob_allocator.c (my_alloc, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its PREASON_NO_RESOURCE fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (make_job_alloc + sched_make_allocs + the -ENOMEM cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole my_alloc(). Plan generation is already abstracted behind xce_alloc_plan_iter_t, whose init takes an unused xce_scheduler_t* even though sim mode has no real scheduler. Tracked as issues/new_requirements.md item 4.

What should change: the scheduler framework owns allocation. A new xce_sched_allocator_iface_t (an optional accepts guard plus plan_init/plan_next/plan_fini), referenced from xce_sched_plugin_t in the same way as picker_iface, lets one framework function, sched_allocator_run(), carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register .job_alloc_fn = sched_allocator_run, supply only their plan iterator and prepare_allocation, and no longer define my_alloc(). The third parameter of xce_alloc_plan_iter_t.init is retyped to xce_sched_core_t*. Allocation decisions, pend reasons (except the per-host reasons of both allocators, which become complete, Requirement 3 AC 6-7, and the shared-resource reasons bug-1893 never recorded, Requirement 2 AC 2), and error and -ENOMEM behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin commit hook and no sched_make_allocs() retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in brief.md → Constraints (the per-bucket alloc_req.host reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where nslots/nhosts are consumed, seqjob accepts for direct callers, and no VLA). Full context: .kiro/specs/scheduler-allocator-runner/brief.md.

Introduction

The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.

This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the per-host pend reasons of both allocators (Requirement 3 AC 6 and AC 7), which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which today are never recorded (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.

Terms:

• Allocation attempt: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
• Plan: one candidate way of splitting a job's tasks into buckets, each bucket placed on one host.
• Claim: one bucket satisfied on one host within the plan being tried.
• Pending signal: the allocator reports "not satisfiable now" (result 0) without an error.

Boundary Context

• In scope: a framework-owned allocation routine used by both built-in allocators; the allocator plugin contract (plan generator, optional acceptance guard, and the existing preparation hook); removal of the per-plugin allocation routines; changing the plan-generator initialisation to take the scheduling core instead of the scheduler object; keeping every existing allocation, pend-reason, error and simulation-parity behaviour apart from the intentional changes listed in the Introduction; closing issues/new_requirements.md item 4.
• Out of scope: a per-plugin override hook for assembling results; retyping the allocation-record constructor (sched_make_allocs) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); publishing pend reasons only when they change (a follow-up: complete per-host reasons make each pending job's reason set larger); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
• Adjacent expectations: the scheduler's allocation dispatch keeps sending maxTasks == 1 primaries to seqjob and every other primary to parjob, and keeps treating result > 0 as satisfied, 0 as pending and < 0 as error, with an allocation published only on > 0. COMPOUND/ALTER handling (bug-1766: a merged allocation is published only if every primary succeeds) and snapshot-restored schedobj construction (bug-1767) stay owned by their existing code. The plan and host iterator contracts from scheduler-allocation keep their current ordering semantics.

Requirements

Requirement 1: Framework-owned allocation

Objective: As a scheduler plugin author, I want the framework to run the allocation algorithm, so that an allocator plugin only has to describe how it generates candidate plans.

Acceptance Criteria

1. The Scheduler Plugin Framework shall provide one allocation routine that both built-in allocator plugins (seqjob and parjob) register as their job-allocation entry point.
2. The Scheduler Plugin Framework shall let an allocator plugin supply a candidate-plan generator (initialise, next plan, finish), an optional acceptance guard, and its existing per-allocation preparation hook, and nothing else, to take part in allocation.
3. Where an allocator plugin supplies no acceptance guard, the Scheduler Plugin Framework shall treat every primary resource request passed to that plugin as accepted.
4. When an allocator plugin's acceptance guard rejects a primary resource request, the Scheduler Plugin Framework shall return the pending signal without creating a pend-reason record, checking resources, or initialising the plan generator.
5. The seqjob and parjob allocator plugins shall contain no private allocation routine that duplicates the shared-resource check, the plan/host search, or the result assembly.
6. The Scheduler Plugin Framework shall keep the allocator plugin contract as data the plugin registry does not free, with the same lifetime as the existing job-picker contract: it lives as long as the registered plugin, and the plugin does not release it separately.

Requirement 2: Shared-resource precheck

Objective: As a scheduler operator, I want jobs whose cluster-wide (shared) resources are unavailable to pend with the same main reason as today and a reason for exactly the shared resources that are short, so that pend diagnostics stay stable and become accurate.

Acceptance Criteria

1. When an allocation attempt has passed the missing-input check and the acceptance guard, and the attempt has no pend-reason record yet, the Scheduler Plugin Framework shall create one before checking shared resources.
2. When the job's shared-resource request cannot be met, the Scheduler Plugin Framework shall add one "no resource" pend reason, with no host and naming the resource, for each entry of the unmet-resource list the shared check returns, and no other shared-resource reason. It shall then return the pending signal without trying any host. (Intentional change, bug-1893: today both allocators skip every successful resource lookup, because the lookup returns 0 on success and the loop skips on a result of 0 or less, so no shared-resource reason is ever recorded; the loop also runs over the number of requested shared resources instead of the unmet ones.)
3. When the job's shared-resource request can be met, the Scheduler Plugin Framework shall count the requested shared amounts as uncommitted for the attempt before trying any plan. It shall count them once per attempt, whatever the number of plans or buckets tried, and shall not release them when the attempt then pends or fails, as today (the attempt context, which holds them, is discarded by its caller, or shared with the next primary of the same COMPOUND job or of the same ALTER alternative; each ALTER alternative gets a fresh context).
4. If creating the pend-reason record, reading the uncommitted shared state, checking shared satisfaction, or counting shared amounts as uncommitted fails, the Scheduler Plugin Framework shall return that failure's negative error code, with no allocation on the attempt.

Requirement 3: Plan and host search

Objective: As a scheduler operator, I want jobs placed on exactly the hosts they are placed on today, so that the refactor cannot change scheduling outcomes.

Acceptance Criteria

1. The Scheduler Plugin Framework shall try candidate plans in the order the allocator plugin's plan generator produces them, and shall stop at the first plan whose every bucket is satisfied.
2. While trying a plan, the Scheduler Plugin Framework shall try, for each bucket in order, the candidate hosts in the shared host-iteration order. It shall claim the first candidate host that satisfies the bucket and has not already been claimed by an earlier bucket of the same plan.
3. If any bucket of a plan cannot be satisfied, the Scheduler Plugin Framework shall release the per-host resource debits of every claim already made for that plan before trying the next plan.
4. When the plan generator cannot be initialised or yields no plan whose buckets are all satisfied, the Scheduler Plugin Framework shall return the pending signal with no allocation on the attempt.
5. Before each bucket's host search, the Scheduler Plugin Framework shall set the attempt's slot count to that bucket's task count and its host count to 1, as parjob does today. For seqjob, whose only plan is one bucket of one task, the slot count and host count shall always be 1, in the job-level preparation pass and in every per-host check.
6. The Scheduler Plugin Framework shall clear the per-host request before each candidate host is considered, for both seqjob and parjob, so that no earlier bucket, earlier candidate host, or earlier primary of the same COMPOUND or ALTER job (bug-1883) affects which hosts the host search skips. (Intentional change, decided 2026-09-30: today parjob clears it once per bucket and seqjob never does, so after the first rejected candidate both pre-filter later candidates against the request that candidate left behind and skip short hosts without a reason.)
7. For every bucket search, in both allocators, the Scheduler Plugin Framework shall record a "no resource" pend reason, naming the host and the resource, for every candidate host that matches the job's host selection, is not already claimed in the plan, and that the existing per-host check finds short of a requested resource, and shall leave the job's main pend reason as it is today. (Intentional change: today such reasons appear only for the first rejected candidate of a search and for later candidates that pass the stale pre-filter.)
8. The Scheduler Plugin Framework shall support plans with any number of buckets up to the job's task count, with no fixed upper limit on the number of buckets that a large task count could exceed.

Requirement 4: Result assembly

Objective: As a scheduler maintainer, I want the successful allocation result built the same way for both allocators, so that dispatch, commit, and snapshot consumers see identical allocation records.

Acceptance Criteria

1. When every bucket of a plan is satisfied, the Scheduler Plugin Framework shall build one host allocation per claim, in claim order. Each is made from the claimed host's name and that host's resource request as it stood when the claim was made.
2. When every bucket of a plan is satisfied and the job requests shared resources, the Scheduler Plugin Framework shall attach one shared allocation record per entry of the attempt's shared-resource request, including entries added by limits. When that request is empty, it shall attach no shared records.
3. The Scheduler Plugin Framework shall produce allocation results as seqjob and parjob do today: job identifier 0 (filled in by later stages), empty reservation name on every resource record, and no duration.
4. When a plan is satisfied, the Scheduler Plugin Framework shall set the attempt's allocation to the newly built result and return a positive result.
5. When seqjob or parjob is given an input that reaches it today, the Scheduler Plugin Framework shall produce an allocation result identical to that allocator's current result: same host count, host names, per-host resource entries and values, shared records, and return code.

Requirement 5: Guards and error handling

Objective: As a scheduler maintainer, I want every error and out-of-contract input handled exactly as the fixed bugs require, so that no past fix regresses.

Acceptance Criteria

1. If the scheduling core, the plugin's scheduler back-reference, or the attempt's primary resource request is missing, the Scheduler Plugin Framework shall return an invalid-argument error without creating a pend-reason record, for both allocators. (Hardening for seqjob, whose current behaviour is undefined.)
2. If parjob receives a primary resource request whose task count is 1 or less, the Scheduler Plugin Framework shall return the pending signal without creating a pend-reason record or checking resources (bug-1767). parjob's acceptance guard provides this (Requirement 1 AC 4), after the missing-input check of AC 1.
3. The Scheduler Plugin Framework shall let seqjob accept every primary resource request that reaches it, whatever its task count, as today.
4. If building any part of the allocation result fails for lack of memory, the Scheduler Plugin Framework shall return an out-of-memory error, leave no allocation on the attempt, and free every partly built piece (bug-1882).
5. If a failure for lack of memory happens after one or more claims were made, the Scheduler Plugin Framework shall release the per-host resource debits of every claim not yet turned into a host allocation, and of the candidate host whose claim could not be recorded, for both allocators. (Hardening for seqjob, which today leaves its single debit in place until the attempt's context is discarded.)
6. The Scheduler Plugin Framework shall set the attempt's allocation only when it returns a positive result, so that callers merging COMPOUND/ALTER results keep publishing only fully satisfied jobs (bug-1766).
7. When an attempt that initialised the plan generator exits (success, pending, or error), the Scheduler Plugin Framework shall finish the plan generator and release every per-claim resource-request copy it holds.
8. The Scheduler Plugin Framework shall size each primary's allocation from that primary's own resource request, including primaries nested in COMPOUND and ALTER jobs (bug-1768).

Requirement 6: Plan-generator initialisation contract

Objective: As a scheduler maintainer, I want plan generators to receive the scheduling core, which exists in both modes, rather than a scheduler object that simulation mode has only as a stand-in (a partial wrapper kept for building allocation records, sim_alloc.c), so that the contract is honest in every mode.

Acceptance Criteria

1. The Scheduler Plugin Framework shall initialise an allocator plugin's plan generator with the primary resource request and the scheduling core.
2. The Scheduler Plugin Framework shall produce identical plan sequences, for seqjob and for parjob, before and after the change to the initialisation parameter.
3. While running in simulation mode, the Scheduler Plugin Framework shall run allocation through the same framework routine and plan generators as normal mode, with no scheduler object passed to any plan generator.

Requirement 7: Equivalence evidence and closure

Objective: As a project maintainer, I want test evidence that nothing changed beyond the intentional changes listed in the Introduction, and the tracked item closed, so that the refactor can be accepted with confidence.

Acceptance Criteria

1. The Scheduler Plugin Framework shall pass, unchanged in their expected outcomes, the existing seqjob allocator, seqjob rewrap-equivalence, parjob combinatorial, parjob end-to-end multi-slot, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, simulation limit fan-out, election simulation fixture, and snapshot-load allocator regression tests. Test changes are limited to how the allocation routine is reached and the parameter type of plan-generator initialisation.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts (for both allocators, the per-host reasons given by Requirement 3 AC 7 rather than today's; for shared resources, the reasons given by Requirement 2 AC 2), mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
3. The Scheduler Plugin Framework shall keep the existing check that pend-reason records are created only through the shared pend-reason factory, and that check shall cover the new framework allocation code.
4. When this feature is complete, issues/new_requirements.md item 4 shall be marked resolved, with a reference to this spec.
5. When this feature is complete, the scheduler state-equivalence invariant audit shall report no new violations in the allocation path.