myxce/.kiro/specs/scheduler-allocator-runner/requirements.md

Requirements Document

Project Description (Input)

Who has the problem: scheduler plugin authors and maintainers of the XCE scheduler allocation path.

Current situation: seqjob_allocator.c (my_alloc, ~215 lines) and parjob_allocator.c (my_alloc, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its PREASON_NO_RESOURCE fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (make_job_alloc + sched_make_allocs + the -ENOMEM cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole my_alloc(). Plan generation is already abstracted behind xce_alloc_plan_iter_t, whose init takes an unused xce_scheduler_t* even though sim mode has no real scheduler. Tracked as issues/new_requirements.md item 4.

What should change: the scheduler framework owns allocation. A new xce_sched_allocator_iface_t (an optional accepts guard plus plan_init/plan_next/plan_fini), referenced from xce_sched_plugin_t in the same way as picker_iface, lets one framework function, sched_allocator_run(), carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register .job_alloc_fn = sched_allocator_run, supply only their plan iterator and prepare_allocation, and no longer define my_alloc(). The third parameter of xce_alloc_plan_iter_t.init is retyped to xce_sched_core_t*. Allocation decisions, pend reasons, and error and -ENOMEM behaviour stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin commit hook and no sched_make_allocs() retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in brief.md → Constraints (the per-bucket alloc_req.host reset, seqjob's new OOM snapshot path, where nslots/nhosts are consumed, seqjob accepts for direct callers, and no VLA). Full context: .kiro/specs/scheduler-allocator-runner/brief.md.

Introduction

The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.

This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same pend reasons, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined.

Terms:

• Allocation attempt: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
• Plan: one candidate way of splitting a job's tasks into buckets, each bucket placed on one host.
• Claim: one bucket satisfied on one host within the plan being tried.
• Pending signal: the allocator reports "not satisfiable now" (result 0) without an error.

Boundary Context

• In scope: a framework-owned allocation routine used by both built-in allocators; the allocator plugin contract (plan generator, optional acceptance guard, and the existing preparation hook); removal of the per-plugin allocation routines; changing the plan-generator initialisation to take the scheduling core instead of the scheduler object; keeping every existing allocation, pend-reason, error and simulation-parity behaviour; closing issues/new_requirements.md item 4.
• Out of scope: a per-plugin override hook for assembling results; retyping the allocation-record constructor (sched_make_allocs) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
• Adjacent expectations: the scheduler's allocation dispatch keeps sending maxTasks == 1 primaries to seqjob and every other primary to parjob, and keeps treating result > 0 as satisfied, 0 as pending and < 0 as error, with an allocation published only on > 0. COMPOUND/ALTER handling (bug-1766: a merged allocation is published only if every primary succeeds) and snapshot-restored schedobj construction (bug-1767) stay owned by their existing code. The plan and host iterator contracts from scheduler-allocation keep their current ordering semantics.

Requirements

Requirement 1: Framework-owned allocation

Objective: As a scheduler plugin author, I want the framework to run the allocation algorithm, so that an allocator plugin only has to describe how it generates candidate plans.

Acceptance Criteria

1. The Scheduler Plugin Framework shall provide one allocation routine that both built-in allocator plugins (seqjob and parjob) register as their job-allocation entry point.
2. The Scheduler Plugin Framework shall let an allocator plugin supply a candidate-plan generator (initialise, next plan, finish), an optional acceptance guard, and its existing per-allocation preparation hook, and nothing else, to take part in allocation.
3. Where an allocator plugin supplies no acceptance guard, the Scheduler Plugin Framework shall treat every primary resource request passed to that plugin as accepted.
4. The seqjob and parjob allocator plugins shall contain no private allocation routine that duplicates the shared-resource check, the plan/host search, or the result assembly.
5. The Scheduler Plugin Framework shall keep the allocator plugin contract as data the plugin registry does not free, with the same lifetime as the existing job-picker contract: it lives as long as the registered plugin, and the plugin does not release it separately.

Requirement 2: Shared-resource precheck

Objective: As a scheduler operator, I want jobs whose cluster-wide (shared) resources are unavailable to pend with the same reasons as today, so that pend diagnostics do not change.

Acceptance Criteria

1. When an allocation attempt starts and the attempt has no pend-reason record yet, the Scheduler Plugin Framework shall create one before checking shared resources.
2. When the job's shared-resource request cannot be met, the Scheduler Plugin Framework shall add one "no resource" pend reason, with no host and naming the resource, for each unmet shared resource. It shall then return the pending signal without trying any host.
3. When the job's shared-resource request can be met, the Scheduler Plugin Framework shall count the requested shared amounts as uncommitted for the attempt before trying any plan. It shall count them once per attempt, whatever the number of plans or buckets tried.
4. If creating the pend-reason record, reading the uncommitted shared state, checking shared satisfaction, or counting shared amounts as uncommitted fails, the Scheduler Plugin Framework shall return that failure's negative error code, with no allocation on the attempt.

Requirement 3: Plan and host search

Objective: As a scheduler operator, I want jobs placed on exactly the hosts they are placed on today, so that the refactor cannot change scheduling outcomes.

Acceptance Criteria

1. The Scheduler Plugin Framework shall try candidate plans in the order the allocator plugin's plan generator produces them, and shall stop at the first plan whose every bucket is satisfied.
2. While trying a plan, the Scheduler Plugin Framework shall try, for each bucket in order, the candidate hosts in the shared host-iteration order. It shall claim the first candidate host that satisfies the bucket and has not already been claimed by an earlier bucket of the same plan.
3. If any bucket of a plan cannot be satisfied, the Scheduler Plugin Framework shall release the per-host resource debits of every claim already made for that plan before trying the next plan.
4. When the plan generator cannot be initialised or yields no plan whose buckets are all satisfied, the Scheduler Plugin Framework shall return the pending signal with no allocation on the attempt.
5. The Scheduler Plugin Framework shall give each bucket the task count and single-host shape that parjob uses today when checking a candidate host, and shall leave seqjob's per-host checks with the one-task, one-host shape they have today.
6. While seqjob runs an attempt for a primary that shares its attempt context with an earlier primary of the same COMPOUND or ALTER job, the Scheduler Plugin Framework shall start seqjob's host search from that primary's own per-host request only, as it does today: the job-level preparation pass resets the per-host request for every primary (bug-1883, fixed in c10f32cc), so no earlier primary's per-host request reaches seqjob's host pre-filtering or pend reasons.
7. While parjob runs an attempt, the Scheduler Plugin Framework shall start every bucket's host search from an empty per-host request, as parjob does today.
8. The Scheduler Plugin Framework shall support plans with any number of buckets up to the job's task count, with no fixed upper limit on the number of buckets that a large task count could exceed.

Requirement 4: Result assembly

Objective: As a scheduler maintainer, I want the successful allocation result built the same way for both allocators, so that dispatch, commit, and snapshot consumers see identical allocation records.

Acceptance Criteria

1. When every bucket of a plan is satisfied, the Scheduler Plugin Framework shall build one host allocation per claim, in claim order. Each is made from the claimed host's name and that host's resource request as it stood when the claim was made.
2. When every bucket of a plan is satisfied and the job requests shared resources, the Scheduler Plugin Framework shall attach one shared allocation record per requested shared resource. When the job requests none, it shall attach no shared records.
3. The Scheduler Plugin Framework shall produce allocation results with the same job identifier, empty reservation name, and absent duration that seqjob and parjob produce today.
4. When a plan is satisfied, the Scheduler Plugin Framework shall set the attempt's allocation to the newly built result and return a positive result.
5. When seqjob or parjob is given an input that reaches it today, the Scheduler Plugin Framework shall produce an allocation result identical to that allocator's current result: same host count, host names, per-host resource entries and values, shared records, and return code.

Requirement 5: Guards and error handling

Objective: As a scheduler maintainer, I want every error and out-of-contract input handled exactly as the fixed bugs require, so that no past fix regresses.

Acceptance Criteria

1. If the scheduling core, the plugin's scheduler back-reference, or the attempt's primary resource request is missing, the Scheduler Plugin Framework shall return an invalid-argument error without creating a pend-reason record, for both allocators. (Hardening for seqjob, whose current behaviour is undefined.)
2. If parjob receives a primary resource request whose task count is 1 or less, the Scheduler Plugin Framework shall return the pending signal without creating a pend-reason record or checking resources (bug-1767).
3. The Scheduler Plugin Framework shall let seqjob accept every primary resource request that reaches it, whatever its task count, as today.
4. If building any part of the allocation result fails for lack of memory, the Scheduler Plugin Framework shall return an out-of-memory error, leave no allocation on the attempt, and free every partly built piece (bug-1882).
5. If a failure for lack of memory happens after one or more claims were made, the Scheduler Plugin Framework shall release the per-host resource debits of every claim not yet handed over to a built result, for both allocators. (Hardening for seqjob, which today leaves its single debit in place until the attempt's context is discarded.)
6. The Scheduler Plugin Framework shall set the attempt's allocation only when it returns a positive result, so that callers merging COMPOUND/ALTER results keep publishing only fully satisfied jobs (bug-1766).
7. When an attempt that initialised the plan generator exits (success, pending, or error), the Scheduler Plugin Framework shall finish the plan generator and release every per-claim resource-request copy it holds.
8. The Scheduler Plugin Framework shall size each primary's allocation from that primary's own resource request, including primaries nested in COMPOUND and ALTER jobs (bug-1768).

Requirement 6: Plan-generator initialisation contract

Objective: As a scheduler maintainer, I want plan generators to receive the scheduling core, which exists in both modes, rather than a scheduler object that simulation mode lacks, so that the contract is honest in every mode.

Acceptance Criteria

1. The Scheduler Plugin Framework shall initialise an allocator plugin's plan generator with the primary resource request and the scheduling core.
2. The Scheduler Plugin Framework shall produce identical plan sequences, for seqjob and for parjob, before and after the change to the initialisation parameter.
3. While running in simulation mode, the Scheduler Plugin Framework shall run allocation through the same framework routine and plan generators as normal mode, with no scheduler object passed to any plan generator.

Requirement 7: Equivalence evidence and closure

Objective: As a project maintainer, I want test evidence that nothing changed and the tracked item closed, so that the refactor can be accepted with confidence.

Acceptance Criteria

1. The Scheduler Plugin Framework shall pass, unchanged in their expected outcomes, the existing seqjob allocator, seqjob rewrap-equivalence, parjob combinatorial, parjob end-to-end multi-slot, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, simulation limit fan-out, election simulation fixture, and snapshot-load allocator regression tests. Test changes are limited to how the allocation routine is reached and the parameter type of plan-generator initialisation.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts, mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
3. The Scheduler Plugin Framework shall keep the existing check that pend-reason records are created only through the shared pend-reason factory, and that check shall cover the new framework allocation code.
4. When this feature is complete, issues/new_requirements.md item 4 shall be marked resolved, with a reference to this spec.
5. When this feature is complete, the scheduler state-equivalence invariant audit shall report no new violations in the allocation path.