lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(spec): scheduler-allocator-runner and scheduler-sim-decision-trace requirements

Browse source 
scheduler-allocator-runner: brief and requirements for moving the duplicated
seqjob/parjob my_alloc() body into one framework-owned allocator runner.

scheduler-sim-decision-trace: brief and requirements for a per-job decision
trace in the simulator REPL (converged after 9 review rounds). The
decision-reasoning seed capability in scheduler-sim-diagnostics now points to
this spec.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-29 08:57:23 +00:00
parent c89e4e4520
commit b283f3b93c
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 554 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

129
.kiro/specs/scheduler-allocator-runner/brief.md Normal file
View file

@ -0,0 +1,129 @@
# Brief: scheduler-allocator-runner
## Problem
Scheduler plugin authors and maintainers. `seqjob_allocator.c` (`my_alloc`,
line 213, ~215 lines) and `parjob_allocator.c` (`my_alloc`, line 545, ~220
lines) each own the *entire* allocation algorithm, although only plan
generation is plugin-specific. ~100+ lines are duplicated verbatim (lazy
pend-reason, `sched_ctx_get_uncomit_shared`, `sched_check_shared_satisfy` +
`PREASON_NO_RESOURCE` fill, add-uncommitted-shared, plan-iterator loop
skeleton, commit tail with `sched_make_allocs` + `-ENOMEM` cleanup chain).
Every bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new
allocator plugin must copy the whole `my_alloc()`.
Driver: `issues/new_requirements.md` item 4.
## Current State
- Framework dispatch `sched_make_primary_alloc_()` (`scheduler.c:235`) picks
`core->seqjob_allocator` (`maxTasks == 1`) or `core->parjob_allocator`
and calls `plugin->job_alloc_fn(plugin, ctx)`.
- Plan generation is already abstracted behind `xce_alloc_plan_iter_t`
(`include/sched/alloc_iter.h`); host scan behind `xce_host_iter_t`.
- The picker side already uses a framework-owned iface pattern
(`xce_job_picker_iface_t`, `plugin->picker_iface`, `plugin.h:88`).
- `alloc_iter.h` `init`'s third parameter is `xce_scheduler_t*`, unused by
both implementations (`(void)sched`); sim mode has no real scheduler.
- parjob has `parjob_try_plan_` / `parjob_rollback_claims_` /
`parjob_snapshot_host_req_` claim bookkeeping; seqjob keeps a single `w`.
## Desired Outcome
- Neither plugin defines `my_alloc()`; both register
`.job_alloc_fn = sched_allocator_run` and supply only an
`xce_sched_allocator_iface_t` (optional `accepts` guard + plan iterator)
plus their existing `prepare_allocation`.
- One framework function owns shared-resource prep, the bucket -> host-claim
loop with rollback, and the commit tail.
- Allocation decisions are bit-for-bit identical to today for every existing
test and sim-parity fixture.
- `new_requirements.md` item 4 closed.
## Approach
**Framework runner, lean** (chosen 2026-09-28). The design proposal of
2026-09-23 (§5), with the optional `commit` hook left out because nothing
uses it yet:
- `xce_sched_allocator_iface_t { accepts; plan_init; plan_next; plan_fini; }`
in `plugin.h`; `allocator_iface` pointer on `xce_sched_plugin_t` with the
same ownership as `picker_iface`.
- `sched_allocator_run()` = guards -> `accepts` -> `sched_alloc_prepare_shared_()`
-> plan loop -> `sched_try_plan_()` (promoted `parjob_try_plan_`) ->
`sched_finalize_job_alloc_()` (one `make_host_alloc` per claim ->
`make_job_alloc` -> `allocs_shared` block).
- Retype `alloc_iter.h` `init`'s third parameter to `xce_sched_core_t*`.
- Staged migration keeps the suite green: (1) extract shared helpers
used by both existing `my_alloc()`s; (2) runner + iface, parjob first;
(3) seqjob; (4) retype `plan_init`; (5) close item 4.
Rejected: adding the `commit` override hook (an unused extension point that
would need its own tests); only extracting helpers (would not meet item 4's
goal that plugins no longer export `my_alloc()`).
## Scope
- **In**: new allocator iface + framework runner; shared prep/try-plan/
finalize helpers; delete both `my_alloc()`s and parjob's claim helpers;
retype `xce_alloc_plan_iter_t.init` to `xce_sched_core_t*`; equivalence
tests; update `issues/new_requirements.md` item 4.
- **Out**: see Out of Boundary.
## Boundary Candidates
- Plugin ABI (`plugin.h` iface + plugin template wiring)
- Framework allocation runner (new `plugin_alloc.c` or `scheduler.c`) —
shared-resource prep, claim loop/rollback, commit tail
- Plan-iterator contract (`alloc_iter.h` retype; seqjob/parjob plan iters)
## Out of Boundary
- Retyping `sched_make_allocs()` to `xce_sched_core_t*` and removing the
`(xce_scheduler_t*)plugin->sched` cast (follow-up; callers outside
allocators).
- Collapsing `sched_make_seqjob_allocation` / `sched_make_parjob_allocation`
dispatch in `scheduler.c` (optional, only if trivially falls out).
- The `commit` override hook.
- The `FIXME: no duration now` (NULL duration to `sched_make_allocs`):
carry it over unchanged, don't fix it here.
- Any change to plan ordering, host ordering, pend-reason content, or
preemption.
## Upstream / Downstream
- **Upstream**: `scheduler-allocation` (plan/host iterators, Req 4
combinatorial parjob, Req 9 no-regression); `scheduler-core`
(plugin/picker iface pattern, sim mode); `libxce-alloc-preason-equality`
(`job_alloc_eq` used by sim verify).
- **Downstream**: any future allocator plugin (it would need only a plan
iterator); `scheduler-preemption`; the follow-up `sched_make_allocs` retype.
## Existing Spec Touchpoints
- **Extends**: none (scheduler-allocation is implementation-complete; its
Req 9 is inherited as a constraint, not reopened).
- **Adjacent**: `scheduler-allocation`, `scheduler-core`,
`scheduler-sim-discrete-clock` (in flight, touches sim_loop — avoid
overlapping files), `libxce-alloc-preason-equality`.
## Constraints
- TDD: tests in `src/scheduler/testsuite/` first; full ASAN build via
`/xce-build scheduler`.
- Behavioral equivalence is mandatory. Preserve exactly: bug-1882 (half-built
alloc never left on `ctx->alloc`), bug-1767 (parjob `maxTasks <= 1`
returns 0 with no fabricated reason, and before the lazy reason is
created), bug-1768; `-ENOMEM` cleanup order of both current tails.
- Divergences found in discovery that the design MUST resolve explicitly
(they are not in the 2026-09-23 proposal):
1. parjob does `xvec_idval_fini/init(&ctx->alloc_req.host)` per bucket
(`parjob_allocator.c:452`); seqjob never resets it. Decide whether the
shared loop always resets, and prove it is a no-op for seqjob.
2. seqjob builds its halloc directly from `ctx->alloc_req.host`; under the
shared per-claim snapshot it gains a new `-ENOMEM` path. It must go
through the same rollback (`sched_ctx_uncommit_host`).
3. seqjob does not set `ctx->nslots/nhosts` in its loop. Its values come
from `seqjob_prepare_allocation` (1,1). `sched_try_alloc_on_host_()`
(`resource.c`) does not read them directly; check which hook does, and
that `bucket_slots[0] == 1` for seqjob.
4. seqjob has no `maxTasks` guard today. A `seqjob_accepts`
(`maxTasks == 1`) changes behaviour for any direct caller with another
value (several testsuite files reference `seqjob_allocator`). Either
make `accepts` NULL for seqjob or audit those callers.
5. Proposal-sketch defects to avoid: `sched_ctx_primary == NULL` typo;
`claims[plan->nbuckets]` VLA (heap-allocate or bound by `maxTasks`).
- Claim `host_req` snapshots are owned by the framework: fini on every exit
path (success, bucket failure, OOM).
- Verify with: `test_seqjob_*`, `test_parjob_*`, `test_seqjob_rewrap_equivalence`,
`test_maxtasks_wiring_*`, `test_sim_*_parity`, `test_sim_live_allocation`,
`test_snapshot_load_allocator_regression`; `/xce-audit-invariants scheduler`
(driver state-equivalence).

99
.kiro/specs/scheduler-allocator-runner/requirements.md Normal file
View file

@ -0,0 +1,99 @@
# Requirements Document
## Project Description (Input)
**Who has the problem:** scheduler plugin authors and maintainers of the XCE scheduler allocation path.
**Current situation:** `seqjob_allocator.c` (`my_alloc`, ~215 lines) and `parjob_allocator.c` (`my_alloc`, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its `PREASON_NO_RESOURCE` fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (`make_job_alloc` + `sched_make_allocs` + the `-ENOMEM` cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole `my_alloc()`. Plan generation is already abstracted behind `xce_alloc_plan_iter_t`, whose `init` takes an unused `xce_scheduler_t*` even though sim mode has no real scheduler. Tracked as `issues/new_requirements.md` item 4.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons, and error and `-ENOMEM` behaviour stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
## Introduction
The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same pend reasons, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined.
Terms:
- **Allocation attempt**: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
- **Plan**: one candidate way of splitting a job's tasks into buckets, each bucket placed on one host.
- **Claim**: one bucket satisfied on one host within the plan being tried.
- **Pending signal**: the allocator reports "not satisfiable now" (result 0) without an error.
## Boundary Context
- **In scope**: a framework-owned allocation routine used by both built-in allocators; the allocator plugin contract (plan generator, optional acceptance guard, and the existing preparation hook); removal of the per-plugin allocation routines; changing the plan-generator initialisation to take the scheduling core instead of the scheduler object; keeping every existing allocation, pend-reason, error and simulation-parity behaviour; closing `issues/new_requirements.md` item 4.
- **Out of scope**: a per-plugin override hook for assembling results; retyping the allocation-record constructor (`sched_make_allocs`) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
- **Adjacent expectations**: the scheduler's allocation dispatch keeps sending `maxTasks == 1` primaries to seqjob and every other primary to parjob, and keeps treating result `> 0` as satisfied, `0` as pending and `< 0` as error, with an allocation published only on `> 0`. COMPOUND/ALTER handling (bug-1766: a merged allocation is published only if every primary succeeds) and snapshot-restored schedobj construction (bug-1767) stay owned by their existing code. The plan and host iterator contracts from `scheduler-allocation` keep their current ordering semantics.
## Requirements
### Requirement 1: Framework-owned allocation
**Objective:** As a scheduler plugin author, I want the framework to run the allocation algorithm, so that an allocator plugin only has to describe how it generates candidate plans.
#### Acceptance Criteria
1. The Scheduler Plugin Framework shall provide one allocation routine that both built-in allocator plugins (seqjob and parjob) register as their job-allocation entry point.
2. The Scheduler Plugin Framework shall let an allocator plugin supply a candidate-plan generator (initialise, next plan, finish), an optional acceptance guard, and its existing per-allocation preparation hook, and nothing else, to take part in allocation.
3. Where an allocator plugin supplies no acceptance guard, the Scheduler Plugin Framework shall treat every primary resource request passed to that plugin as accepted.
4. The seqjob and parjob allocator plugins shall contain no private allocation routine that duplicates the shared-resource check, the plan/host search, or the result assembly.
5. The Scheduler Plugin Framework shall keep the allocator plugin contract as data the plugin registry does not free, with the same lifetime as the existing job-picker contract: it lives as long as the registered plugin, and the plugin does not release it separately.
### Requirement 2: Shared-resource precheck
**Objective:** As a scheduler operator, I want jobs whose cluster-wide (shared) resources are unavailable to pend with the same reasons as today, so that pend diagnostics do not change.
#### Acceptance Criteria
1. When an allocation attempt starts and the attempt has no pend-reason record yet, the Scheduler Plugin Framework shall create one before checking shared resources.
2. When the job's shared-resource request cannot be met, the Scheduler Plugin Framework shall add one "no resource" pend reason, with no host and naming the resource, for each unmet shared resource. It shall then return the pending signal without trying any host.
3. When the job's shared-resource request can be met, the Scheduler Plugin Framework shall count the requested shared amounts as uncommitted for the attempt before trying any plan. It shall count them once per attempt, whatever the number of plans or buckets tried.
4. If creating the pend-reason record, reading the uncommitted shared state, checking shared satisfaction, or counting shared amounts as uncommitted fails, the Scheduler Plugin Framework shall return that failure's negative error code, with no allocation on the attempt.
### Requirement 3: Plan and host search
**Objective:** As a scheduler operator, I want jobs placed on exactly the hosts they are placed on today, so that the refactor cannot change scheduling outcomes.
#### Acceptance Criteria
1. The Scheduler Plugin Framework shall try candidate plans in the order the allocator plugin's plan generator produces them, and shall stop at the first plan whose every bucket is satisfied.
2. While trying a plan, the Scheduler Plugin Framework shall try, for each bucket in order, the candidate hosts in the shared host-iteration order. It shall claim the first candidate host that satisfies the bucket and has not already been claimed by an earlier bucket of the same plan.
3. If any bucket of a plan cannot be satisfied, the Scheduler Plugin Framework shall release the per-host resource debits of every claim already made for that plan before trying the next plan.
4. When the plan generator cannot be initialised or yields no plan whose buckets are all satisfied, the Scheduler Plugin Framework shall return the pending signal with no allocation on the attempt.
5. The Scheduler Plugin Framework shall give each bucket the task count and single-host shape that parjob uses today when checking a candidate host, and shall leave seqjob's per-host checks with the one-task, one-host shape they have today.
6. While seqjob runs an attempt for a primary that shares its attempt context with an earlier primary of the same COMPOUND or ALTER job, the Scheduler Plugin Framework shall start seqjob's host search from that primary's own per-host request only, as it does today: the job-level preparation pass resets the per-host request for every primary (bug-1883, fixed in `c10f32cc`), so no earlier primary's per-host request reaches seqjob's host pre-filtering or pend reasons.
7. While parjob runs an attempt, the Scheduler Plugin Framework shall start every bucket's host search from an empty per-host request, as parjob does today.
8. The Scheduler Plugin Framework shall support plans with any number of buckets up to the job's task count, with no fixed upper limit on the number of buckets that a large task count could exceed.
### Requirement 4: Result assembly
**Objective:** As a scheduler maintainer, I want the successful allocation result built the same way for both allocators, so that dispatch, commit, and snapshot consumers see identical allocation records.
#### Acceptance Criteria
1. When every bucket of a plan is satisfied, the Scheduler Plugin Framework shall build one host allocation per claim, in claim order. Each is made from the claimed host's name and that host's resource request as it stood when the claim was made.
2. When every bucket of a plan is satisfied and the job requests shared resources, the Scheduler Plugin Framework shall attach one shared allocation record per requested shared resource. When the job requests none, it shall attach no shared records.
3. The Scheduler Plugin Framework shall produce allocation results with the same job identifier, empty reservation name, and absent duration that seqjob and parjob produce today.
4. When a plan is satisfied, the Scheduler Plugin Framework shall set the attempt's allocation to the newly built result and return a positive result.
5. When seqjob or parjob is given an input that reaches it today, the Scheduler Plugin Framework shall produce an allocation result identical to that allocator's current result: same host count, host names, per-host resource entries and values, shared records, and return code.
### Requirement 5: Guards and error handling
**Objective:** As a scheduler maintainer, I want every error and out-of-contract input handled exactly as the fixed bugs require, so that no past fix regresses.
#### Acceptance Criteria
1. If the scheduling core, the plugin's scheduler back-reference, or the attempt's primary resource request is missing, the Scheduler Plugin Framework shall return an invalid-argument error without creating a pend-reason record, for both allocators. (Hardening for seqjob, whose current behaviour is undefined.)
2. If parjob receives a primary resource request whose task count is 1 or less, the Scheduler Plugin Framework shall return the pending signal without creating a pend-reason record or checking resources (bug-1767).
3. The Scheduler Plugin Framework shall let seqjob accept every primary resource request that reaches it, whatever its task count, as today.
4. If building any part of the allocation result fails for lack of memory, the Scheduler Plugin Framework shall return an out-of-memory error, leave no allocation on the attempt, and free every partly built piece (bug-1882).
5. If a failure for lack of memory happens after one or more claims were made, the Scheduler Plugin Framework shall release the per-host resource debits of every claim not yet handed over to a built result, for both allocators. (Hardening for seqjob, which today leaves its single debit in place until the attempt's context is discarded.)
6. The Scheduler Plugin Framework shall set the attempt's allocation only when it returns a positive result, so that callers merging COMPOUND/ALTER results keep publishing only fully satisfied jobs (bug-1766).
7. When an attempt that initialised the plan generator exits (success, pending, or error), the Scheduler Plugin Framework shall finish the plan generator and release every per-claim resource-request copy it holds.
8. The Scheduler Plugin Framework shall size each primary's allocation from that primary's own resource request, including primaries nested in COMPOUND and ALTER jobs (bug-1768).
### Requirement 6: Plan-generator initialisation contract
**Objective:** As a scheduler maintainer, I want plan generators to receive the scheduling core, which exists in both modes, rather than a scheduler object that simulation mode lacks, so that the contract is honest in every mode.
#### Acceptance Criteria
1. The Scheduler Plugin Framework shall initialise an allocator plugin's plan generator with the primary resource request and the scheduling core.
2. The Scheduler Plugin Framework shall produce identical plan sequences, for seqjob and for parjob, before and after the change to the initialisation parameter.
3. While running in simulation mode, the Scheduler Plugin Framework shall run allocation through the same framework routine and plan generators as normal mode, with no scheduler object passed to any plan generator.
### Requirement 7: Equivalence evidence and closure
**Objective:** As a project maintainer, I want test evidence that nothing changed and the tracked item closed, so that the refactor can be accepted with confidence.
#### Acceptance Criteria
1. The Scheduler Plugin Framework shall pass, unchanged in their expected outcomes, the existing seqjob allocator, seqjob rewrap-equivalence, parjob combinatorial, parjob end-to-end multi-slot, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, simulation limit fan-out, election simulation fixture, and snapshot-load allocator regression tests. Test changes are limited to how the allocation routine is reached and the parameter type of plan-generator initialisation.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts, mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
3. The Scheduler Plugin Framework shall keep the existing check that pend-reason records are created only through the shared pend-reason factory, and that check shall cover the new framework allocation code.
4. When this feature is complete, `issues/new_requirements.md` item 4 shall be marked resolved, with a reference to this spec.
5. When this feature is complete, the scheduler state-equivalence invariant audit shall report no new violations in the allocation path.