lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(scheduler-allocator-runner): characterize the parjob allocator (task 2.2)

Browse source 
15 tests drive the real parjob plugin through job_alloc_fn: success,
short shared, pending on hosts, mid-plan rollback, task counts 0/1,
missing inputs, six OOM steps with their debits, COMPOUND. Values the
runner changes on purpose sit in *_today tests.

Spec: the task showed that parjob records host reasons only for a bucket
search's first rejected candidate (the rejected host's request stays in
alloc_req.host and the host iterator pre-filters later hosts against
it). Decision: the runner clears alloc_req.host before every candidate,
for both allocators, so every short host gets its reasons (Req 3.6/3.7);
publishing reasons only when they change is a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-30 08:40:35 +00:00
parent 91a8580948
commit b4470b2b94
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 874 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

65
.kiro/specs/scheduler-allocator-runner/brief.md
View file

@ -117,40 +117,37 @@ goal that plugins no longer export `my_alloc()`).
the seqjob OOM rollback of item 2 (Req 5 AC 5).
- Divergences found in discovery that the design MUST resolve explicitly
(they are not in the 2026-09-23 proposal):
1. **Decided 2026-09-29: the shared loop resets `ctx->alloc_req.host` to
empty before every bucket's host scan, for both allocators.** Background:
the per-host request used by the authoritative check is already rebuilt
for every candidate (`sched_try_alloc_on_host_()` -> `plugin_prepare_
allocation()` with `ctx->worker` set resets and refills it,
`plugin.c:639`). The per-bucket reset only affects the host iterator's
pre-filter (`host_iter.c:143`), which reads `alloc_req.host` before that
rebuild and silently skips hosts that fail. parjob already resets
(`parjob_allocator.c:479`), so its pre-filter does no resource filtering.
seqjob does not, so its pre-filter reads stale content: the job-level
`slots:1` for the first candidate, then the last rejected candidate's
full request. Resetting for seqjob too drops the stale-state dependency
and makes seqjob's pre-filter a resource no-op, like parjob's.
**Accepted consequence (intentional change):** host choice is unchanged
(a host the stale pre-filter rejects also fails the authoritative check
while the per-host request is host-independent; where a limit makes it
host-specific, the stale filter could wrongly skip a host, which the
reset fixes). But a pending seqjob job now gets a `PREASON_NO_RESOURCE`
host/resource reason from the authoritative check
(`resource.c:575`) for every host that passes the select filter and
lacks resources, not only for hosts the stale pre-filter let through.
`main_reason` is unaffected (host/resource reasons go to
`res_based_reasons`, `lib/preason.c:147`), so sim parity
(`sim_verify_pend_main_reason_eq`) is unaffected; no testsuite file
asserts `PREASON_*` host reasons (checked 2026-09-29).
Also accepted (found in design review): seqjob now runs the per-host
prepare fan-out on those hosts too, so the limit plugin may instantiate
a host limit's virtual resource and set its value on the worker earlier
than before (decision-neutral, lazy creation moves earlier), at the cost
of one prepare + check per short host per pending seqjob attempt.
Rejected: keeping the reset parjob-only (preserves the stale-state
quirk in shared code); staging each bucket's real request before the
scan so the pre-filter filters correctly (changes parjob's pend
reasons too; a separate follow-up if wanted).
1. **Decided 2026-09-29, revised 2026-09-30 (task 2.2 finding, user
decision): the runner clears `ctx->alloc_req.host` before every
candidate host (at the start of each bucket and after every rejected
candidate), for both allocators.** The per-host request used by the
authoritative check is rebuilt for every candidate anyway
(`plugin_prepare_allocation()` with `ctx->worker` set, `plugin.c:639`).
What the clear changes is the host iterator's pre-filter
(`host_iter.c:143`), which reads `alloc_req.host` before that rebuild and
silently skips hosts that fail. The 2026-09-29 plan (clear once per
bucket, as parjob does) turned out not to be enough: task 2.2's
characterization showed that a rejected candidate leaves its own request
behind, so parjob today records reasons only for the first rejected
candidate of a bucket search, and a once-per-bucket clear would give
seqjob the same partial reasons. With a clear before every candidate,
the pre-filter only applies the select expression, and every
selection-matching unclaimed short host gets its host/resource reasons,
in both allocators.
**Accepted consequences (intentional changes):** host choice, main
reason and decisions are unchanged (a host the stale pre-filter rejects
also fails the authoritative check while the per-host request is
host-independent; where a limit makes it host-specific, the stale filter
could wrongly skip a host, which the clear fixes). Pending jobs of both
allocators carry more host/resource reasons (`res_based_reasons`; not
`main_reason`, so the unsatisfiable cache and sim parity are unaffected).
The per-host prepare fan-out, including the limit plugin's lazy
host-limit instantiation, now runs on every short host (decision-neutral;
one prepare + check per short host per bucket search). Larger reason sets
are published and persisted; publishing reasons only when they change is
a follow-up (user, 2026-09-30), out of scope here.
Rejected: clearing once per bucket (partial reasons, still stale-state
dependent); keeping today's per-plugin behaviour.
2. **Decided (requirements Req 5 AC 5):** seqjob builds its halloc directly
from `ctx->alloc_req.host`; under the shared per-claim snapshot it gains
a new `-ENOMEM` path, which goes through the same rollback

22
.kiro/specs/scheduler-allocator-runner/design.md
View file

@ -5,7 +5,7 @@
**Users**: Scheduler plugin authors (a new allocator needs only a plan iterator) and scheduler maintainers (an allocation fix lands once).
**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the seqjob per-bucket reset are under System Flows → Consequences.
**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the per-candidate reset are under System Flows → Consequences.
### Goals
- One allocation algorithm, used by seqjob and parjob in normal and simulation mode (1.1, 6.3).
@ -18,12 +18,13 @@
- Merging `sched_make_seqjob_allocation()` / `sched_make_parjob_allocation()` dispatch.
- Adding an allocation duration (the `FIXME: no duration now` stays).
- Fixing bug-1892 (`unsatisfy_()` reads uncommitted amounts by position).
- Publishing pend reasons only when they change (follow-up to the larger per-host reason sets of 3.7).
## Boundary Commitments
### This Spec Owns
- `xce_sched_allocator_iface_t` (shape, ownership, lifetime) and the `allocator_iface` field of `xce_sched_plugin_t`.
- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-bucket `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-candidate `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
- The `xce_alloc_plan_iter_t.init` parameter type.
- The seqjob/parjob plugin wiring (templates, iface instances, `parjob_accepts`) and the removal of their private allocation code.
- Equivalence and runner tests; the pend-reason factory audit's file list; `issues/new_requirements.md` item 4.
@ -145,7 +146,7 @@ flowchart TD
Finalize -->|ok| Done[publish ctx alloc then fini and return 1]
```
**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan; set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.
**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan (and reset `alloc_req.host` again after every rejected candidate, before the next `next()`); set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.
**Key decisions**: the reset and the `nslots`/`nhosts` assignment are unconditional (brief items 1 and 3); every return after `plan_init` succeeded calls `plan_fini` and frees the claims buffer; claim snapshots are freed by whichever function owns them at that point (see AllocRunner → State Management).
@ -168,8 +169,8 @@ flowchart TD
| 3.3 | Roll back claims on unsatisfied bucket | ClaimSearch | `sched_ctx_uncommit_host` | TryPlan |
| 3.4 | Init failure / no plan = pending | AllocRunner | `plan_init` | Pend1, Fini0 |
| 3.5 | `nslots` = bucket slots, `nhosts` = 1 | ClaimSearch | — | TryPlan |
| 3.6 | Empty per-host request per bucket | ClaimSearch | — | TryPlan |
| 3.7 | seqjob per-host reasons complete | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
| 3.6 | Empty per-host request before each candidate | ClaimSearch | — | TryPlan |
| 3.7 | Per-host reasons complete, both allocators | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
| 3.8 | No fixed bucket limit | AllocRunner (claims buffer) | — | Grow |
| 4.1 | One host alloc per claim, claim order, snapshot request | ResultAssembly | `sched_make_allocs`, `make_host_alloc` | Finalize |
| 4.2 | Shared record per shared-request entry | ResultAssembly | `sched_make_allocs` | Finalize |
@ -318,7 +319,7 @@ static int sched_snapshot_host_req_(xvec_idval_t* dst, const xvec_idval_t* src)
```
- **Preconditions**: `claims` has capacity `>= plan->nbuckets`.
- **Postconditions**: on 0 or `< 0`, every per-host debit made in this call is released (3.3, 5.5) and no snapshot remains.
- Behaviour is parjob's current `parjob_try_plan_()` exactly, with `plan->ctx = ctx`, the per-bucket reset (3.6) and the per-candidate `nslots`/`nhosts` assignment (3.5) now applied to every plugin. seqjob's complete per-host reasons (3.7) follow from 3.6: the pre-filter no longer skips hosts on stale state, so `sched_try_alloc_on_host_()` records its reasons for every selection-matching host.
- Behaviour is parjob's current `parjob_try_plan_()` with `plan->ctx = ctx` and the per-candidate `nslots`/`nhosts` assignment (3.5), applied to every plugin, and one change: `alloc_req.host` is cleared at the start of each bucket AND after every rejected candidate, i.e. before every `host_iter->next()` call (3.6). Clearing only once per bucket (parjob today) is not enough: a rejected candidate leaves its own per-host request in `alloc_req.host`, and the host iterator's pre-filter (`host_iter.c`) would then skip later short hosts silently. With the request always empty when `next()` pre-filters, the pre-filter does select-expression filtering only, every selection-matching unclaimed host reaches `sched_try_alloc_on_host_()`, and each short one gets its reasons (3.7).
#### ResultAssembly
@ -367,10 +368,11 @@ static const xce_sched_allocator_iface_t seqjob_alloc_iface = {
```
- `prepare_allocation_hook`, `init_fn`, `fini_fn`, `*_new_plugin()`, `register_*()` unchanged; sim registration in `sim_alloc.c` therefore reaches the runner unchanged (6.3).
### Consequences of the per-bucket reset for seqjob (3.6)
Because seqjob's pre-filter no longer skips hosts on stale state, a pending seqjob attempt now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching host that is short of resources, as parjob already does. Besides the per-host pend reasons (3.7), this means:
### Consequences of the per-candidate reset (3.6, both allocators)
Because the host iterator's pre-filter no longer skips hosts on stale state, a pending attempt of either allocator now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching unclaimed host that is short of resources. Today that happens only for the first rejected candidate of each bucket search (parjob) or for candidates that pass the stale filter (seqjob). Besides the complete per-host pend reasons (3.7), this means:
- **Limit plugin side effects** (`limit.c` `match_job_host_limit_`): for a host with a matching host-based limit it allocates the job's `obj->hlimits[worker]` list, may instantiate the limit's virtual resource in `resmgr` (`instantiate_limit`) and writes the limit value into `worker->resvals`. This is decision-neutral: any later check of that host by any job runs the same lazy instantiation before reading the value, so it only moves when the instance is created. It happens identically in normal and simulation mode (driver state-equivalence holds).
- **Cost**: one prepare fan-out plus satisfy check per short host per pending seqjob attempt, where the stale pre-filter used to reject some hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
- **Cost**: one prepare fan-out plus satisfy check per short host per bucket search, where the stale pre-filter used to reject later hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
- **Pend-reason volume**: a pending job can now carry one reason per short host (per bucket search for parjob). These reasons are published to the controller and persisted with the job. Publishing reasons only when they change is a recorded follow-up (Non-Goals), not part of this spec.
## Data Models
No persisted or wire data changes. In-memory only: `sched_alloc_claim_t` (attempt-scoped) and the `allocator_iface` pointer (static). Snapshot/WAL/recording field-completeness is not affected (scheduler audit dimension 4: no change).
@ -406,7 +408,7 @@ TDD order follows the staged migration (research.md): characterization tests are
- `test_alloc_runner.c` (white-box `#include` of `core.c`, `resource.c`, `schedobj.c`, `host_iter.c`, `alloc_runner.c`; stubbed `sched_make_allocs` with OOM injection; stubbed `plugin_prepare_allocation()` that records `ctx->nslots`/`ctx->nhosts`/`ctx->alloc_req.host.count` on each call and stages a fixed per-host request, the same link-stub convention `test_resource_accounting.c` uses; an in-test fake iface whose plans are scripted):
- guards: each missing input → `-EINVAL`, `ctx->reason` still NULL (5.1); rejecting `accepts` → 0, no reason, `plan_init` not called (1.4); NULL `accepts` reaches `plan_init` (1.3).
- shared: two shared resources, one short → exactly one host-less `NO_RESOURCE` reason naming it (2.2, bug-1893); met → uncommitted added once across a 3-plan attempt (2.3).
- search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty when each bucket's host scan starts, observed through a candidate whose stale request would have been rejected by the pre-filter (3.6); a plan with more buckets than a first small plan triggers buffer growth (3.8).
- search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty before every candidate's pre-filter: with three short hosts in one bucket, all three get host/resource reasons (today only the first would) (3.6, 3.7); a plan with more buckets than a first small plan triggers buffer growth (3.8).
- errors: snapshot OOM, host-alloc OOM at claim 0 and claim k, job-alloc OOM, shared-records OOM → `-ENOMEM`, `ctx->alloc == NULL`, released debits as in the Error table, ASAN clean (5.4, 5.5, 5.6, 5.7).
- `test_parjob_combinatorial.c`: plan sequences identical for `NULL` and poisoned core (6.2).

12
.kiro/specs/scheduler-allocator-runner/requirements.md
View file

@ -5,12 +5,12 @@
**Current situation:** `seqjob_allocator.c` (`my_alloc`, ~215 lines) and `parjob_allocator.c` (`my_alloc`, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its `PREASON_NO_RESOURCE` fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (`make_job_alloc` + `sched_make_allocs` + the `-ENOMEM` cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole `my_alloc()`. Plan generation is already abstracted behind `xce_alloc_plan_iter_t`, whose `init` takes an unused `xce_scheduler_t*` even though sim mode has no real scheduler. Tracked as `issues/new_requirements.md` item 4.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons (except seqjob's per-host reasons, which become complete, Requirement 3 AC 6-7, and the shared-resource reasons bug-1893 never recorded, Requirement 2 AC 2), and error and `-ENOMEM` behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons (except the per-host reasons of both allocators, which become complete, Requirement 3 AC 6-7, and the shared-resource reasons bug-1893 never recorded, Requirement 2 AC 2), and error and `-ENOMEM` behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
## Introduction
The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the sequential-allocator host pend reasons of Requirement 3 AC 6 and AC 7, which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which today are never recorded (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the per-host pend reasons of both allocators (Requirement 3 AC 6 and AC 7), which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which today are never recorded (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.
Terms:
- **Allocation attempt**: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
@ -20,7 +20,7 @@ Terms:
## Boundary Context
- **In scope**: a framework-owned allocation routine used by both built-in allocators; the allocator plugin contract (plan generator, optional acceptance guard, and the existing preparation hook); removal of the per-plugin allocation routines; changing the plan-generator initialisation to take the scheduling core instead of the scheduler object; keeping every existing allocation, pend-reason, error and simulation-parity behaviour apart from the intentional changes listed in the Introduction; closing `issues/new_requirements.md` item 4.
- **Out of scope**: a per-plugin override hook for assembling results; retyping the allocation-record constructor (`sched_make_allocs`) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
- **Out of scope**: a per-plugin override hook for assembling results; retyping the allocation-record constructor (`sched_make_allocs`) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); publishing pend reasons only when they change (a follow-up: complete per-host reasons make each pending job's reason set larger); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
- **Adjacent expectations**: the scheduler's allocation dispatch keeps sending `maxTasks == 1` primaries to seqjob and every other primary to parjob, and keeps treating result `> 0` as satisfied, `0` as pending and `< 0` as error, with an allocation published only on `> 0`. COMPOUND/ALTER handling (bug-1766: a merged allocation is published only if every primary succeeds) and snapshot-restored schedobj construction (bug-1767) stay owned by their existing code. The plan and host iterator contracts from `scheduler-allocation` keep their current ordering semantics.
## Requirements
@ -54,8 +54,8 @@ Terms:
3. If any bucket of a plan cannot be satisfied, the Scheduler Plugin Framework shall release the per-host resource debits of every claim already made for that plan before trying the next plan.
4. When the plan generator cannot be initialised or yields no plan whose buckets are all satisfied, the Scheduler Plugin Framework shall return the pending signal with no allocation on the attempt.
5. Before each bucket's host search, the Scheduler Plugin Framework shall set the attempt's slot count to that bucket's task count and its host count to 1, as parjob does today. For seqjob, whose only plan is one bucket of one task, the slot count and host count shall always be 1, in the job-level preparation pass and in every per-host check.
6. The Scheduler Plugin Framework shall start every bucket's host search from an empty per-host request, for both seqjob and parjob, so that no earlier bucket, earlier candidate host, or earlier primary of the same COMPOUND or ALTER job (bug-1883) affects which hosts the host search skips. This is today's parjob behaviour and an intentional change for seqjob, which today pre-filters candidates against whatever per-host request was left from the job-level preparation pass or the last rejected candidate.
7. When seqjob's host search finds no host for a pending job, the Scheduler Plugin Framework shall record a "no resource" pend reason, naming the host and the resource, for every candidate host that matches the job's host selection and that the existing per-host check finds short of a requested resource, and shall leave the job's main pend reason as it is today. (Intentional change: today such reasons appear only for hosts that also pass the stale pre-filter.)
6. The Scheduler Plugin Framework shall clear the per-host request before each candidate host is considered, for both seqjob and parjob, so that no earlier bucket, earlier candidate host, or earlier primary of the same COMPOUND or ALTER job (bug-1883) affects which hosts the host search skips. (Intentional change, decided 2026-09-30: today parjob clears it once per bucket and seqjob never does, so after the first rejected candidate both pre-filter later candidates against the request that candidate left behind and skip short hosts without a reason.)
7. For every bucket search, in both allocators, the Scheduler Plugin Framework shall record a "no resource" pend reason, naming the host and the resource, for every candidate host that matches the job's host selection, is not already claimed in the plan, and that the existing per-host check finds short of a requested resource, and shall leave the job's main pend reason as it is today. (Intentional change: today such reasons appear only for the first rejected candidate of a search and for later candidates that pass the stale pre-filter.)
8. The Scheduler Plugin Framework shall support plans with any number of buckets up to the job's task count, with no fixed upper limit on the number of buckets that a large task count could exceed.
### Requirement 4: Result assembly
@ -94,7 +94,7 @@ Terms:
#### Acceptance Criteria
1. The Scheduler Plugin Framework shall pass, unchanged in their expected outcomes, the existing seqjob allocator, seqjob rewrap-equivalence, parjob combinatorial, parjob end-to-end multi-slot, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, simulation limit fan-out, election simulation fixture, and snapshot-load allocator regression tests. Test changes are limited to how the allocation routine is reached and the parameter type of plan-generator initialisation.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts (for seqjob, the per-host reasons given by Requirement 3 AC 7 rather than today's; for shared resources, the reasons given by Requirement 2 AC 2), mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts (for both allocators, the per-host reasons given by Requirement 3 AC 7 rather than today's; for shared resources, the reasons given by Requirement 2 AC 2), mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
3. The Scheduler Plugin Framework shall keep the existing check that pend-reason records are created only through the shared pend-reason factory, and that check shall cover the new framework allocation code.
4. When this feature is complete, `issues/new_requirements.md` item 4 shall be marked resolved, with a reference to this spec.
5. When this feature is complete, the scheduler state-equivalence invariant audit shall report no new violations in the allocation path.

15
.kiro/specs/scheduler-allocator-runner/tasks.md
View file

@ -22,7 +22,7 @@
- Done when the new target passes against the unchanged seqjob allocator
- _Requirements: 4.5, 5.4, 5.8, 7.2_
- [ ] 2.2 Characterize the parallel-job allocator through its registered entry point
- [x] 2.2 Characterize the parallel-job allocator through its registered entry point
- New test target for the real parjob plugin with the same stubs, injection mechanism and assertions as 2.1
- Fixtures: multi-bucket success; pending on a shared resource requesting exactly the short resources; pending on hosts; mid-plan rollback where a later bucket fails and a later plan succeeds; task counts 0 and 1 (pending, no pend-reason record); missing core, scheduler back-reference or primary request (invalid-argument, no pend-reason record); out of memory at the allocation records of host 0 and of a later host, at host allocation 0 and at a later host allocation, at the job allocation and at the shared records; the second primary of a COMPOUND job with non-overlapping resources
- Done when the new target passes against the unchanged parjob allocator
@ -41,12 +41,12 @@
- [ ] 3.2 Build the plan and claim search with rollback, test first
- Plans are tried in generator order and the first fully satisfied plan wins; each bucket claims the first unclaimed satisfying host in host-iteration order
- Shared amounts are counted as uncommitted exactly once across an attempt that tries several plans
- Before each bucket's host scan the per-host request is empty; before each candidate the slot count is the bucket's task count and the host count is 1
- The per-host request is empty before every candidate host is pre-filtered (cleared at the start of each bucket and after every rejected candidate), so every selection-matching unclaimed short host gets its host/resource reasons; before each candidate the slot count is the bucket's task count and the host count is 1
- A failing bucket releases every claim of the plan; a claim-record (snapshot) failure releases the candidate and earlier claims and returns the error; a plan with no buckets or a zero-slot bucket is skipped
- Claims-list growth failure is injected with a call-counting replacement of the heap allocator applied only to the white-box-included runner source (reused by 3.3); snapshot failure through a new link-time wrap of the libxce vector-copy function (same precedent) that is armed to fail only its next call, because other libxce code also calls it
- The claims list grows to the largest plan with no fixed limit; plan-generator initialisation failure returns pending; every return path after initialisation finishes the generator and frees the claims list
- Done when runner tests covering order, single shared count, distinct hosts, rollback debits, empty-request-per-bucket, slot/host counts, growth, malformed plans and snapshot failure pass under ASAN
- _Requirements: 2.3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.8, 5.5, 5.7, 7.2_
- Done when runner tests covering order, single shared count, distinct hosts, rollback debits, empty-request-before-every-candidate with complete reasons, slot/host counts, growth, malformed plans and snapshot failure pass under ASAN
- _Requirements: 2.3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 5.5, 5.7, 7.2_
- [ ] 3.3 Build result assembly and its out-of-memory paths, test first
- One host allocation per claim in claim order from the claimed host's name and its request snapshot; one shared record per shared-request entry, none when empty; job identifier 0, empty reservation name, no duration
@ -58,13 +58,14 @@
- [ ] 4. Move both allocators onto the framework routine
- [ ] 4.1 Switch the parallel-job allocator
- First add the parjob bug-1893 fixture (two shared resources, one short: exactly one host-less reason, naming the short one; today none is recorded) to the parjob characterization tests and see it fail against the old code
- First rewrite the parjob characterization tests' `_today` host-reason assertions to the complete per-host reasons of 3.7 (every selection-matching unclaimed short host of each bucket search) and see them fail against the old code
- Also add the parjob bug-1893 fixture (two shared resources, one short: exactly one host-less reason, naming the short one; today none is recorded) to the parjob characterization tests and see it fail against the old code
- Give parjob its allocator contract with a task-count acceptance guard (more than one task), register the framework routine as its allocation entry point, and remove its private allocation routine and claim helpers
- Build the runner into the parjob characterization target (2.2) so it keeps reaching the plugin through its entry point
- Update the parjob combinatorial tests, the only permitted kind of change: build the runner into the white-box test, give their hand-built plugin structs the parjob allocator contract, and call the framework routine and the framework's claim search and rollback instead of the removed helpers
- Done when every parjob characterization fixture (including the new one), the parjob combinatorial and end-to-end multi-slot tests, and the full scheduler suite pass under ASAN
- _Depends: 2.2, 3.3_
- _Requirements: 1.1, 1.2, 1.5, 1.6, 2.2, 5.2, 7.1_
- _Requirements: 1.1, 1.2, 1.5, 1.6, 2.2, 3.6, 3.7, 5.2, 7.1_
- [ ] 4.2 Switch the sequential-job allocator
- First add the seqjob intentional-change fixtures to the seqjob characterization tests and see them fail against the old code: a candidate host that the stale per-host request left by the job-level pass used to skip is now checked; complete per-host "no resource" reasons for every selection-matching short host with the main reason unchanged; bug-1893 (the 2.1 fixture that records no shared reason today now expects one host-less reason naming the short resource); release of the claimed host's debit when result building runs out of memory (this rewrites the 2.1 host-allocation out-of-memory fixture's per-host debit assertion to the new expected value)
@ -96,4 +97,6 @@
- Several white-box scheduler test targets run with `detect_leaks=0` because their fixtures are not torn down (`testsuite/meson.build:1756`); a leak summary from running such a binary directly is pre-existing, not a regression. New tests that assert "no leak" (3.3) need leak detection on for their own target.
- bug-1893 (found in 2.1): `vget()` returns 0 on success, so today's shared-reason loop (`rc <= 0` skip) records NO shared reason at all. The runner must skip only on `rc < 0`. Characterization fixtures record "no shared reason today" in a separate `_today` test that 4.1/4.2 rewrite.
- `xce_sched_core_fini()` does not free the string table `xce_sched_core_init()` creates; leak-checked test targets call `xstring_table_fini()` in teardown (see `test_alloc_equivalence_seqjob.c`).
- Per-host reasons (decided 2026-09-30 after 2.2): the runner clears `alloc_req.host` before every candidate (start of bucket + after each rejected candidate), both allocators, so every selection-matching unclaimed short host gets reasons. Today parjob gives reasons only for a bucket search's first rejected candidate; the 2.2 file keeps those counts in `_today` tests that 4.1 rewrites.
- `test_alloc_equivalence_seqjob.c` is at ~794/800 lines: 4.2's new fixtures go in a second file or after trimming.
- 4.1 expectations for the parjob `_today` host-reason test under the per-candidate clear: two-bucket success 2 reasons (A slots, B slots); pending on hosts 6 (A mem, A slots, B mem, B slots, A mem, B mem); mid-plan rollback 5 slots reasons (A, B, C from plan [4]; B, C from plan [2,2] bucket 2). `eq_assert_reason_prefix` checks `>=`, so 4.1 must pin exact counts. The parjob file is at 799/800 lines.