lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(scheduler-allocator-runner): characterize the parjob allocator (task 2.2)

Browse source 
15 tests drive the real parjob plugin through job_alloc_fn: success,
short shared, pending on hosts, mid-plan rollback, task counts 0/1,
missing inputs, six OOM steps with their debits, COMPOUND. Values the
runner changes on purpose sit in *_today tests.

Spec: the task showed that parjob records host reasons only for a bucket
search's first rejected candidate (the rejected host's request stays in
alloc_req.host and the host iterator pre-filters later hosts against
it). Decision: the runner clears alloc_req.host before every candidate,
for both allocators, so every short host gets its reasons (Req 3.6/3.7);
publishing reasons only when they change is a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-30 08:40:35 +00:00
parent 91a8580948
commit b4470b2b94
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 874 additions and 56 deletions
Download patch file Download diff file Expand all files Collapse all files

65
.kiro/specs/scheduler-allocator-runner/brief.md
View file

@ -117,40 +117,37 @@ goal that plugins no longer export `my_alloc()`).
the seqjob OOM rollback of item 2 (Req 5 AC 5).
- Divergences found in discovery that the design MUST resolve explicitly
(they are not in the 2026-09-23 proposal):
1. **Decided 2026-09-29: the shared loop resets `ctx->alloc_req.host` to
empty before every bucket's host scan, for both allocators.** Background:
the per-host request used by the authoritative check is already rebuilt
for every candidate (`sched_try_alloc_on_host_()` -> `plugin_prepare_
allocation()` with `ctx->worker` set resets and refills it,
`plugin.c:639`). The per-bucket reset only affects the host iterator's
pre-filter (`host_iter.c:143`), which reads `alloc_req.host` before that
rebuild and silently skips hosts that fail. parjob already resets
(`parjob_allocator.c:479`), so its pre-filter does no resource filtering.
seqjob does not, so its pre-filter reads stale content: the job-level
`slots:1` for the first candidate, then the last rejected candidate's
full request. Resetting for seqjob too drops the stale-state dependency
and makes seqjob's pre-filter a resource no-op, like parjob's.
**Accepted consequence (intentional change):** host choice is unchanged
(a host the stale pre-filter rejects also fails the authoritative check
while the per-host request is host-independent; where a limit makes it
host-specific, the stale filter could wrongly skip a host, which the
reset fixes). But a pending seqjob job now gets a `PREASON_NO_RESOURCE`
host/resource reason from the authoritative check
(`resource.c:575`) for every host that passes the select filter and
lacks resources, not only for hosts the stale pre-filter let through.
`main_reason` is unaffected (host/resource reasons go to
`res_based_reasons`, `lib/preason.c:147`), so sim parity
(`sim_verify_pend_main_reason_eq`) is unaffected; no testsuite file
asserts `PREASON_*` host reasons (checked 2026-09-29).
Also accepted (found in design review): seqjob now runs the per-host
prepare fan-out on those hosts too, so the limit plugin may instantiate
a host limit's virtual resource and set its value on the worker earlier
than before (decision-neutral, lazy creation moves earlier), at the cost
of one prepare + check per short host per pending seqjob attempt.
Rejected: keeping the reset parjob-only (preserves the stale-state
quirk in shared code); staging each bucket's real request before the
scan so the pre-filter filters correctly (changes parjob's pend
reasons too; a separate follow-up if wanted).
1. **Decided 2026-09-29, revised 2026-09-30 (task 2.2 finding, user
decision): the runner clears `ctx->alloc_req.host` before every
candidate host (at the start of each bucket and after every rejected
candidate), for both allocators.** The per-host request used by the
authoritative check is rebuilt for every candidate anyway
(`plugin_prepare_allocation()` with `ctx->worker` set, `plugin.c:639`).
What the clear changes is the host iterator's pre-filter
(`host_iter.c:143`), which reads `alloc_req.host` before that rebuild and
silently skips hosts that fail. The 2026-09-29 plan (clear once per
bucket, as parjob does) turned out not to be enough: task 2.2's
characterization showed that a rejected candidate leaves its own request
behind, so parjob today records reasons only for the first rejected
candidate of a bucket search, and a once-per-bucket clear would give
seqjob the same partial reasons. With a clear before every candidate,
the pre-filter only applies the select expression, and every
selection-matching unclaimed short host gets its host/resource reasons,
in both allocators.
**Accepted consequences (intentional changes):** host choice, main
reason and decisions are unchanged (a host the stale pre-filter rejects
also fails the authoritative check while the per-host request is
host-independent; where a limit makes it host-specific, the stale filter
could wrongly skip a host, which the clear fixes). Pending jobs of both
allocators carry more host/resource reasons (`res_based_reasons`; not
`main_reason`, so the unsatisfiable cache and sim parity are unaffected).
The per-host prepare fan-out, including the limit plugin's lazy
host-limit instantiation, now runs on every short host (decision-neutral;
one prepare + check per short host per bucket search). Larger reason sets
are published and persisted; publishing reasons only when they change is
a follow-up (user, 2026-09-30), out of scope here.
Rejected: clearing once per bucket (partial reasons, still stale-state
dependent); keeping today's per-plugin behaviour.
2. **Decided (requirements Req 5 AC 5):** seqjob builds its halloc directly
from `ctx->alloc_req.host`; under the shared per-claim snapshot it gains
a new `-ENOMEM` path, which goes through the same rollback

22
.kiro/specs/scheduler-allocator-runner/design.md
View file

@ -5,7 +5,7 @@
**Users**: Scheduler plugin authors (a new allocator needs only a plan iterator) and scheduler maintainers (an allocation fix lands once).
**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the seqjob per-bucket reset are under System Flows → Consequences.
**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the per-candidate reset are under System Flows → Consequences.
### Goals
- One allocation algorithm, used by seqjob and parjob in normal and simulation mode (1.1, 6.3).
@ -18,12 +18,13 @@
- Merging `sched_make_seqjob_allocation()` / `sched_make_parjob_allocation()` dispatch.
- Adding an allocation duration (the `FIXME: no duration now` stays).
- Fixing bug-1892 (`unsatisfy_()` reads uncommitted amounts by position).
- Publishing pend reasons only when they change (follow-up to the larger per-host reason sets of 3.7).
## Boundary Commitments
### This Spec Owns
- `xce_sched_allocator_iface_t` (shape, ownership, lifetime) and the `allocator_iface` field of `xce_sched_plugin_t`.
- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-bucket `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-candidate `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
- The `xce_alloc_plan_iter_t.init` parameter type.
- The seqjob/parjob plugin wiring (templates, iface instances, `parjob_accepts`) and the removal of their private allocation code.
- Equivalence and runner tests; the pend-reason factory audit's file list; `issues/new_requirements.md` item 4.
@ -145,7 +146,7 @@ flowchart TD
Finalize -->|ok| Done[publish ctx alloc then fini and return 1]
```
**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan; set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.
**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan (and reset `alloc_req.host` again after every rejected candidate, before the next `next()`); set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.
**Key decisions**: the reset and the `nslots`/`nhosts` assignment are unconditional (brief items 1 and 3); every return after `plan_init` succeeded calls `plan_fini` and frees the claims buffer; claim snapshots are freed by whichever function owns them at that point (see AllocRunner → State Management).
@ -168,8 +169,8 @@ flowchart TD
| 3.3 | Roll back claims on unsatisfied bucket | ClaimSearch | `sched_ctx_uncommit_host` | TryPlan |
| 3.4 | Init failure / no plan = pending | AllocRunner | `plan_init` | Pend1, Fini0 |
| 3.5 | `nslots` = bucket slots, `nhosts` = 1 | ClaimSearch | — | TryPlan |
| 3.6 | Empty per-host request per bucket | ClaimSearch | — | TryPlan |
| 3.7 | seqjob per-host reasons complete | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
| 3.6 | Empty per-host request before each candidate | ClaimSearch | — | TryPlan |
| 3.7 | Per-host reasons complete, both allocators | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
| 3.8 | No fixed bucket limit | AllocRunner (claims buffer) | — | Grow |
| 4.1 | One host alloc per claim, claim order, snapshot request | ResultAssembly | `sched_make_allocs`, `make_host_alloc` | Finalize |
| 4.2 | Shared record per shared-request entry | ResultAssembly | `sched_make_allocs` | Finalize |
@ -318,7 +319,7 @@ static int sched_snapshot_host_req_(xvec_idval_t* dst, const xvec_idval_t* src)
```
- **Preconditions**: `claims` has capacity `>= plan->nbuckets`.
- **Postconditions**: on 0 or `< 0`, every per-host debit made in this call is released (3.3, 5.5) and no snapshot remains.
- Behaviour is parjob's current `parjob_try_plan_()` exactly, with `plan->ctx = ctx`, the per-bucket reset (3.6) and the per-candidate `nslots`/`nhosts` assignment (3.5) now applied to every plugin. seqjob's complete per-host reasons (3.7) follow from 3.6: the pre-filter no longer skips hosts on stale state, so `sched_try_alloc_on_host_()` records its reasons for every selection-matching host.
- Behaviour is parjob's current `parjob_try_plan_()` with `plan->ctx = ctx` and the per-candidate `nslots`/`nhosts` assignment (3.5), applied to every plugin, and one change: `alloc_req.host` is cleared at the start of each bucket AND after every rejected candidate, i.e. before every `host_iter->next()` call (3.6). Clearing only once per bucket (parjob today) is not enough: a rejected candidate leaves its own per-host request in `alloc_req.host`, and the host iterator's pre-filter (`host_iter.c`) would then skip later short hosts silently. With the request always empty when `next()` pre-filters, the pre-filter does select-expression filtering only, every selection-matching unclaimed host reaches `sched_try_alloc_on_host_()`, and each short one gets its reasons (3.7).
#### ResultAssembly
@ -367,10 +368,11 @@ static const xce_sched_allocator_iface_t seqjob_alloc_iface = {
```
- `prepare_allocation_hook`, `init_fn`, `fini_fn`, `*_new_plugin()`, `register_*()` unchanged; sim registration in `sim_alloc.c` therefore reaches the runner unchanged (6.3).
### Consequences of the per-bucket reset for seqjob (3.6)
Because seqjob's pre-filter no longer skips hosts on stale state, a pending seqjob attempt now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching host that is short of resources, as parjob already does. Besides the per-host pend reasons (3.7), this means:
### Consequences of the per-candidate reset (3.6, both allocators)
Because the host iterator's pre-filter no longer skips hosts on stale state, a pending attempt of either allocator now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching unclaimed host that is short of resources. Today that happens only for the first rejected candidate of each bucket search (parjob) or for candidates that pass the stale filter (seqjob). Besides the complete per-host pend reasons (3.7), this means:
- **Limit plugin side effects** (`limit.c` `match_job_host_limit_`): for a host with a matching host-based limit it allocates the job's `obj->hlimits[worker]` list, may instantiate the limit's virtual resource in `resmgr` (`instantiate_limit`) and writes the limit value into `worker->resvals`. This is decision-neutral: any later check of that host by any job runs the same lazy instantiation before reading the value, so it only moves when the instance is created. It happens identically in normal and simulation mode (driver state-equivalence holds).
- **Cost**: one prepare fan-out plus satisfy check per short host per pending seqjob attempt, where the stale pre-filter used to reject some hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
- **Cost**: one prepare fan-out plus satisfy check per short host per bucket search, where the stale pre-filter used to reject later hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
- **Pend-reason volume**: a pending job can now carry one reason per short host (per bucket search for parjob). These reasons are published to the controller and persisted with the job. Publishing reasons only when they change is a recorded follow-up (Non-Goals), not part of this spec.
## Data Models
No persisted or wire data changes. In-memory only: `sched_alloc_claim_t` (attempt-scoped) and the `allocator_iface` pointer (static). Snapshot/WAL/recording field-completeness is not affected (scheduler audit dimension 4: no change).
@ -406,7 +408,7 @@ TDD order follows the staged migration (research.md): characterization tests are
- `test_alloc_runner.c` (white-box `#include` of `core.c`, `resource.c`, `schedobj.c`, `host_iter.c`, `alloc_runner.c`; stubbed `sched_make_allocs` with OOM injection; stubbed `plugin_prepare_allocation()` that records `ctx->nslots`/`ctx->nhosts`/`ctx->alloc_req.host.count` on each call and stages a fixed per-host request, the same link-stub convention `test_resource_accounting.c` uses; an in-test fake iface whose plans are scripted):
- guards: each missing input → `-EINVAL`, `ctx->reason` still NULL (5.1); rejecting `accepts` → 0, no reason, `plan_init` not called (1.4); NULL `accepts` reaches `plan_init` (1.3).
- shared: two shared resources, one short → exactly one host-less `NO_RESOURCE` reason naming it (2.2, bug-1893); met → uncommitted added once across a 3-plan attempt (2.3).
- search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty when each bucket's host scan starts, observed through a candidate whose stale request would have been rejected by the pre-filter (3.6); a plan with more buckets than a first small plan triggers buffer growth (3.8).
- search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty before every candidate's pre-filter: with three short hosts in one bucket, all three get host/resource reasons (today only the first would) (3.6, 3.7); a plan with more buckets than a first small plan triggers buffer growth (3.8).
- errors: snapshot OOM, host-alloc OOM at claim 0 and claim k, job-alloc OOM, shared-records OOM → `-ENOMEM`, `ctx->alloc == NULL`, released debits as in the Error table, ASAN clean (5.4, 5.5, 5.6, 5.7).
- `test_parjob_combinatorial.c`: plan sequences identical for `NULL` and poisoned core (6.2).

12
.kiro/specs/scheduler-allocator-runner/requirements.md
View file

@ -5,12 +5,12 @@
**Current situation:** `seqjob_allocator.c` (`my_alloc`, ~215 lines) and `parjob_allocator.c` (`my_alloc`, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its `PREASON_NO_RESOURCE` fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (`make_job_alloc` + `sched_make_allocs` + the `-ENOMEM` cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole `my_alloc()`. Plan generation is already abstracted behind `xce_alloc_plan_iter_t`, whose `init` takes an unused `xce_scheduler_t*` even though sim mode has no real scheduler. Tracked as `issues/new_requirements.md` item 4.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons (except seqjob's per-host reasons, which become complete, Requirement 3 AC 6-7, and the shared-resource reasons bug-1893 never recorded, Requirement 2 AC 2), and error and `-ENOMEM` behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons (except the per-host reasons of both allocators, which become complete, Requirement 3 AC 6-7, and the shared-resource reasons bug-1893 never recorded, Requirement 2 AC 2), and error and `-ENOMEM` behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
## Introduction
The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the sequential-allocator host pend reasons of Requirement 3 AC 6 and AC 7, which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which today are never recorded (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the per-host pend reasons of both allocators (Requirement 3 AC 6 and AC 7), which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which today are never recorded (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.
Terms:
- **Allocation attempt**: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
@ -20,7 +20,7 @@ Terms:
## Boundary Context
- **In scope**: a framework-owned allocation routine used by both built-in allocators; the allocator plugin contract (plan generator, optional acceptance guard, and the existing preparation hook); removal of the per-plugin allocation routines; changing the plan-generator initialisation to take the scheduling core instead of the scheduler object; keeping every existing allocation, pend-reason, error and simulation-parity behaviour apart from the intentional changes listed in the Introduction; closing `issues/new_requirements.md` item 4.
- **Out of scope**: a per-plugin override hook for assembling results; retyping the allocation-record constructor (`sched_make_allocs`) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
- **Out of scope**: a per-plugin override hook for assembling results; retyping the allocation-record constructor (`sched_make_allocs`) or removing the plugin's scheduler back-pointer; merging the two dispatch paths (sequential vs parallel) in the scheduler; fixing the missing allocation duration (the existing "no duration now" FIXME is kept as is); publishing pend reasons only when they change (a follow-up: complete per-host reasons make each pending job's reason set larger); any change to plan order, host order, pend-reason vocabulary, the unsatisfiable-request cache, preemption, or the job picker.
- **Adjacent expectations**: the scheduler's allocation dispatch keeps sending `maxTasks == 1` primaries to seqjob and every other primary to parjob, and keeps treating result `> 0` as satisfied, `0` as pending and `< 0` as error, with an allocation published only on `> 0`. COMPOUND/ALTER handling (bug-1766: a merged allocation is published only if every primary succeeds) and snapshot-restored schedobj construction (bug-1767) stay owned by their existing code. The plan and host iterator contracts from `scheduler-allocation` keep their current ordering semantics.
## Requirements
@ -54,8 +54,8 @@ Terms:
3. If any bucket of a plan cannot be satisfied, the Scheduler Plugin Framework shall release the per-host resource debits of every claim already made for that plan before trying the next plan.
4. When the plan generator cannot be initialised or yields no plan whose buckets are all satisfied, the Scheduler Plugin Framework shall return the pending signal with no allocation on the attempt.
5. Before each bucket's host search, the Scheduler Plugin Framework shall set the attempt's slot count to that bucket's task count and its host count to 1, as parjob does today. For seqjob, whose only plan is one bucket of one task, the slot count and host count shall always be 1, in the job-level preparation pass and in every per-host check.
6. The Scheduler Plugin Framework shall start every bucket's host search from an empty per-host request, for both seqjob and parjob, so that no earlier bucket, earlier candidate host, or earlier primary of the same COMPOUND or ALTER job (bug-1883) affects which hosts the host search skips. This is today's parjob behaviour and an intentional change for seqjob, which today pre-filters candidates against whatever per-host request was left from the job-level preparation pass or the last rejected candidate.
7. When seqjob's host search finds no host for a pending job, the Scheduler Plugin Framework shall record a "no resource" pend reason, naming the host and the resource, for every candidate host that matches the job's host selection and that the existing per-host check finds short of a requested resource, and shall leave the job's main pend reason as it is today. (Intentional change: today such reasons appear only for hosts that also pass the stale pre-filter.)
6. The Scheduler Plugin Framework shall clear the per-host request before each candidate host is considered, for both seqjob and parjob, so that no earlier bucket, earlier candidate host, or earlier primary of the same COMPOUND or ALTER job (bug-1883) affects which hosts the host search skips. (Intentional change, decided 2026-09-30: today parjob clears it once per bucket and seqjob never does, so after the first rejected candidate both pre-filter later candidates against the request that candidate left behind and skip short hosts without a reason.)
7. For every bucket search, in both allocators, the Scheduler Plugin Framework shall record a "no resource" pend reason, naming the host and the resource, for every candidate host that matches the job's host selection, is not already claimed in the plan, and that the existing per-host check finds short of a requested resource, and shall leave the job's main pend reason as it is today. (Intentional change: today such reasons appear only for the first rejected candidate of a search and for later candidates that pass the stale pre-filter.)
8. The Scheduler Plugin Framework shall support plans with any number of buckets up to the job's task count, with no fixed upper limit on the number of buckets that a large task count could exceed.
### Requirement 4: Result assembly
@ -94,7 +94,7 @@ Terms:
#### Acceptance Criteria
1. The Scheduler Plugin Framework shall pass, unchanged in their expected outcomes, the existing seqjob allocator, seqjob rewrap-equivalence, parjob combinatorial, parjob end-to-end multi-slot, task-count wiring, unsatisfiable-cache dispatch, simulation parity, simulation live-allocation, simulation limit fan-out, election simulation fixture, and snapshot-load allocator regression tests. Test changes are limited to how the allocation routine is reached and the parameter type of plan-generator initialisation.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts (for seqjob, the per-host reasons given by Requirement 3 AC 7 rather than today's; for shared resources, the reasons given by Requirement 2 AC 2), mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
2. The Scheduler Plugin Framework shall have tests showing that, for the same inputs, the framework routine gives the same result code, allocation result, pend reasons, and per-host debits as the removed seqjob and parjob routines. These tests cover success, pending on shared resources, pending on hosts (for both allocators, the per-host reasons given by Requirement 3 AC 7 rather than today's; for shared resources, the reasons given by Requirement 2 AC 2), mid-plan rollback, out-of-memory at each result-building step, the parjob out-of-contract task count, and missing inputs.
3. The Scheduler Plugin Framework shall keep the existing check that pend-reason records are created only through the shared pend-reason factory, and that check shall cover the new framework allocation code.
4. When this feature is complete, `issues/new_requirements.md` item 4 shall be marked resolved, with a reference to this spec.
5. When this feature is complete, the scheduler state-equivalence invariant audit shall report no new violations in the allocation path.

15
.kiro/specs/scheduler-allocator-runner/tasks.md
View file

@ -22,7 +22,7 @@
- Done when the new target passes against the unchanged seqjob allocator
- _Requirements: 4.5, 5.4, 5.8, 7.2_
- [ ] 2.2 Characterize the parallel-job allocator through its registered entry point
- [x] 2.2 Characterize the parallel-job allocator through its registered entry point
- New test target for the real parjob plugin with the same stubs, injection mechanism and assertions as 2.1
- Fixtures: multi-bucket success; pending on a shared resource requesting exactly the short resources; pending on hosts; mid-plan rollback where a later bucket fails and a later plan succeeds; task counts 0 and 1 (pending, no pend-reason record); missing core, scheduler back-reference or primary request (invalid-argument, no pend-reason record); out of memory at the allocation records of host 0 and of a later host, at host allocation 0 and at a later host allocation, at the job allocation and at the shared records; the second primary of a COMPOUND job with non-overlapping resources
- Done when the new target passes against the unchanged parjob allocator
@ -41,12 +41,12 @@
- [ ] 3.2 Build the plan and claim search with rollback, test first
- Plans are tried in generator order and the first fully satisfied plan wins; each bucket claims the first unclaimed satisfying host in host-iteration order
- Shared amounts are counted as uncommitted exactly once across an attempt that tries several plans
- Before each bucket's host scan the per-host request is empty; before each candidate the slot count is the bucket's task count and the host count is 1
- The per-host request is empty before every candidate host is pre-filtered (cleared at the start of each bucket and after every rejected candidate), so every selection-matching unclaimed short host gets its host/resource reasons; before each candidate the slot count is the bucket's task count and the host count is 1
- A failing bucket releases every claim of the plan; a claim-record (snapshot) failure releases the candidate and earlier claims and returns the error; a plan with no buckets or a zero-slot bucket is skipped
- Claims-list growth failure is injected with a call-counting replacement of the heap allocator applied only to the white-box-included runner source (reused by 3.3); snapshot failure through a new link-time wrap of the libxce vector-copy function (same precedent) that is armed to fail only its next call, because other libxce code also calls it
- The claims list grows to the largest plan with no fixed limit; plan-generator initialisation failure returns pending; every return path after initialisation finishes the generator and frees the claims list
- Done when runner tests covering order, single shared count, distinct hosts, rollback debits, empty-request-per-bucket, slot/host counts, growth, malformed plans and snapshot failure pass under ASAN
- _Requirements: 2.3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.8, 5.5, 5.7, 7.2_
- Done when runner tests covering order, single shared count, distinct hosts, rollback debits, empty-request-before-every-candidate with complete reasons, slot/host counts, growth, malformed plans and snapshot failure pass under ASAN
- _Requirements: 2.3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 5.5, 5.7, 7.2_
- [ ] 3.3 Build result assembly and its out-of-memory paths, test first
- One host allocation per claim in claim order from the claimed host's name and its request snapshot; one shared record per shared-request entry, none when empty; job identifier 0, empty reservation name, no duration
@ -58,13 +58,14 @@
- [ ] 4. Move both allocators onto the framework routine
- [ ] 4.1 Switch the parallel-job allocator
- First add the parjob bug-1893 fixture (two shared resources, one short: exactly one host-less reason, naming the short one; today none is recorded) to the parjob characterization tests and see it fail against the old code
- First rewrite the parjob characterization tests' `_today` host-reason assertions to the complete per-host reasons of 3.7 (every selection-matching unclaimed short host of each bucket search) and see them fail against the old code
- Also add the parjob bug-1893 fixture (two shared resources, one short: exactly one host-less reason, naming the short one; today none is recorded) to the parjob characterization tests and see it fail against the old code
- Give parjob its allocator contract with a task-count acceptance guard (more than one task), register the framework routine as its allocation entry point, and remove its private allocation routine and claim helpers
- Build the runner into the parjob characterization target (2.2) so it keeps reaching the plugin through its entry point
- Update the parjob combinatorial tests, the only permitted kind of change: build the runner into the white-box test, give their hand-built plugin structs the parjob allocator contract, and call the framework routine and the framework's claim search and rollback instead of the removed helpers
- Done when every parjob characterization fixture (including the new one), the parjob combinatorial and end-to-end multi-slot tests, and the full scheduler suite pass under ASAN
- _Depends: 2.2, 3.3_
- _Requirements: 1.1, 1.2, 1.5, 1.6, 2.2, 5.2, 7.1_
- _Requirements: 1.1, 1.2, 1.5, 1.6, 2.2, 3.6, 3.7, 5.2, 7.1_
- [ ] 4.2 Switch the sequential-job allocator
- First add the seqjob intentional-change fixtures to the seqjob characterization tests and see them fail against the old code: a candidate host that the stale per-host request left by the job-level pass used to skip is now checked; complete per-host "no resource" reasons for every selection-matching short host with the main reason unchanged; bug-1893 (the 2.1 fixture that records no shared reason today now expects one host-less reason naming the short resource); release of the claimed host's debit when result building runs out of memory (this rewrites the 2.1 host-allocation out-of-memory fixture's per-host debit assertion to the new expected value)
@ -96,4 +97,6 @@
- Several white-box scheduler test targets run with `detect_leaks=0` because their fixtures are not torn down (`testsuite/meson.build:1756`); a leak summary from running such a binary directly is pre-existing, not a regression. New tests that assert "no leak" (3.3) need leak detection on for their own target.
- bug-1893 (found in 2.1): `vget()` returns 0 on success, so today's shared-reason loop (`rc <= 0` skip) records NO shared reason at all. The runner must skip only on `rc < 0`. Characterization fixtures record "no shared reason today" in a separate `_today` test that 4.1/4.2 rewrite.
- `xce_sched_core_fini()` does not free the string table `xce_sched_core_init()` creates; leak-checked test targets call `xstring_table_fini()` in teardown (see `test_alloc_equivalence_seqjob.c`).
- Per-host reasons (decided 2026-09-30 after 2.2): the runner clears `alloc_req.host` before every candidate (start of bucket + after each rejected candidate), both allocators, so every selection-matching unclaimed short host gets reasons. Today parjob gives reasons only for a bucket search's first rejected candidate; the 2.2 file keeps those counts in `_today` tests that 4.1 rewrites.
- `test_alloc_equivalence_seqjob.c` is at ~794/800 lines: 4.2's new fixtures go in a second file or after trimming.
- 4.1 expectations for the parjob `_today` host-reason test under the per-candidate clear: two-bucket success 2 reasons (A slots, B slots); pending on hosts 6 (A mem, A slots, B mem, B slots, A mem, B mem); mid-plan rollback 5 slots reasons (A, B, C from plan [4]; B, C from plan [2,2] bucket 2). `eq_assert_reason_prefix` checks `>=`, so 4.1 must pin exact counts. The parjob file is at 799/800 lines.

17
src/scheduler/testsuite/meson.build
View file

@ -1729,6 +1729,20 @@ sched_alloc_equivalence_seqjob_test = executable(
'-Wl,--wrap=make_job_alloc'],
install: false)
# scheduler-allocator-runner task 2.2 (Req 4.5, 5.1, 5.2, 5.4, 5.8, 7.2):
# parjob characterization through the plugin's registered job_alloc_fn,
# against today's allocator and (from task 4.1) the framework runner. Same
# white-box stack and make_host_alloc/make_job_alloc wraps as
# sched_alloc_equivalence_seqjob_test.
sched_alloc_equivalence_parjob_test = executable(
'sched_alloc_equivalence_parjob_test',
['test_alloc_equivalence_parjob.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
link_args: ['-Wl,--wrap=make_host_alloc',
'-Wl,--wrap=make_job_alloc'],
install: false)
# Task 3.2 (scheduler-allocation spec, Req 9.1): the dedicated pre/post-
# rewrap host-selection EQUIVALENCE proof for seqjob_allocator.c's
# my_alloc() -- fixture states task 3.1's own smoke test does not cover
@ -1998,6 +2012,9 @@ test('scheduler seqjob allocator test',
# Leak detection stays on: every fixture tears down its ctx, core and plugin.
test('scheduler alloc equivalence seqjob test',
sched_alloc_equivalence_seqjob_test, suite: 'scheduler')
# Leak detection stays on (as for the seqjob characterization above).
test('scheduler alloc equivalence parjob test',
sched_alloc_equivalence_parjob_test, suite: 'scheduler')
test('scheduler seqjob rewrap equivalence test',
sched_seqjob_rewrap_equivalence_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])

799
src/scheduler/testsuite/test_alloc_equivalence_parjob.c Normal file
View file

@ -0,0 +1,799 @@
/*
* test_alloc_equivalence_parjob.c - scheduler-allocator-runner task 2.2:
* characterization of the parallel-job allocator (Req 4.5, 5.1, 5.2, 5.4,
* 5.8, 7.2) through the registered plugin->job_alloc_fn, never the static
* routine by name; must still hold once task 4.1 switches parjob to the
* framework runner. Stubs and OOM injection are those of
* test_alloc_equivalence_seqjob.c: sched_make_allocs() failing on call N
* (claimed hosts' records in claim order, then shared records) and
* --wrap'ed make_host_alloc/make_job_alloc failing on call N; the
* host-array allocation is not injected (task 3.3).
*
* Values the switch changes on purpose sit in `_today` tests: bug-1893
* (Req 2.2) and host reasons (Req 3.6/3.7). Today the host request is
* reset once per bucket, so later short candidates are pre-filtered
* (host_iter.c) against the request the first one left and get no reason;
* the runner clears it before every candidate. Stable tests assert only
* reason entries both rules give. bug-1892 layouts are kept out.
*/
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "preason.h"
#include "resobj.h"
#include "resreq.h"
#include "sched/alloc_iter.h"
#include "scheduler.h"
#include "xcealloc.h"
#include "xstring.h"
/* White-box sources under test. Task 4.1 adds the framework runner
* ("../src/alloc_runner.c") as one more line of this list. */
#include "../src/core.c"
#include "../src/resource.c"
#include "../src/schedobj.c"
#include "../src/host_iter.c"
#include "../src/plugins/parjob_allocator.c"
/* Link-satisfying stubs (same convention as test_seqjob_allocator.c). */
int plugin_register(void* s, void* p) { return (void)s, (void)p, 0; }
void plugin_unregister(void* arg) { (void)arg; }
/* plugin.c's fan-out with parjob's hook only: reset the request vectors as
* plugin.c does (job-level pass: .shared and .host, bug-1883; per-host
* pass: .host), then call the hook. */
int plugin_prepare_allocation(xce_sched_core_t* sched, xce_sched_ctx_t* ctx)
{
xce_sched_plugin_t fake_plugin;
if (ctx->worker == NULL) {
xvec_idval_fini(&(ctx->alloc_req.shared));
xvec_idval_init(&(ctx->alloc_req.shared), 8);
}
xvec_idval_fini(&(ctx->alloc_req.host));
xvec_idval_init(&(ctx->alloc_req.host), 8);
memset(&fake_plugin, 0, sizeof(fake_plugin));
fake_plugin.sched = sched;
fake_plugin.core = sched;
return parjob_prepare_allocation(&fake_plugin, ctx);
}
int plugin_job_update(void* s, xce_sched_job_t* j)
{
return (void)s, (void)j, 0;
}
int plugin_queue_update(void* s, xce_sched_queue_t* q)
{
return (void)s, (void)q, 0;
}
int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r)
{
return (void)s, (void)j, (void)r, 0;
}
/* Counters count every call; fail_on N > 0 fails call N (NULL, ENOMEM). */
static struct {
int make_allocs_calls, make_allocs_fail_on;
int host_alloc_calls, host_alloc_fail_on;
int job_alloc_calls, job_alloc_fail_on;
} g;
static void oom_injection_reset(void) { memset(&g, 0, sizeof(g)); }
/* Stub for scheduler.c's sched_make_allocs(): real make_alloc() records
* (resmgr name, requested value, empty reservation, duration -1 for a NULL
* duration array, instid 0); NULL for nres <= 0 as the real one. */
xce_alloc_t** sched_make_allocs(xce_scheduler_t* sched, char* reserve,
int nres, int* idxs, float* vals,
int* duration)
{
xce_alloc_t** allocs;
int i;
g.make_allocs_calls++;
if ((g.make_allocs_fail_on > 0)
&& (g.make_allocs_calls == g.make_allocs_fail_on)) {
errno = ENOMEM;
return NULL;
}
if ((sched == NULL) || (nres <= 0) || (idxs == NULL) || (vals == NULL)) {
errno = EINVAL;
return NULL;
}
allocs = (xce_alloc_t**)calloc(nres, sizeof(*allocs));
if (allocs == NULL) {
return NULL;
}
for (i = 0; i < nres; i++) {
xce_resource_t* r = NULL;
vget(sched->core.resmgr, idxs[i], (void**)&r);
if (r == NULL) {
continue;
}
allocs[i] = make_alloc(r->name, reserve, 0, vals[i],
(duration != NULL) ? duration[i] : -1);
if (allocs[i] == NULL) {
free_ResAlloc_list(nres, allocs);
errno = ENOMEM;
return NULL;
}
}
return allocs;
}
xce_host_alloc_t* __real_make_host_alloc(char*, int, xce_alloc_t**);
xce_job_alloc_t* __real_make_job_alloc(uint64_t, int, xce_host_alloc_t**);
xce_host_alloc_t* __wrap_make_host_alloc(char* host, int nalloc,
xce_alloc_t** allocs)
{
g.host_alloc_calls++;
if ((g.host_alloc_fail_on > 0)
&& (g.host_alloc_calls == g.host_alloc_fail_on)) {
errno = ENOMEM;
return NULL;
}
return __real_make_host_alloc(host, nalloc, allocs);
}
xce_job_alloc_t* __wrap_make_job_alloc(uint64_t jobid, int nhost,
xce_host_alloc_t** hallocs)
{
g.job_alloc_calls++;
if ((g.job_alloc_fail_on > 0)
&& (g.job_alloc_calls == g.job_alloc_fail_on)) {
errno = ENOMEM;
return NULL;
}
return __real_make_job_alloc(jobid, nhost, hallocs);
}
void setUp(void) { oom_injection_reset(); }
void tearDown(void) { oom_injection_reset(); }
#define HOST_A "pq-host-a"
#define HOST_B "pq-host-b"
#define HOST_C "pq-host-c"
typedef struct {
xce_scheduler_t sched;
xce_resource_t* slots;
xce_resource_t* mem;
xce_resource_t* rack;
xce_resource_t* lic; /* shared (XCE_RESF_GLOBAL) */
xce_sched_plugin_t* plugin;
xce_sched_obj_t obj;
xce_primary_schedobj_t primary_obj;
xce_sched_ctx_t ctx;
int ctx_ready;
} eq_fixture_t;
/* One primary request: `mem` per task, `lic` per job (0 = not requested),
* optional `rack == N` select. */
typedef struct {
xce_primary_resreq_t req;
rusage_value_t mem_rusage;
rusage_value_t lic_rusage;
xce_select_t select;
xce_select_node_t sel_res, sel_num, sel_eq;
} eq_primary_t;
/* Shared pool "licenses" has `lic_total` capacity, none in use. */
static void eq_fixture_init(eq_fixture_t* f, float lic_total)
{
int idx[1];
float val[1];
memset(f, 0, sizeof(*f));
TEST_ASSERT_EQUAL_INT(0, xce_sched_core_init(&(f->sched.core)));
/* xce_sched_core_fini() finalizes these; both drivers calloc them */
for (idx[0] = 0; idx[0] < 2; idx[0]++) {
f->sched.core.pend_updates[idx[0]]
= (dataq_t*)calloc(1, sizeof(dataq_t));
TEST_ASSERT_NOT_NULL(f->sched.core.pend_updates[idx[0]]);
dataq_init(f->sched.core.pend_updates[idx[0]], NULL);
}
#define EQ_RES(n, fl) \
makeResource(f->sched.core.resmgr, &(xce_resource_t){ .name = (char*)n, \
.flags = fl })
f->slots = EQ_RES("slots", 0);
f->mem = EQ_RES("mem", 0);
f->rack = EQ_RES("rack", 0);
f->lic = EQ_RES("licenses", XCE_RESF_GLOBAL);
#undef EQ_RES
TEST_ASSERT_NOT_NULL(f->slots);
TEST_ASSERT_NOT_NULL(f->mem);
TEST_ASSERT_NOT_NULL(f->rack);
TEST_ASSERT_NOT_NULL(f->lic);
idx[0] = f->lic->idx;
val[0] = lic_total;
put_resvals(&(f->sched.core.sharedInst), 1, idx, val);
f->plugin = parjob_allocator_new_plugin();
TEST_ASSERT_NOT_NULL(f->plugin);
TEST_ASSERT_NOT_NULL(f->plugin->job_alloc_fn);
f->plugin->sched = &(f->sched);
f->plugin->core = &(f->sched.core);
}
static xce_sched_worker_t* eq_add_host(eq_fixture_t* f, const char* name,
float slots, float mem, float rack)
{
xce_sched_worker_t* w;
int idx[3] = { f->slots->idx, f->mem->idx, f->rack->idx };
float total[3] = { slots, mem, rack };
float inuse[2] = { 0.0f, 0.0f };
w = (xce_sched_worker_t*)calloc(1, sizeof(*w));
TEST_ASSERT_NOT_NULL(w);
w->name = makeString((char*)name);
TEST_ASSERT_NOT_NULL(w->name);
zarray_init(&(w->resvals), 8, (zarray_item_killer_t)kill_sched_resval);
zarray_init(&(w->resInUse), 8, (zarray_item_killer_t)kill_sched_resval);
put_resvals(&(w->resvals), 3, idx, total);
put_resvals(&(w->resInUse), 2, idx, inuse);
TEST_ASSERT_TRUE(sched_add_worker(&(f->sched.core), w) >= 0);
return w;
}
static void eq_primary_init(eq_primary_t* p, int max_tasks, float mem,
float lic, const eq_fixture_t* f, int rack)
{
memset(p, 0, sizeof(*p));
p->req.minTasks = (max_tasks > 0) ? 1 : 0;
p->req.maxTasks = max_tasks;
xlist_init(&(p->req.rusages), 4);
if (mem > 0.0f) {
p->mem_rusage.res = (char*)"mem";
p->mem_rusage.value = mem;
p->mem_rusage.per_flag = PER_TASK;
xlist_store(&(p->req.rusages), &(p->mem_rusage));
}
if (lic > 0.0f) {
p->lic_rusage.res = (char*)"licenses";
p->lic_rusage.value = lic;
p->lic_rusage.per_flag = PER_JOB;
xlist_store(&(p->req.rusages), &(p->lic_rusage));
}
if (rack > 0) {
p->sel_res.type = SEL_RESNAME;
p->sel_res.u.resname.res_idx = f->rack->idx;
p->sel_res.u.resname.res_type = RES_TYPE_NUMERIC;
p->sel_num.type = SEL_NUM;
p->sel_num.u.numval = (double)rack;
p->sel_eq.type = SEL_EQ;
p->sel_eq.u.binop.left = &(p->sel_res);
p->sel_eq.u.binop.right = &(p->sel_num);
p->select.root = &(p->sel_eq);
p->req.select = &(p->select);
}
}
/* ctx over a PRIMARY obj for `p`, job-level prepare pass included
* (sched_ctx_set_primary() -> plugin_prepare_allocation(), ctx->worker
* NULL), exactly as sched_make_primary_allocation() drives it. */
static void eq_ctx_begin_primary(eq_fixture_t* f, eq_primary_t* p)
{
f->primary_obj.resreq = &(p->req);
f->obj.tag = XCE_RESREQ_PRIMARY;
f->obj.u.primary = &(f->primary_obj);
TEST_ASSERT_EQUAL_INT(0, sched_ctx_init(&(f->ctx), &(f->sched.core),
&(f->obj)));
f->ctx_ready = 1;
sched_ctx_set_primary(&(f->ctx), &(p->req));
}
static int eq_alloc(eq_fixture_t* f)
{
return f->plugin->job_alloc_fn(f->plugin, &(f->ctx));
}
static void eq_fixture_fini(eq_fixture_t* f)
{
if (f->ctx_ready) {
free_JobAlloc_ptr(&(f->ctx.alloc));
free_PendReason_ptr(&(f->ctx.reason));
sched_ctx_fini(&(f->ctx));
}
f->plugin->fini_fn(f->plugin);
xce_sched_core_fini(&(f->sched.core));
/* xce_sched_core_init() creates the process-wide string table on every
* call and xce_sched_core_fini() leaves it; release it per fixture */
xstring_table_fini();
}
static void eq_primary_fini(eq_primary_t* p) { xlist_fini(&(p->req.rusages)); }
static void eq_finish(eq_fixture_t* f, eq_primary_t* p)
{
eq_fixture_fini(f);
eq_primary_fini(p);
}
static float eq_host_uncomit(eq_fixture_t* f, xce_sched_worker_t* w,
xce_resource_t* r)
{
xvec_t* v = (xvec_t*)zarray_get(&(f->ctx.uncomitted.hosts), w->idx);
return (v == NULL) ? 0.0f : ((float*)v->data)[r->idx];
}
/* Per-host debit left in the attempt context: mem then slots. */
static void eq_assert_debit(eq_fixture_t* f, xce_sched_worker_t* w, float mem,
float slots)
{
TEST_ASSERT_EQUAL_FLOAT(mem, eq_host_uncomit(f, w, f->mem));
TEST_ASSERT_EQUAL_FLOAT(slots, eq_host_uncomit(f, w, f->slots));
}
static float eq_shared_uncomit(eq_fixture_t* f)
{
return ((float*)f->ctx.uncomitted.shared->data)[f->lic->idx];
}
static void eq_assert_record(xce_alloc_t* a, const char* resname, float value)
{
TEST_ASSERT_NOT_NULL(a);
TEST_ASSERT_EQUAL_STRING(resname, a->resname);
TEST_ASSERT_EQUAL_STRING("", a->reserve);
TEST_ASSERT_EQUAL_FLOAT(value, a->value);
TEST_ASSERT_EQUAL_INT(-1, a->duration);
}
/* One host allocation carrying mem then slots (parjob's per-host prepare
* computes the rusage wants first, then stores the bucket's slot count). */
static void eq_assert_host(xce_host_alloc_t* ha, const char* host, float mem,
float slots)
{
TEST_ASSERT_NOT_NULL(ha);
TEST_ASSERT_EQUAL_STRING(host, ha->host);
TEST_ASSERT_EQUAL_INT(2, ha->nallocs);
TEST_ASSERT_NOT_NULL(ha->allocs);
eq_assert_record(ha->allocs[0], "mem", mem);
eq_assert_record(ha->allocs[1], "slots", slots);
}
static void eq_assert_reason_counts(eq_fixture_t* f, int main_reason,
int nhost, int nres)
{
TEST_ASSERT_NOT_NULL(f->ctx.reason);
TEST_ASSERT_NOT_NULL(f->ctx.reason->reasons);
TEST_ASSERT_EQUAL_INT(main_reason, f->ctx.reason->reasons->main_reason);
TEST_ASSERT_EQUAL_INT(nhost, f->ctx.reason->reasons->nhost_reasons);
TEST_ASSERT_EQUAL_INT(nres, f->ctx.reason->reasons->nres_reasons);
}
static void eq_assert_res_reason(eq_fixture_t* f, int i, const char* host,
const char* res)
{
xce_res_reason_t* rr = f->ctx.reason->reasons->res_based_reasons[i];
TEST_ASSERT_NOT_NULL(rr);
TEST_ASSERT_EQUAL_STRING(host, rr->host);
TEST_ASSERT_EQUAL_STRING(res, rr->resource);
TEST_ASSERT_EQUAL_INT(PREASON_NO_RESOURCE, rr->reason);
}
static void eq_assert_calls(int make_allocs, int host_alloc, int job_alloc)
{
TEST_ASSERT_EQUAL_INT(make_allocs, g.make_allocs_calls);
TEST_ASSERT_EQUAL_INT(host_alloc, g.host_alloc_calls);
TEST_ASSERT_EQUAL_INT(job_alloc, g.job_alloc_calls);
}
/* A fixture: `pool` shared licenses; hosts A, B, C (rack 1) with the given
* slots and mem; a job of `tasks` tasks, `task_mem` mem per task and `lic`
* licenses per job. */
typedef struct {
float pool;
int nhosts;
float slots[3], mem[3];
int tasks;
float task_mem, lic;
} eq_spec_t;
/* 1: plan [4] fails (slots), plan [2,2] claims A then B */
static const eq_spec_t EQ_TWO_BUCKETS = { 10, 2, { 2, 2 }, { 8, 8 }, 4, 1, 3 };
/* 2: 2 licenses, 3 wanted */
static const eq_spec_t EQ_SHORT_SHARED = { 2, 2, { 4, 4 }, { 8, 8 }, 2, 1, 3 };
/* 3: plans [2] and [1,1] both fail (mem, and slots for [2]) */
static const eq_spec_t EQ_SHORT_HOSTS = { 10, 2, { 1, 1 }, { 1, 1 }, 2, 4, 3 };
/* 4: [4] fails; [2,2] claims A, fails bucket 2, releases A; [2,1,1] wins */
static const eq_spec_t EQ_ROLLBACK
= { 10, 3, { 2, 1, 1 }, { 8, 8, 8 }, 4, 1, 3 };
static void eq_setup(eq_fixture_t* f, eq_primary_t* p, const eq_spec_t* s,
xce_sched_worker_t** w)
{
static const char* names[3] = { HOST_A, HOST_B, HOST_C };
int i;
eq_fixture_init(f, s->pool);
for (i = 0; i < s->nhosts; i++) {
w[i] = eq_add_host(f, names[i], s->slots[i], s->mem[i], 1.0f);
}
eq_primary_init(p, s->tasks, s->task_mem, s->lic, f, 0);
eq_ctx_begin_primary(f, p);
}
static int eq_run(eq_fixture_t* f, eq_primary_t* p, const eq_spec_t* s,
xce_sched_worker_t** w)
{
eq_setup(f, p, s, w);
return eq_alloc(f);
}
/* Main and host reasons none; at least `n` resource reasons, the first `n`
* of which both rules give (count asserted only in `_today` tests). */
static void eq_assert_reason_prefix(eq_fixture_t* f, int n)
{
TEST_ASSERT_NOT_NULL(f->ctx.reason);
TEST_ASSERT_EQUAL_INT(0, f->ctx.reason->reasons->main_reason);
TEST_ASSERT_EQUAL_INT(0, f->ctx.reason->reasons->nhost_reasons);
TEST_ASSERT_TRUE(f->ctx.reason->reasons->nres_reasons >= n);
}
/* EQ_TWO_BUCKETS: plan [4] checks A first (slots short) under both rules. */
static void eq_assert_plan1_reasons(eq_fixture_t* f)
{
eq_assert_reason_prefix(f, 1);
eq_assert_res_reason(f, 0, HOST_A, "slots");
}
/* 1. Multi-bucket success: two host allocations in claim order, each sized
* from its own bucket (2 tasks), one shared record. */
static void test_parjob_success_two_buckets_with_shared(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
TEST_ASSERT_EQUAL_INT(1, eq_run(&f, &p, &EQ_TWO_BUCKETS, w));
TEST_ASSERT_NOT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_UINT64(0, f.ctx.alloc->jobid);
TEST_ASSERT_EQUAL_INT(2, f.ctx.alloc->nhosts);
eq_assert_host(f.ctx.alloc->allocs[0], HOST_A, 2.0f, 2.0f);
eq_assert_host(f.ctx.alloc->allocs[1], HOST_B, 2.0f, 2.0f);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nres_shared);
eq_assert_record(f.ctx.alloc->allocs_shared[0], "licenses", 3.0f);
eq_assert_debit(&f, w[0], 2.0f, 2.0f);
eq_assert_debit(&f, w[1], 2.0f, 2.0f);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_plan1_reasons(&f);
eq_assert_calls(3, 2, 1);
eq_finish(&f, &p);
}
/* 2. Pending on the one shared resource the job requests, which is short:
* no host is tried and nothing is counted. */
static void test_parjob_pending_on_short_shared_resource(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
TEST_ASSERT_EQUAL_INT(0, eq_run(&f, &p, &EQ_SHORT_SHARED, w));
TEST_ASSERT_NULL(f.ctx.alloc);
TEST_ASSERT_NOT_NULL(f.ctx.reason);
TEST_ASSERT_EQUAL_INT(0, f.ctx.reason->reasons->main_reason);
TEST_ASSERT_EQUAL_INT(0, f.ctx.reason->reasons->nhost_reasons);
TEST_ASSERT_NULL(zarray_get(&(f.ctx.uncomitted.hosts), w[0]->idx));
TEST_ASSERT_NULL(zarray_get(&(f.ctx.uncomitted.hosts), w[1]->idx));
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_shared_uncomit(&f));
eq_assert_calls(0, 0, 0);
eq_finish(&f, &p);
}
/* 2, resource reasons, TODAY's behaviour: none. The reason loop skips
* every unmet entry because vget() returns 0 on success and the loop
* tests `rc <= 0` (bug-1893). Requirement 2.2 changes this in task 4.1
* (one host-less NO_RESOURCE reason naming "licenses"). */
static void test_parjob_pending_on_short_shared_no_reason_today(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
TEST_ASSERT_EQUAL_INT(0, eq_run(&f, &p, &EQ_SHORT_SHARED, w));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_finish(&f, &p);
}
/* 3. Pending on hosts: no debit remains; the met shared request stays
* counted as uncommitted. */
static void test_parjob_pending_on_hosts(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
TEST_ASSERT_EQUAL_INT(0, eq_run(&f, &p, &EQ_SHORT_HOSTS, w));
TEST_ASSERT_NULL(f.ctx.alloc);
/* plan [2]'s first candidate A, under both rules */
eq_assert_reason_prefix(&f, 2);
eq_assert_res_reason(&f, 0, HOST_A, "mem");
eq_assert_res_reason(&f, 1, HOST_A, "slots");
eq_assert_debit(&f, w[0], 0.0f, 0.0f);
eq_assert_debit(&f, w[1], 0.0f, 0.0f);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_calls(0, 0, 0);
eq_finish(&f, &p);
}
/* 4. Mid-plan rollback, then a later plan wins: only the winning plan's
* debits remain (A holds 2 slots, not 4). */
static void test_parjob_mid_plan_rollback_then_later_plan_wins(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
TEST_ASSERT_EQUAL_INT(1, eq_run(&f, &p, &EQ_ROLLBACK, w));
TEST_ASSERT_NOT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_INT(3, f.ctx.alloc->nhosts);
eq_assert_host(f.ctx.alloc->allocs[0], HOST_A, 2.0f, 2.0f);
eq_assert_host(f.ctx.alloc->allocs[1], HOST_B, 1.0f, 1.0f);
eq_assert_host(f.ctx.alloc->allocs[2], HOST_C, 1.0f, 1.0f);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nres_shared);
eq_assert_record(f.ctx.alloc->allocs_shared[0], "licenses", 3.0f);
eq_assert_debit(&f, w[0], 2.0f, 2.0f);
eq_assert_debit(&f, w[1], 1.0f, 1.0f);
eq_assert_debit(&f, w[2], 1.0f, 1.0f);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
/* A then B, slots: plan [4]'s first two checks (new rule), or plan
* [4]'s A and plan [2,2] bucket 2's B (today) */
eq_assert_reason_prefix(&f, 2);
eq_assert_res_reason(&f, 0, HOST_A, "slots");
eq_assert_res_reason(&f, 1, HOST_B, "slots");
eq_assert_calls(4, 3, 1);
eq_finish(&f, &p);
}
/* 1, 3, 4: host reasons, TODAY's behaviour (stale pre-filter): only each
* bucket search's first short candidate gets reasons. Req 3.6/3.7 change
* this in task 4.1 to every selection-matching unclaimed short host:
* 1 -> (A,slots) (B,slots); 3 -> A mem, A slots, B mem, B slots, A mem,
* B mem; 4 -> A, B, C (plan [4]) then B, C (plan [2,2] bucket 2), slots. */
static void test_parjob_host_reasons_first_candidate_only_today(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
TEST_ASSERT_EQUAL_INT(1, eq_run(&f, &p, &EQ_TWO_BUCKETS, w));
eq_assert_reason_counts(&f, 0, 0, 1);
eq_finish(&f, &p);
TEST_ASSERT_EQUAL_INT(0, eq_run(&f, &p, &EQ_SHORT_HOSTS, w));
eq_assert_reason_counts(&f, 0, 0, 3);
eq_assert_res_reason(&f, 2, HOST_A, "mem");
eq_finish(&f, &p);
TEST_ASSERT_EQUAL_INT(1, eq_run(&f, &p, &EQ_ROLLBACK, w));
eq_assert_reason_counts(&f, 0, 0, 2);
eq_finish(&f, &p);
}
/* One host (4 slots, 8 mem), 10 licenses; 2 tasks of 1 mem, 3 licenses. */
static const eq_spec_t EQ_ONE_HOST = { 10, 1, { 4 }, { 8 }, 2, 1, 3 };
/* Nothing checked, counted or built, and no pend-reason record. */
static void eq_assert_untouched(eq_fixture_t* f, xce_sched_worker_t* a)
{
TEST_ASSERT_NULL(f->ctx.alloc);
TEST_ASSERT_NULL(f->ctx.reason);
TEST_ASSERT_NULL(zarray_get(&(f->ctx.uncomitted.hosts), a->idx));
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_shared_uncomit(f));
eq_assert_calls(0, 0, 0);
}
/* 5. A task count of 0 or 1 reaching parjob directly (bug-1767, Req 5.2):
* pending, no pend-reason record, no resource check. */
static void test_parjob_task_count_zero_and_one_pend_without_reason(void)
{
int tasks;
for (tasks = 0; tasks <= 1; tasks++) {
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
eq_spec_t s = EQ_ONE_HOST;
s.tasks = tasks;
TEST_ASSERT_EQUAL_INT(0, eq_run(&f, &p, &s, w));
eq_assert_untouched(&f, w[0]);
eq_finish(&f, &p);
}
}
/* 6. Missing core, scheduler back-reference or primary (Req 5.1):
* -EINVAL, no pend-reason record, nothing checked or counted. */
static void test_parjob_missing_input_einval_no_reason(void)
{
int which;
for (which = 0; which < 3; which++) {
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
eq_setup(&f, &p, &EQ_ONE_HOST, w);
if (which == 0) {
f.plugin->core = NULL;
} else if (which == 1) {
f.plugin->sched = NULL;
} else {
f.ctx.primary = NULL;
}
TEST_ASSERT_EQUAL_INT(-EINVAL, eq_alloc(&f));
eq_assert_untouched(&f, w[0]);
eq_finish(&f, &p);
}
}
/* 7. Out of memory while building the result of EQ_TWO_BUCKETS (claims A
* then B). Host stage: the claims not yet turned into a host allocation
* are released, converted ones keep their debit; job allocation and shared
* records: every debit stays (design Error table; today's parjob too).
* `a_kept` / `b_kept`: whether A's / B's debit (2 mem, 2 slots) remains.
* Cases: records of host 0 / host 1 (sched_make_allocs() call 1 / 2),
* make_host_alloc() for host 0 / host 1, make_job_alloc(), shared records
* (sched_make_allocs() call 3, after both hosts). */
static void eq_check_oom(int make_allocs_fail_on, int host_alloc_fail_on,
int job_alloc_fail_on, int a_kept, int b_kept,
int make_allocs, int host_alloc, int job_alloc)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* w[3];
g.make_allocs_fail_on = make_allocs_fail_on;
g.host_alloc_fail_on = host_alloc_fail_on;
g.job_alloc_fail_on = job_alloc_fail_on;
TEST_ASSERT_EQUAL_INT(-ENOMEM, eq_run(&f, &p, &EQ_TWO_BUCKETS, w));
TEST_ASSERT_NULL(f.ctx.alloc);
eq_assert_debit(&f, w[0], a_kept ? 2.0f : 0.0f, a_kept ? 2.0f : 0.0f);
eq_assert_debit(&f, w[1], b_kept ? 2.0f : 0.0f, b_kept ? 2.0f : 0.0f);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_plan1_reasons(&f);
eq_assert_calls(make_allocs, host_alloc, job_alloc);
eq_finish(&f, &p);
}
/* One named test per injected step: fail-on counts, kept debits, calls. */
#define EQ_OOM_TEST(name, ...) \
static void name(void) { eq_check_oom(__VA_ARGS__); }
EQ_OOM_TEST(test_parjob_oom_records_host0_releases_all,
1, 0, 0, 0, 0, 1, 0, 0)
EQ_OOM_TEST(test_parjob_oom_records_later_host_releases_unconverted,
2, 0, 0, 1, 0, 2, 1, 0)
EQ_OOM_TEST(test_parjob_oom_host_alloc0_releases_all,
0, 1, 0, 0, 0, 1, 1, 0)
EQ_OOM_TEST(test_parjob_oom_host_alloc_later_releases_unconverted,
0, 2, 0, 1, 0, 2, 2, 0)
EQ_OOM_TEST(test_parjob_oom_job_alloc_debits_stay,
0, 0, 1, 1, 1, 2, 2, 1)
EQ_OOM_TEST(test_parjob_oom_shared_records_debits_stay,
3, 0, 0, 1, 1, 3, 2, 1)
/* 8. COMPOUND: two primaries allocated in turn on one ctx, as
* sched_make_compound_alloc_() does; rack 1 / rack 2 selects keep them on
* distinct hosts (bug-1892 not exercised). Each is sized from its own
* request (Req 5.8): 2 x 1 mem + 3 licenses on A, 3 x 2 mem on B. */
static void test_parjob_compound_second_primary_sized_from_own_request(void)
{
eq_fixture_t f;
eq_primary_t p1;
eq_primary_t p2;
xce_compound_resreq_t cres;
xce_compound_schedobj_t cobj;
xce_job_alloc_t* first = NULL;
xce_sched_worker_t * a, *b;
eq_fixture_init(&f, 10.0f);
a = eq_add_host(&f, HOST_A, 2.0f, 8.0f, 1.0f);
b = eq_add_host(&f, HOST_B, 4.0f, 8.0f, 2.0f);
eq_primary_init(&p1, 2, 1.0f, 3.0f, &f, 1);
eq_primary_init(&p2, 3, 2.0f, 0.0f, &f, 2);
xlist_init(&(cres.primaries), 2);
xlist_store(&(cres.primaries), &(p1.req));
xlist_store(&(cres.primaries), &(p2.req));
memset(&cobj, 0, sizeof(cobj));
cobj.owner = &(f.obj);
cobj.resreq = &cres;
f.obj.tag = XCE_RESREQ_COMPOUND;
f.obj.u.compound = &cobj;
TEST_ASSERT_EQUAL_INT(0, sched_ctx_init(&(f.ctx), &(f.sched.core),
&(f.obj)));
f.ctx_ready = 1;
sched_ctx_set_primary(&(f.ctx), &(p1.req));
TEST_ASSERT_EQUAL_INT(1, eq_alloc(&f));
MOVE_PTR(first, f.ctx.alloc);
sched_ctx_set_primary(&(f.ctx), &(p2.req));
TEST_ASSERT_EQUAL_INT(1, eq_alloc(&f));
TEST_ASSERT_NOT_NULL(first);
TEST_ASSERT_EQUAL_INT(1, first->nhosts);
eq_assert_host(first->allocs[0], HOST_A, 2.0f, 2.0f);
TEST_ASSERT_EQUAL_INT(1, first->nres_shared);
eq_assert_record(first->allocs_shared[0], "licenses", 3.0f);
TEST_ASSERT_NOT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nhosts);
eq_assert_host(f.ctx.alloc->allocs[0], HOST_B, 6.0f, 3.0f);
TEST_ASSERT_EQUAL_INT(0, f.ctx.alloc->nres_shared);
TEST_ASSERT_NULL(f.ctx.alloc->allocs_shared);
eq_assert_debit(&f, a, 2.0f, 2.0f);
eq_assert_debit(&f, b, 6.0f, 3.0f);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(3, 2, 2);
free_JobAlloc_ptr(&first);
xlist_fini(&(cres.primaries));
eq_fixture_fini(&f);
eq_primary_fini(&p1);
eq_primary_fini(&p2);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_parjob_success_two_buckets_with_shared);
RUN_TEST(test_parjob_pending_on_short_shared_resource);
RUN_TEST(test_parjob_pending_on_short_shared_no_reason_today);
RUN_TEST(test_parjob_pending_on_hosts);
RUN_TEST(test_parjob_mid_plan_rollback_then_later_plan_wins);
RUN_TEST(test_parjob_host_reasons_first_candidate_only_today);
RUN_TEST(test_parjob_task_count_zero_and_one_pend_without_reason);
RUN_TEST(test_parjob_missing_input_einval_no_reason);
RUN_TEST(test_parjob_oom_records_host0_releases_all);
RUN_TEST(test_parjob_oom_records_later_host_releases_unconverted);
RUN_TEST(test_parjob_oom_host_alloc0_releases_all);
RUN_TEST(test_parjob_oom_host_alloc_later_releases_unconverted);
RUN_TEST(test_parjob_oom_job_alloc_debits_stay);
RUN_TEST(test_parjob_oom_shared_records_debits_stay);
RUN_TEST(test_parjob_compound_second_primary_sized_from_own_request);
return UNITY_END();
}