myxce/.kiro/specs/scheduler-allocator-runner/design.md

# Design Document: scheduler-allocator-runner

## Overview
**Purpose**: The scheduler framework takes ownership of the job-allocation algorithm. One framework routine, `sched_allocator_run()`, performs the shared-resource precheck, the plan → bucket → host claim search with rollback, and result assembly for every allocator plugin. An allocator plugin supplies only a candidate-plan iterator, an optional acceptance guard, and its existing `prepare_allocation_hook`.

**Users**: Scheduler plugin authors (a new allocator needs only a plan iterator) and scheduler maintainers (an allocation fix lands once).

**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the per-candidate reset are under System Flows → Consequences.

### Goals
- One allocation algorithm, used by seqjob and parjob in normal and simulation mode (1.1, 6.3).
- Behavioural equivalence proven by characterization tests written against the current code before it is removed (7.2).
- `issues/new_requirements.md` item 4 closed (7.4).

### Non-Goals
- A per-plugin result-assembly (`commit`) hook.
- Retyping `sched_make_allocs()` or removing `plugin->sched`.
- Merging `sched_make_seqjob_allocation()` / `sched_make_parjob_allocation()` dispatch.
- Adding an allocation duration (the `FIXME: no duration now` stays).
- Fixing bug-1892 (`unsatisfy_()` reads uncommitted amounts by position).
- Publishing pend reasons only when they change (follow-up to the larger per-host reason sets of 3.7).

## Boundary Commitments

### This Spec Owns
- `xce_sched_allocator_iface_t` (shape, ownership, lifetime) and the `allocator_iface` field of `xce_sched_plugin_t`.
- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-candidate `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
- The `xce_alloc_plan_iter_t.init` parameter type.
- The seqjob/parjob plugin wiring (templates, iface instances, `parjob_accepts`) and the removal of their private allocation code.
- Equivalence and runner tests; the pend-reason factory audit's file list; `issues/new_requirements.md` item 4.

### Out of Boundary
- Dispatch (`scheduler.c` `sched_make_primary_alloc_()` and the COMPOUND/ALTER callers): routing by `maxTasks`, the unsatisfiable cache, result merging, session stamping.
- `resource.c` satisfy checks (`sched_check_shared_satisfy`, `sched_check_host_satisfy`, `unsatisfy_`), `sched_try_alloc_on_host_()`, `plugin_prepare_allocation()`, the host iterator, and both plugins' `prepare_allocation_hook`s: called unchanged.
- The plan iterators' plan sequences (only the parameter type changes).
- `sched_make_allocs()`, `make_*_alloc` constructors in libxce, `sim_alloc.c` registration (comment updates only).
- Snapshot, WAL, recording formats: nothing persisted changes.

### Allowed Dependencies
- `alloc_runner.c` → `resource.c` (`sched_try_alloc_on_host_`, `sched_check_shared_satisfy`, `sched_ctx_uncommit_host`), `schedobj.c` (`sched_ctx_get_uncomit_shared`), `host_iter.c` (`sched_host_iter_*`), `scheduler.c` (`sched_make_allocs` only), libxce (`make_pendreason`, `add_job_preason`, `make_host_alloc`, `make_job_alloc`, `free_*`, `xvec_*`).
- Plugins → `alloc_runner.c` only through `sched_allocator_run` and the iface type.
- `alloc_runner.c` must not reference any plugin file or plugin-specific symbol; plugins must not call runner internals.

### Revalidation Triggers
- Any change to `xce_sched_allocator_iface_t` or `xce_alloc_plan_iter_t` signatures (all allocator plugins, `test_parjob_combinatorial.c`, `test_alloc_plan_iter.c`, `test_alloc_runner.c`'s fake iface).
- Any change to the order of guard → precheck → search → assembly, or to when `ctx->reason`/`ctx->alloc` are set (dispatch, unsatisfiable cache, COMPOUND merge, `scheduler-alter-pend-reasons`).
- Fixing bug-1892 (changes COMPOUND admission; re-run COMPOUND characterization tests).
- A new allocator plugin (must supply a plan iterator whose buckets are ≤ its task count).

## Architecture

### Existing Architecture Analysis
`sched_make_primary_alloc_()` picks `core->seqjob_allocator` (`maxTasks == 1`) or `core->parjob_allocator` and calls `plugin->job_alloc_fn(plugin, ctx)`. Each plugin's `my_alloc()` then repeats the whole algorithm. Plan generation already sits behind `xce_alloc_plan_iter_t`, host scanning behind `xce_host_iter_t`, per-host preparation behind `plugin_prepare_allocation()`. The picker side already uses a framework-owned iface (`picker_iface`) with a forward typedef in `plugin.h`.

### Architecture Pattern & Boundary Map
Template method: the framework owns the algorithm; the plugin fills in plan generation.

```mermaid
graph TB
    Dispatch[scheduler dispatch] --> JobAllocFn[plugin job_alloc_fn]
    JobAllocFn --> Runner[sched_allocator_run]
    Runner --> Iface[allocator iface]
    Iface --> SeqPlans[seqjob plan iterator]
    Iface --> ParPlans[parjob plan iterator]
    Iface --> ParAccepts[parjob accepts]
    Runner --> SharedCheck[resource shared check]
    Runner --> HostIter[host iterator]
    Runner --> TryHost[try alloc on host]
    TryHost --> Prepare[plugin prepare allocation]
    Prepare --> SeqPrep[seqjob prepare hook]
    Prepare --> ParPrep[parjob prepare hook]
    Runner --> MakeAllocs[sched make allocs]
```

- **Dependency direction**: libxce → `resource.c`/`schedobj.c`/`host_iter.c` → `alloc_runner.c` → plugins (via iface data only) → dispatch. Plugins reference `sched_allocator_run` by address in their template; the runner never names a plugin. One sanctioned exception: the runner calls `scheduler.c`'s `sched_make_allocs()` (a leftward call into the dispatch file), exactly as both `my_alloc()`s do today; removing it is the out-of-boundary `sched_make_allocs()` retype follow-up. No other `scheduler.c` symbol may be used.
- **Existing patterns preserved**: static iface storage referenced from the plugin struct (picker); white-box test `#include` of `.c` files; `sched_make_allocs()` stubbed in white-box tests.
- **Steering compliance**: new file under the 800-line cap; plugins stay under `src/plugins/`; no I/O, no locks, no blocking added (allocator thread only).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Scheduler daemon | C (gnu17), Meson | Runner, iface, plugin wiring | No new dependency |
| Tests | Unity, white-box `#include` | Characterization + runner tests | Existing convention |

## File Structure Plan

### Directory Structure
```
src/scheduler/
├── include/
│   ├── plugin.h                 # + forward typedef xce_sched_allocator_iface_t; + allocator_iface field
│   └── sched/alloc_iter.h       # plan iterator init retyped; + xce_sched_allocator_iface_t; + sched_allocator_run prototype
├── src/
│   ├── alloc_runner.c           # NEW: sched_allocator_run and its static steps
│   ├── sim_alloc.c              # comments only (my_alloc references)
│   └── plugins/
│       ├── seqjob_allocator.c   # - my_alloc; + seqjob_alloc_iface; template -> sched_allocator_run
│       └── parjob_allocator.c   # - my_alloc, claim helpers; + parjob_accepts, parjob_alloc_iface
├── meson.build                  # + src/alloc_runner.c in scheduler_srcs
└── testsuite/
    ├── test_alloc_runner.c      # NEW: runner tests with an in-test fake plan iterator
    ├── test_alloc_equivalence_seqjob.c   # NEW: seqjob characterization tests via job_alloc_fn
    ├── test_alloc_equivalence_parjob.c   # NEW: parjob characterization tests via job_alloc_fn
    └── meson.build              # + three targets
```

### Modified Files
- `src/scheduler/include/sched/alloc_iter.h` — `init`'s third parameter becomes `xce_sched_core_t* core`; defines `xce_sched_allocator_iface_t` and declares `sched_allocator_run`.
- `src/scheduler/include/plugin.h` — forward typedef and `const xce_sched_allocator_iface_t* allocator_iface;` in `xce_sched_plugin_t`, documented with the same ownership note as `picker_iface`.
- `src/scheduler/src/plugins/seqjob_allocator.c` — plan-iterator `init` signature; delete `my_alloc()`; add `static const xce_sched_allocator_iface_t seqjob_alloc_iface` (`accepts = NULL`); template `.job_alloc_fn = sched_allocator_run`, `.allocator_iface = &seqjob_alloc_iface`; update comments.
- `src/scheduler/src/plugins/parjob_allocator.c` — same, plus `parjob_accepts()` (`maxTasks > 1`); delete `parjob_claim_t`, `parjob_is_claimed_`, `parjob_snapshot_host_req_`, `parjob_rollback_claims_`, `parjob_try_plan_`.
- `src/scheduler/src/sim_alloc.c` — comment text only.
- `src/scheduler/meson.build` — add `src/alloc_runner.c`.
- `src/scheduler/testsuite/test_seqjob_allocator.c`, `test_seqjob_rewrap_equivalence.c`, `test_parjob_combinatorial.c` — add `#include "../src/alloc_runner.c"`; replace direct `my_alloc()`/`parjob_try_plan_()`/`parjob_rollback_claims_()` calls with `sched_allocator_run()`/`sched_try_plan_()`/`sched_rollback_claims_()`; plan-iterator `init` third argument becomes a core pointer.
- `src/scheduler/testsuite/test_alloc_plan_iter.c` — its `stub_plan_iter_init` third parameter becomes `xce_sched_core_t*` (plan-iterator retype).
- `src/scheduler/testsuite/test_picker_pendreason_factory_audit.c` — add `src/alloc_runner.c` to the scanned list.
- `src/scheduler/testsuite/meson.build` — three new targets.
- `issues/new_requirements.md` — item 4 marked resolved.

## System Flows

```mermaid
flowchart TD
    Start[sched_allocator_run] --> Inputs{inputs present}
    Inputs -->|no| EInval[return EINVAL]
    Inputs -->|yes| Accept{accepts}
    Accept -->|rejected| Pend0[return 0]
    Accept -->|accepted or NULL| Reason[ensure pend reason record]
    Reason --> Shared{shared request met}
    Shared -->|error| Err[return error]
    Shared -->|no| SharedReasons[one reason per unmet entry then return 0]
    Shared -->|yes| Uncommit[count shared as uncommitted]
    Uncommit --> Init{plan init ok}
    Init -->|no| Pend1[return 0]
    Init -->|yes| Next{next plan}
    Next -->|none| Fini0[fini and return 0]
    Next -->|plan| Grow{claims buffer fits}
    Grow -->|grow fails| FiniOom[fini and return ENOMEM]
    Grow -->|malformed plan| Next
    Grow -->|ok| TryPlan[try plan]
    TryPlan -->|bucket unsatisfied| Next
    TryPlan -->|snapshot OOM| FiniErr[fini and return error]
    TryPlan -->|all buckets claimed| Finalize[build result]
    Finalize -->|OOM| FiniOom2[fini and return ENOMEM]
    Finalize -->|ok| Done[publish ctx alloc then fini and return 1]
```

**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan (and reset `alloc_req.host` again after every rejected candidate, before the next `next()`); set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.

**Key decisions**: the reset and the `nslots`/`nhosts` assignment are unconditional (brief items 1 and 3); every return after `plan_init` succeeded calls `plan_fini` and frees the claims buffer; claim snapshots are freed by whichever function owns them at that point (see AllocRunner → State Management).

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | One routine for both plugins | AllocRunner, PluginWiring | `sched_allocator_run` | Start |
| 1.2 | Plugin supplies plan iterator, accepts, prepare hook only | AllocatorIface | `xce_sched_allocator_iface_t` | — |
| 1.3 | NULL accepts = accept all | AllocRunner | `accepts` | Accept |
| 1.4 | Rejected = pending, no reason, no checks | AllocRunner | `accepts` | Accept → Pend0 |
| 1.5 | No private allocation routine in plugins | PluginWiring | — | — |
| 1.6 | Iface static, not freed, plugin lifetime | AllocatorIface, PluginWiring | `allocator_iface` | — |
| 2.1 | Reason record created after guards | AllocRunner (`ensure_reason`) | — | Reason |
| 2.2 | One reason per unmet entry (bug-1893) | AllocRunner (`prepare_shared`) | — | SharedReasons |
| 2.3 | Shared counted once, not released | AllocRunner (`prepare_shared`) | — | Uncommit |
| 2.4 | Prep failures return their error | AllocRunner | — | Err |
| 3.1 | Plans in generator order, stop at first satisfied | AllocRunner | `plan_next` | Next |
| 3.2 | Buckets in order, first unclaimed satisfying host | ClaimSearch | `sched_host_iter_*` | TryPlan |
| 3.3 | Roll back claims on unsatisfied bucket | ClaimSearch | `sched_ctx_uncommit_host` | TryPlan |
| 3.4 | Init failure / no plan = pending | AllocRunner | `plan_init` | Pend1, Fini0 |
| 3.5 | `nslots` = bucket slots, `nhosts` = 1 | ClaimSearch | — | TryPlan |
| 3.6 | Empty per-host request before each candidate | ClaimSearch | — | TryPlan |
| 3.7 | Per-host reasons complete, both allocators | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
| 3.8 | No fixed bucket limit | AllocRunner (claims buffer) | — | Grow |
| 4.1 | One host alloc per claim, claim order, snapshot request | ResultAssembly | `sched_make_allocs`, `make_host_alloc` | Finalize |
| 4.2 | Shared record per shared-request entry | ResultAssembly | `sched_make_allocs` | Finalize |
| 4.3 | jobid 0, empty reserve, no duration | ResultAssembly | `make_job_alloc(0, …)` | Finalize |
| 4.4 | Satisfied → set alloc, return 1 | AllocRunner | — | Done |
| 4.5 | Results identical to current allocators | ResultAssembly, EquivalenceTests | — | — |
| 5.1 | Missing inputs → EINVAL, no reason | AllocRunner | — | EInval |
| 5.2 | parjob `maxTasks <= 1` → pending | ParjobAccepts | `parjob_accepts` | Pend0 |
| 5.3 | seqjob accepts all | PluginWiring (`accepts = NULL`) | — | Accept |
| 5.4 | OOM in result → ENOMEM, nothing left | ResultAssembly | — | FiniOom2 |
| 5.5 | OOM releases unconverted claims + failed candidate | ClaimSearch, ResultAssembly | `sched_rollback_claims_` | FiniErr, FiniOom2 |
| 5.6 | `ctx->alloc` set only on success | ResultAssembly | — | Done |
| 5.7 | Fini iterator and free snapshots on every exit | AllocRunner | `plan_fini` | all Fini nodes |
| 5.8 | Size from each primary's own request | AllocRunner (uses `ctx->primary` / `ctx->alloc_req` only) | — | — |
| 6.1 | Plan init takes core | PlanIterContract | `init(iter, resreq, core)` | Init |
| 6.2 | Identical plan sequences | PlanIterContract, EquivalenceTests | — | — |
| 6.3 | Sim uses the same routine, no scheduler object to plan iterators | PluginWiring, PlanIterContract | — | — |
| 7.1 | Existing tests pass, limited test edits | TestMigration | — | — |
| 7.2 | Equivalence tests | EquivalenceTests, RunnerTests | — | — |
| 7.3 | Factory audit covers runner | TestMigration | — | — |
| 7.4 | Item 4 resolved | Closure | — | — |
| 7.5 | Invariant audit clean | Closure | — | — |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|--------------|------------------|-----------|
| AllocatorIface | Plugin ABI (`alloc_iter.h`, `plugin.h`) | Declares what an allocator plugin supplies | 1.2, 1.3, 1.6 | — | Service |
| PlanIterContract | Plugin ABI (`alloc_iter.h`) | Plan iterator `init` takes the core | 6.1, 6.2, 6.3 | — | Service |
| AllocRunner | Framework (`alloc_runner.c`) | Guards, shared precheck, plan loop, cleanup | 1.1, 1.3, 1.4, 2.x, 3.1, 3.4, 3.8, 4.4, 5.1, 5.7, 5.8 | resource.c (P0), schedobj.c (P0) | Service, State |
| ClaimSearch | Framework (`alloc_runner.c`) | Bucket → host claim, snapshot, rollback | 3.2, 3.3, 3.5, 3.6, 3.7, 5.5 | host_iter.c (P0), resource.c (P0) | Service |
| ResultAssembly | Framework (`alloc_runner.c`) | Build and publish `xce_job_alloc_t` | 4.1–4.5, 5.4–5.6 | `sched_make_allocs` (P0), libxce (P0) | Service |
| PluginWiring | Plugins (`seqjob_allocator.c`, `parjob_allocator.c`) | Templates + iface instances; delete private code | 1.1, 1.5, 1.6, 5.2, 5.3, 6.3 | AllocRunner (P0) | — |
| ParjobAccepts | Plugin (`parjob_allocator.c`) | `maxTasks > 1` guard | 5.2 | — | Service |
| EquivalenceTests / RunnerTests / TestMigration / Closure | Tests, docs | Evidence and closure | 4.5, 6.2, 7.1–7.5 | — | — |

### Plugin ABI

#### AllocatorIface and PlanIterContract

| Field | Detail |
|-------|--------|
| Intent | The allocator plugin contract and the retyped plan-iterator initialiser |
| Requirements | 1.2, 1.3, 1.6, 6.1, 6.2, 6.3 |

**Contracts**: Service [x]

##### Service Interface
```c
/* include/sched/alloc_iter.h */
struct xce_alloc_plan_iter_s {
    int (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq,
                xce_sched_core_t* core);            /* was xce_scheduler_t* */
    xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter);
    void (*fini)(xce_alloc_plan_iter_t* iter);
    void* private;
};

struct xce_sched_allocator_iface_s {
    /* Optional. 1 = this plugin allocates `primary`, 0 = not (pending, no
     * reason, no checks). NULL = accept every request. */
    int (*accepts)(const xce_primary_resreq_t* primary);
    /* Required. Same contract as xce_alloc_plan_iter_t init/next/fini. */
    int (*plan_init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq,
                     xce_sched_core_t* core);
    xce_alloc_plan_t* (*plan_next)(xce_alloc_plan_iter_t* iter);
    void (*plan_fini)(xce_alloc_plan_iter_t* iter);
};

/* job_alloc_fn for every allocator plugin; `plugin` is xce_sched_plugin_t*.
 * > 0 satisfied (ctx->alloc set), 0 pending, < 0 negative errno. */
extern int sched_allocator_run(void* plugin, xce_sched_ctx_t* ctx);

/* include/plugin.h */
typedef struct xce_sched_allocator_iface_s xce_sched_allocator_iface_t;
/* in xce_sched_plugin_t: static storage owned by the plugin file; the
 * registry never frees it (same ownership as picker_iface). NULL for
 * non-allocator plugins. */
const xce_sched_allocator_iface_t* allocator_iface;
```
- **Preconditions**: for an allocator plugin, `allocator_iface` and its three `plan_*` members are non-NULL; every plan's `nbuckets >= 1` and each `bucket_slots[i] >= 1`.
- **Postconditions**: `plan_init` returns 0 and leaves `iter` ready for `plan_next`, or non-zero with nothing for the runner to release (the runner then skips `plan_fini`). It may read the core; both built-in iterators ignore it.
- **Invariants**: the iface is `static const` in the plugin file; plugin templates copy the pointer on registration (1.6).

**Implementation Notes**
- Integration: seqjob/parjob iterator functions and `test_alloc_plan_iter.c`'s stub iterator change only their third parameter's type; `test_parjob_combinatorial.c`'s poisoned-scheduler case becomes a poisoned-core case (6.2 evidence: plan sequences equal with `NULL` and poisoned core).
- Risks: a plugin that sets `job_alloc_fn = sched_allocator_run` without an iface gets `-EINVAL` (5.1), not a crash.

### Framework runner (`src/scheduler/src/alloc_runner.c`)

#### AllocRunner

| Field | Detail |
|-------|--------|
| Intent | The one allocation routine |
| Requirements | 1.1, 1.3, 1.4, 2.1–2.4, 3.1, 3.4, 3.8, 4.4, 5.1, 5.7, 5.8 |

**Responsibilities & Constraints**
- Order: (1) input guard: `plugin`, `ctx`, `plugin->core`, `plugin->sched`, `ctx->primary`, `plugin->allocator_iface` and its `plan_*` members non-NULL, else `-EINVAL` before touching `ctx->reason` (5.1); (2) `accepts` if non-NULL, reject → 0 (1.3, 1.4); (3) `ensure_reason`: create `make_pendreason(0)` if `ctx->reason == NULL`, `-ENOMEM` on failure (2.1, 2.4); (4) `prepare_shared` (below); (5) `plan_init(iter, ctx->primary, plugin->core)`, non-zero → 0 without `plan_fini` (3.4, as today); (6) plan loop; (7) cleanup.
- `prepare_shared`: `sched_ctx_get_uncomit_shared`; `xvec_idval_fini(&ctx->unmeet)`; `sched_check_shared_satisfy(core, uncomit, &ctx->alloc_req.shared, &ctx->unmeet)`; `< 0` → return it; `== 0` → for `i < ctx->unmeet.count`: `vget(resmgr, unmeet.idxs[i])`, skip only on `rc < 0` (`vget` returns 0 on success; today's `rc <= 0` skips every hit, bug-1893), `add_job_preason(reason, PREASON_NO_RESOURCE, NULL, r->name)`; return 0 (2.2); `> 0` → `xvec_idval(uncomit, &ctx->alloc_req.shared, XVEC_ADD_FLOAT)` once, error → return it (2.3, 2.4). Shared amounts are never subtracted by the runner (2.3).
- Claims buffer: `sched_alloc_claim_t* claims` (heap), capacity grown with `realloc` to `plan->nbuckets` before a plan is tried; growth failure → `plan_fini`, free, `-ENOMEM` (3.8). No claim is outstanding between plans.
- Uses only `ctx->primary` and `ctx->alloc_req` for sizing (5.8).
- Precondition `ctx->alloc == NULL` on entry (the dispatch and COMPOUND/ALTER callers move or free any previous result before the next attempt); the runner never frees or overwrites a non-NULL `ctx->alloc` except with its own result on success.
- A plan with `nbuckets < 1` or any `bucket_slots[i] < 1` violates the iterator precondition; the runner skips it as unsatisfied (built-in iterators never produce one, so this changes no current outcome).

**Contracts**: Service [x] / State [x]

##### Service Interface
```c
int sched_allocator_run(void* plugin, xce_sched_ctx_t* ctx);

/* static, reachable from white-box tests */
static int sched_alloc_ensure_reason_(xce_sched_ctx_t* ctx);          /* 0 | -ENOMEM */
static int sched_alloc_prepare_shared_(xce_sched_core_t* core,
                                       xce_sched_ctx_t* ctx);          /* 1 met | 0 pend | <0 */
```

##### State Management
- Attempt-owned: plan iterator private state and the claims buffer, released by the runner on every return path after `plan_init` succeeded (5.7). Claim snapshots are owned by exactly one function at a time: `sched_try_plan_()` creates them and frees them itself on 0 / `< 0`; on 1 it hands all of them to `sched_finalize_job_alloc_()`, which frees every one on every path. The runner itself never frees a snapshot, so no path can free one twice.
- Thread cancellation: `plan_next` (both iterators) and `sched_host_iter_next()` are `pthread_testcancel()` points, and the allocator thread is cancelled only at shutdown (`scheduler.c:905`, `sim.c:727`). A cancel inside an attempt abandons the attempt-scoped heap (iterator state, host-iterator state, claim snapshots, and now the claims buffer) exactly as today; the runner adds no `pthread_cleanup_push()` handlers, consistent with the rest of the scheduler, which has none. "Every return" in 5.7 means every return path, not an asynchronous cancel.
- Context-owned (not released by the runner): `ctx->reason`, shared uncommitted amounts, per-host debits of claims already converted into host allocations (5.5 wording), `ctx->alloc` on success.

#### ClaimSearch

| Field | Detail |
|-------|--------|
| Intent | Try one plan: claim a distinct host per bucket, or roll back |
| Requirements | 3.2, 3.3, 3.5, 3.6, 3.7, 5.5 |

##### Service Interface
```c
typedef struct {
    xce_sched_worker_t* w;
    xvec_idval_t        host_req;   /* deep copy of ctx->alloc_req.host at claim */
} sched_alloc_claim_t;

/* 1 all buckets claimed (claims[0..*nclaimed) owned by caller)
 * 0 a bucket had no host (all claims rolled back, snapshots freed)
 * <0 snapshot failure (candidate + earlier claims rolled back, snapshots freed) */
static int  sched_try_plan_(xce_sched_core_t* core, xce_sched_ctx_t* ctx,
                            xce_alloc_plan_t* plan, sched_alloc_claim_t* claims,
                            int* nclaimed);
static void sched_rollback_claims_(xce_sched_ctx_t* ctx,
                                   sched_alloc_claim_t* claims, int n);
static int  sched_snapshot_host_req_(xvec_idval_t* dst, const xvec_idval_t* src);
```
- **Preconditions**: `claims` has capacity `>= plan->nbuckets`.
- **Postconditions**: on 0 or `< 0`, every per-host debit made in this call is released (3.3, 5.5) and no snapshot remains.
- Behaviour is parjob's current `parjob_try_plan_()` with `plan->ctx = ctx` and the per-candidate `nslots`/`nhosts` assignment (3.5), applied to every plugin, and one change: `alloc_req.host` is cleared at the start of each bucket AND after every rejected candidate, i.e. before every `host_iter->next()` call (3.6). Clearing only once per bucket (parjob today) is not enough: a rejected candidate leaves its own per-host request in `alloc_req.host`, and the host iterator's pre-filter (`host_iter.c`) would then skip later short hosts silently. With the request always empty when `next()` pre-filters, the pre-filter does select-expression filtering only, every selection-matching unclaimed host reaches `sched_try_alloc_on_host_()`, and each short one gets its reasons (3.7).

#### ResultAssembly

| Field | Detail |
|-------|--------|
| Intent | Turn claims into the published `xce_job_alloc_t` |
| Requirements | 4.1–4.5, 5.4, 5.5, 5.6 |

##### Service Interface
```c
/* 1: ctx->alloc set; -ENOMEM: ctx->alloc untouched, everything built freed.
 * Consumes claims: each snapshot is freed once its host alloc is built;
 * on failure, claims not yet converted are rolled back and freed. */
static int sched_finalize_job_alloc_(xce_scheduler_t* sched, xce_sched_ctx_t* ctx,
                                     sched_alloc_claim_t* claims, int nclaimed);
```
- Steps, matching parjob's current tail: `calloc(nclaimed)` host-alloc array (failure: roll back all claims); per claim `sched_make_allocs(sched, "", nres, idxs, vals, NULL)` (`/* FIXME: no duration now */` kept) then `make_host_alloc(w->name, nres, allocs)`, where `nres == 0` allows a NULL `allocs` (failure: free `allocs`, roll back claims `[i, n)`, `free_HostAlloc_list(nclaimed, hallocs)`); `make_job_alloc(0, nclaimed, hallocs)` into a local (failure: `free_HostAlloc_list`); shared records `sched_make_allocs(sched, "", shared.count, …, NULL)` when `alloc_req.shared.count > 0`, else `allocs_shared = NULL, nres_shared = 0` (failure: `free_JobAlloc_ptr(&local)`); then `ctx->alloc = local` (5.6).
- `sched` is `(xce_scheduler_t*)plugin->sched` (the stand-in wrapper in sim), used only for `sched_make_allocs()` (out-of-boundary retype).
- Equivalence note (4.5): seqjob today builds its one-element host array with `make_host_allocs(1, halloc)`; the runner's `calloc` + store produces the same array, freed by the same `free_HostAlloc_list`.

### Plugins

#### PluginWiring and ParjobAccepts

| Field | Detail |
|-------|--------|
| Intent | Register both plugins on the runner |
| Requirements | 1.1, 1.5, 1.6, 5.2, 5.3, 6.3 |

```c
/* parjob_allocator.c */
static int parjob_accepts(const xce_primary_resreq_t* primary)
{ return primary->maxTasks > 1; }                       /* bug-1767 */
static const xce_sched_allocator_iface_t parjob_alloc_iface = {
    .accepts = parjob_accepts, .plan_init = parjob_plan_iter_init,
    .plan_next = parjob_plan_iter_next, .plan_fini = parjob_plan_iter_fini };

/* seqjob_allocator.c */
static const xce_sched_allocator_iface_t seqjob_alloc_iface = {
    .accepts = NULL, .plan_init = seqjob_plan_iter_init,
    .plan_next = seqjob_plan_iter_next, .plan_fini = seqjob_plan_iter_fini };

/* both templates */
.job_alloc_fn = (plugin_job_allocate_fn_t)sched_allocator_run,
.allocator_iface = &<plugin>_alloc_iface,
```
- `prepare_allocation_hook`, `init_fn`, `fini_fn`, `*_new_plugin()`, `register_*()` unchanged; sim registration in `sim_alloc.c` therefore reaches the runner unchanged (6.3).

### Consequences of the per-candidate reset (3.6, both allocators)
Because the host iterator's pre-filter no longer skips hosts on stale state, a pending attempt of either allocator now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching unclaimed host that is short of resources. Today that happens only for the first rejected candidate of each bucket search (parjob) or for candidates that pass the stale filter (seqjob). Besides the complete per-host pend reasons (3.7), this means:
- **Limit plugin side effects** (`limit.c` `match_job_host_limit_`): for a host with a matching host-based limit it allocates the job's `obj->hlimits[worker]` list, may instantiate the limit's virtual resource in `resmgr` (`instantiate_limit`) and writes the limit value into `worker->resvals`. This is decision-neutral: any later check of that host by any job runs the same lazy instantiation before reading the value, so it only moves when the instance is created. It happens identically in normal and simulation mode (driver state-equivalence holds).
- **Cost**: one prepare fan-out plus satisfy check per short host per bucket search, where the stale pre-filter used to reject later hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
- **Pend-reason volume**: a pending job can now carry one reason per short host (per bucket search for parjob). These reasons are published to the controller and persisted with the job. Publishing reasons only when they change is a recorded follow-up (Non-Goals), not part of this spec.

## Data Models
No persisted or wire data changes. In-memory only: `sched_alloc_claim_t` (attempt-scoped) and the `allocator_iface` pointer (static). Snapshot/WAL/recording field-completeness is not affected (scheduler audit dimension 4: no change).

## Error Handling

### Error Strategy
The runner follows the allocator contract: `> 0` satisfied, `0` pending, `< 0` negative errno, early return on each error (project C convention), with cleanup of attempt-owned state on every path after `plan_init`.

| Condition | Result | State left behind |
|-----------|--------|-------------------|
| Missing plugin/ctx/core/sched/primary/iface | `-EINVAL` | nothing; `ctx->reason` untouched |
| `accepts` rejects | 0 | nothing |
| `make_pendreason` fails | `-ENOMEM` | `ctx->reason == NULL` |
| Shared check error / uncommit add error | that error | reason record |
| Shared unmet | 0 | reason record with one `NO_RESOURCE` per unmet entry |
| `plan_init` fails | 0 | reason record, shared uncommitted |
| No plan satisfied | 0 | reason record (+ host reasons), shared uncommitted |
| Plan with `nbuckets < 1` or a bucket of `< 1` slots | plan skipped as unsatisfied | nothing from that plan |
| Claims buffer growth fails | `-ENOMEM` | as above; no host debit outstanding |
| Snapshot fails | its error (`-ENOMEM`) | candidate and earlier claims released |
| Host alloc build fails | `-ENOMEM` | unconverted claims released; converted ones' debits stay in ctx (as today) |
| Job alloc / shared records fail | `-ENOMEM` | `ctx->alloc` untouched; debits stay in ctx (as today) |

### Monitoring
Existing `TRACE` lines in `resource.c` are unchanged; the runner adds none (the allocator path runs per candidate and is already traced there).

## Testing Strategy

TDD order follows the staged migration (research.md): characterization tests are written and pass against the current code first, then must pass unchanged after each switch.

### Unit Tests
- `test_alloc_runner.c` (white-box `#include` of `core.c`, `resource.c`, `schedobj.c`, `host_iter.c`, `alloc_runner.c`; stubbed `sched_make_allocs` with OOM injection; stubbed `plugin_prepare_allocation()` that records `ctx->nslots`/`ctx->nhosts`/`ctx->alloc_req.host.count` on each call and stages a fixed per-host request, the same link-stub convention `test_resource_accounting.c` uses; an in-test fake iface whose plans are scripted):
  - guards: each missing input → `-EINVAL`, `ctx->reason` still NULL (5.1); rejecting `accepts` → 0, no reason, `plan_init` not called (1.4); NULL `accepts` reaches `plan_init` (1.3).
  - shared: two shared resources, one short → exactly one host-less `NO_RESOURCE` reason naming it (2.2, bug-1893); met → uncommitted added once across a 3-plan attempt (2.3).
  - search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty before every candidate's pre-filter: with three short hosts in one bucket, all three get host/resource reasons (today only the first would) (3.6, 3.7); a plan with more buckets than a first small plan triggers buffer growth (3.8).
  - errors: snapshot OOM, host-alloc OOM at claim 0 and claim k, job-alloc OOM, shared-records OOM → `-ENOMEM`, `ctx->alloc == NULL`, released debits as in the Error table, ASAN clean (5.4, 5.5, 5.6, 5.7).
- `test_parjob_combinatorial.c`: plan sequences identical for `NULL` and poisoned core (6.2).

### Equivalence (characterization) Tests
- `test_alloc_equivalence_seqjob.c` / `test_alloc_equivalence_parjob.c` (white-box `#include` of `core.c`, `resource.c`, `schedobj.c`, `host_iter.c`, the plugin file, and from its switch step `alloc_runner.c`; `sched_make_allocs` stubbed with OOM injection; `plugin_prepare_allocation()` stubbed by the existing convention of `test_seqjob_allocator.c:84` — reset `alloc_req.shared` or `.host` like `plugin.c`, then call the plugin's own prepare hook — so limit fill is not exercised here; limits stay covered through the real `plugin.c` by `test_sim_alloc_limit_fanout`): drive the real plugin through `plugin->job_alloc_fn` on fixed fixtures (success; pending on shared; pending on hosts; parjob mid-plan rollback; parjob `maxTasks` 0 and 1; seqjob with `maxTasks` 0/2 via direct call; COMPOUND second primary with non-overlapping resources so bug-1892 is not exercised). Assert result code, host count/names, per-host resource entries/values, shared records, per-host and shared uncommitted debits, and pend reasons. Written and passing against today's code first (step 2). The fixtures for the intentional changes that the current code can run (seqjob per-host reasons, 3.7; bug-1893, 2.2; seqjob OOM rollback, 5.5) assert the new expected values: they are added at the start of the step that switches that plugin (RED against the not-yet-switched plugin) and turn GREEN within that step, so no step ends with a failing test (4.5, 7.2). seqjob's missing-input fixtures (5.1) are added after the switch only: today's seqjob dereferences the missing pointer, so they cannot fail as an assertion first; `test_alloc_runner.c` covers the same guard with the fake iface before the switch.

### Regression (existing)
- The Req 7.1 list, unchanged in expected outcomes: `test_seqjob_allocator`, `test_seqjob_rewrap_equivalence`, `test_parjob_combinatorial`, `test_parjob_e2e_multislot`, `test_maxtasks_wiring_*`, `test_scheduler_cache_dispatch*`, `test_sim_contention_*parity`, `test_sim_live_allocation`, `test_sim_alloc_limit_fanout`, `test_sched_election_sim_fixture`, `test_snapshot_load_allocator_regression`; `test_picker_pendreason_factory_audit` with the new file listed (7.3). Full `/xce-build scheduler` (ASAN) suite green.

### Invariant audit
- `/xce-audit-invariants scheduler` after the switch (7.5). Dimensions checked at design time: (1) driver state-equivalence — the runner is reached identically from `driver.c` and `sim_loop.c` via the same plugins; (2) core I/O isolation — `alloc_runner.c` does no I/O; (3) picker→allocator rendezvous — untouched; (4) snapshot parity — no persisted field changes.

## Migration Strategy
1. Retype plan-iterator `init` (header, both iterators, `test_parjob_combinatorial.c`, `test_alloc_plan_iter.c`).
2. Characterization tests against current code (pass).
3. Add `alloc_runner.c`, iface, `plugin.h` field; switch parjob; delete parjob's private code; update white-box tests.
4. Switch seqjob; delete `my_alloc()`; update white-box tests.
5. Closure: comments in both plugins and `sim_alloc.c`, factory audit list, `issues/new_requirements.md` item 4, invariant audit.

Each step ends with the full scheduler suite green; the only RED tests within a step are that step's own intentional-change fixtures, and a failing step-2 characterization test at steps 3–4 blocks the step.