test(scheduler-allocator-runner): characterize the parjob allocator (task 2.2)

15 tests drive the real parjob plugin through job_alloc_fn: success,
short shared, pending on hosts, mid-plan rollback, task counts 0/1,
missing inputs, six OOM steps with their debits, COMPOUND. Values the
runner changes on purpose sit in *_today tests.

Spec: the task showed that parjob records host reasons only for a bucket
search's first rejected candidate (the rejected host's request stays in
alloc_req.host and the host iterator pre-filters later hosts against
it). Decision: the runner clears alloc_req.host before every candidate,
for both allocators, so every short host gets its reasons (Req 3.6/3.7);
publishing reasons only when they change is a follow-up.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/scheduler-allocator-runner/design.md

@@ -5,7 +5,7 @@
 
 **Users**: Scheduler plugin authors (a new allocator needs only a plan iterator) and scheduler maintainers (an allocation fix lands once).
 
-**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the seqjob per-bucket reset are under System Flows → Consequences.
+**Impact**: Deletes `my_alloc()` from `seqjob_allocator.c` and `parjob_allocator.c` and parjob's claim helpers; adds `alloc_runner.c`; adds an allocator iface to the plugin ABI; retypes the plan iterator's `init` third parameter from `xce_scheduler_t*` to `xce_sched_core_t*`. Allocation decisions, host choice, main pend reason and result codes are unchanged for every input that reaches the allocators today. The intentional changes are the four listed in `requirements.md` → Introduction; the side effects of the per-candidate reset are under System Flows → Consequences.
 
 ### Goals
 - One allocation algorithm, used by seqjob and parjob in normal and simulation mode (1.1, 6.3).
@@ -18,12 +18,13 @@
 - Merging `sched_make_seqjob_allocation()` / `sched_make_parjob_allocation()` dispatch.
 - Adding an allocation duration (the `FIXME: no duration now` stays).
 - Fixing bug-1892 (`unsatisfy_()` reads uncommitted amounts by position).
+- Publishing pend reasons only when they change (follow-up to the larger per-host reason sets of 3.7).
 
 ## Boundary Commitments
 
 ### This Spec Owns
 - `xce_sched_allocator_iface_t` (shape, ownership, lifetime) and the `allocator_iface` field of `xce_sched_plugin_t`.
-- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-bucket `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
+- `sched_allocator_run()` and its internal steps: input/acceptance guards, shared-resource precheck (incl. the bug-1893 loop-bound fix), claim search and rollback, per-candidate `alloc_req.host` reset and `nslots`/`nhosts` assignment, result assembly, cleanup of every attempt-owned resource.
 - The `xce_alloc_plan_iter_t.init` parameter type.
 - The seqjob/parjob plugin wiring (templates, iface instances, `parjob_accepts`) and the removal of their private allocation code.
 - Equivalence and runner tests; the pend-reason factory audit's file list; `issues/new_requirements.md` item 4.
@@ -145,7 +146,7 @@ flowchart TD
     Finalize -->|ok| Done[publish ctx alloc then fini and return 1]
 ```
 
-**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan; set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.
+**Try plan** (per bucket, in order): reset `alloc_req.host` to empty → new host iterator → for each candidate: skip if already claimed in this plan (and reset `alloc_req.host` again after every rejected candidate, before the next `next()`); set `nslots = bucket_slots[b]`, `nhosts = 1`; `sched_try_alloc_on_host_() > 0` → snapshot `alloc_req.host` into the claim (snapshot OOM: uncommit the candidate with the live request, roll back earlier claims, return the error) → next bucket. A bucket with no host rolls back every claim of the plan and returns 0.
 
 **Key decisions**: the reset and the `nslots`/`nhosts` assignment are unconditional (brief items 1 and 3); every return after `plan_init` succeeded calls `plan_fini` and frees the claims buffer; claim snapshots are freed by whichever function owns them at that point (see AllocRunner → State Management).
 
@@ -168,8 +169,8 @@ flowchart TD
 | 3.3 | Roll back claims on unsatisfied bucket | ClaimSearch | `sched_ctx_uncommit_host` | TryPlan |
 | 3.4 | Init failure / no plan = pending | AllocRunner | `plan_init` | Pend1, Fini0 |
 | 3.5 | `nslots` = bucket slots, `nhosts` = 1 | ClaimSearch | — | TryPlan |
-| 3.6 | Empty per-host request per bucket | ClaimSearch | — | TryPlan |
-| 3.7 | seqjob per-host reasons complete | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
+| 3.6 | Empty per-host request before each candidate | ClaimSearch | — | TryPlan |
+| 3.7 | Per-host reasons complete, both allocators | ClaimSearch (via 3.6) | `sched_try_alloc_on_host_` | TryPlan |
 | 3.8 | No fixed bucket limit | AllocRunner (claims buffer) | — | Grow |
 | 4.1 | One host alloc per claim, claim order, snapshot request | ResultAssembly | `sched_make_allocs`, `make_host_alloc` | Finalize |
 | 4.2 | Shared record per shared-request entry | ResultAssembly | `sched_make_allocs` | Finalize |
@@ -318,7 +319,7 @@ static int  sched_snapshot_host_req_(xvec_idval_t* dst, const xvec_idval_t* src)
 ```
 - **Preconditions**: `claims` has capacity `>= plan->nbuckets`.
 - **Postconditions**: on 0 or `< 0`, every per-host debit made in this call is released (3.3, 5.5) and no snapshot remains.
-- Behaviour is parjob's current `parjob_try_plan_()` exactly, with `plan->ctx = ctx`, the per-bucket reset (3.6) and the per-candidate `nslots`/`nhosts` assignment (3.5) now applied to every plugin. seqjob's complete per-host reasons (3.7) follow from 3.6: the pre-filter no longer skips hosts on stale state, so `sched_try_alloc_on_host_()` records its reasons for every selection-matching host.
+- Behaviour is parjob's current `parjob_try_plan_()` with `plan->ctx = ctx` and the per-candidate `nslots`/`nhosts` assignment (3.5), applied to every plugin, and one change: `alloc_req.host` is cleared at the start of each bucket AND after every rejected candidate, i.e. before every `host_iter->next()` call (3.6). Clearing only once per bucket (parjob today) is not enough: a rejected candidate leaves its own per-host request in `alloc_req.host`, and the host iterator's pre-filter (`host_iter.c`) would then skip later short hosts silently. With the request always empty when `next()` pre-filters, the pre-filter does select-expression filtering only, every selection-matching unclaimed host reaches `sched_try_alloc_on_host_()`, and each short one gets its reasons (3.7).
 
 #### ResultAssembly
 
@@ -367,10 +368,11 @@ static const xce_sched_allocator_iface_t seqjob_alloc_iface = {
 ```
 - `prepare_allocation_hook`, `init_fn`, `fini_fn`, `*_new_plugin()`, `register_*()` unchanged; sim registration in `sim_alloc.c` therefore reaches the runner unchanged (6.3).
 
-### Consequences of the per-bucket reset for seqjob (3.6)
-Because seqjob's pre-filter no longer skips hosts on stale state, a pending seqjob attempt now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching host that is short of resources, as parjob already does. Besides the per-host pend reasons (3.7), this means:
+### Consequences of the per-candidate reset (3.6, both allocators)
+Because the host iterator's pre-filter no longer skips hosts on stale state, a pending attempt of either allocator now runs `sched_try_alloc_on_host_()`, and so the full `plugin_prepare_allocation()` fan-out, on every selection-matching unclaimed host that is short of resources. Today that happens only for the first rejected candidate of each bucket search (parjob) or for candidates that pass the stale filter (seqjob). Besides the complete per-host pend reasons (3.7), this means:
 - **Limit plugin side effects** (`limit.c` `match_job_host_limit_`): for a host with a matching host-based limit it allocates the job's `obj->hlimits[worker]` list, may instantiate the limit's virtual resource in `resmgr` (`instantiate_limit`) and writes the limit value into `worker->resvals`. This is decision-neutral: any later check of that host by any job runs the same lazy instantiation before reading the value, so it only moves when the instance is created. It happens identically in normal and simulation mode (driver state-equivalence holds).
-- **Cost**: one prepare fan-out plus satisfy check per short host per pending seqjob attempt, where the stale pre-filter used to reject some hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
+- **Cost**: one prepare fan-out plus satisfy check per short host per bucket search, where the stale pre-filter used to reject later hosts with a single satisfy check. Repeated attempts for the same unsatisfiable request are still short-circuited by the unsatisfiable cache before the allocator runs (`scheduler.c:248`).
+- **Pend-reason volume**: a pending job can now carry one reason per short host (per bucket search for parjob). These reasons are published to the controller and persisted with the job. Publishing reasons only when they change is a recorded follow-up (Non-Goals), not part of this spec.
 
 ## Data Models
 No persisted or wire data changes. In-memory only: `sched_alloc_claim_t` (attempt-scoped) and the `allocator_iface` pointer (static). Snapshot/WAL/recording field-completeness is not affected (scheduler audit dimension 4: no change).
@@ -406,7 +408,7 @@ TDD order follows the staged migration (research.md): characterization tests are
 - `test_alloc_runner.c` (white-box `#include` of `core.c`, `resource.c`, `schedobj.c`, `host_iter.c`, `alloc_runner.c`; stubbed `sched_make_allocs` with OOM injection; stubbed `plugin_prepare_allocation()` that records `ctx->nslots`/`ctx->nhosts`/`ctx->alloc_req.host.count` on each call and stages a fixed per-host request, the same link-stub convention `test_resource_accounting.c` uses; an in-test fake iface whose plans are scripted):
   - guards: each missing input → `-EINVAL`, `ctx->reason` still NULL (5.1); rejecting `accepts` → 0, no reason, `plan_init` not called (1.4); NULL `accepts` reaches `plan_init` (1.3).
   - shared: two shared resources, one short → exactly one host-less `NO_RESOURCE` reason naming it (2.2, bug-1893); met → uncommitted added once across a 3-plan attempt (2.3).
-  - search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty when each bucket's host scan starts, observed through a candidate whose stale request would have been rejected by the pre-filter (3.6); a plan with more buckets than a first small plan triggers buffer growth (3.8).
+  - search: plan order and first-satisfied stop (3.1); distinct hosts per bucket (3.2); failing bucket 2 rolls back bucket 1's debit (`w` uncommitted back to 0) (3.3); `nslots/nhosts` seen by a recording prepare hook equal `bucket_slots[b]`/1 (3.5); `alloc_req.host` empty before every candidate's pre-filter: with three short hosts in one bucket, all three get host/resource reasons (today only the first would) (3.6, 3.7); a plan with more buckets than a first small plan triggers buffer growth (3.8).
   - errors: snapshot OOM, host-alloc OOM at claim 0 and claim k, job-alloc OOM, shared-records OOM → `-ENOMEM`, `ctx->alloc == NULL`, released debits as in the Error table, ASAN clean (5.4, 5.5, 5.6, 5.7).
 - `test_parjob_combinatorial.c`: plan sequences identical for `NULL` and poisoned core (6.2).