myxce/.kiro/specs/scheduler-allocation/requirements.md

Requirements Document

Introduction

The XCE scheduler has a plugin architecture (plugin.h/plugin.c, xce_sched_plugin_t) but three gaps keep it from being a real framework. The first gap -- job selection policy is hardcoded (sched_job_picker() sets job_pick_plugin = "xce_sched_fcfs" as a literal) -- is addressed in the sibling scheduler-core spec.

This spec closes the remaining two allocation gaps:

1. Parallel job allocation is a stub: parjob_allocator.c's my_alloc() takes (plugin, ctx) and immediately return 0 -- any job with maxTasks > 1 (routed there by sched_make_primary_alloc_() at scheduler.c:93-103) produces no allocation at all.

2. Allocation is eager and one-shot: the sibling seqjob_allocator.c's my_alloc() (used when maxTasks == 1) calls sched_collect_workers() then xlist_find() for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.

This spec introduces an iterator-driven allocation architecture with these components:

• A lazy xce_alloc_plan_iter_t -- candidate plans (each a partition of the job's task count into per-host slot buckets, e.g. [2,1] for 3 tasks across 2 hosts), most-preferred first
• A lazy xce_host_iter_t -- candidate hosts for a given plan, filtered by the existing select-expression evaluator
• A parjob_allocator that walks the plan iterator to do real combinatorial multi-slot allocation
• A session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session
• A seqjob_allocator rewrap onto the iterator interfaces as a trivial 1-plan/1-slot case, validating the abstraction before applying it to the complex parjob case

Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in .kiro/specs/scheduler-allocation/brief.md.

Boundary Context

• In scope: xce_alloc_plan_iter_t and xce_host_iter_t interfaces (lazy, cancel-safe, composable); seqjob_allocator rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, maxTasks == 1 only); parjob_allocator's my_alloc() implementing real combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) for maxTasks > 1; xce_alloc_mode_t on xce_sched_ctx_t for allocation mode (NORMAL/PREEMPT); session-level unsatisfiable-resreq fingerprint cache keyed by xce_cache_key_t (O(1) content hash from parsed resreq + identity); unit tests for plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence; integration test for end-to-end multi-slot parallel job allocation.

• Out of scope: Pluggable job picker interface (xce_job_picker_iface_t) -- in scheduler-core; builtin_fcfs picker -- in scheduler-core; configurable picker plugin with per-queue override -- in scheduler-core; preemption (Req 6 of scheduler-framework) -- in scheduler-preemption; core/driver separation (Req 10 of scheduler-framework) -- in scheduler-core; Queue vs Project roles -- in scheduler-core; new job selection policies beyond the builtin FCFS picker; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins).

• Adjacent expectations: depends on the existing plugin framework (plugin.h/plugin.c), the existing select-expression evaluator (resreq.h, memoized via select_eval_ctx_t), and existing shared-resource tracking (xce_scheduler_t.sharedInst/sharedInUse) -- none of these change. Depends on scheduler-core for the core event interface that pushes worker and resource state; the iterators are independently unit-testable with constructed state. ha-consistency-invariants' commit barrier is a future consumer of scheduling decisions; this spec does not wire it.

Allocation Architecture (Reqs 2-4)

Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER):

Resreq Type Hierarchy

ALTER     ->  OR-of-compounds: first compound that fully allocates wins
  └─ COMPOUND ->  AND-of-primaries: ALL primaries must allocate for success
       └─ PRIMARY  ->  plan_iter -> host_iter -> try_alloc  (leaf level)

The iterator composition (xce_alloc_plan_iter_t + xce_host_iter_t) operates at the PRIMARY leaf. COMPOUND and ALTER are composition rules above the iterators -- they call into the same primary allocator, unchanged from today's sched_make_compound_allocation() and sched_make_alter_allocation() dispatch.

Iterator Composition (PRIMARY leaf)

                    ┌──────────────────────────┐
                    │  xce_alloc_plan_iter_t    │  Req 2
                    │                            │
                    │  init(primary_resreq, sched)  │
                    │  next() -> plan            │  generates per-host bucket
                    │  fini()                    │  partitions, most-preferred first
                    └────────────┬───────────────┘
                                 │ plan
                                 ▼
                    ┌──────────────────────────┐
                    │  xce_host_iter_t          │  Req 3
                    │                            │
                    │  init(plan, workers)       │  for a given plan, iterate
                    │  next() -> worker           │  candidate hosts matching
                    │  fini()                    │  select expression + availability
                    └────────────┬───────────────┘
                                 │ worker
                                 ▼
                    ┌──────────────────────────┐
                    │  try_alloc(worker, plan)  │  Req 4 (parjob) or
                    │                            │  existing seqjob logic
                    │  check slots, shared res,  │
                    │  commit or advance         │
                    └────────────────────────────┘

Composition Algorithm (All Three Types)

A plan is a partition of the primary's task count into per-host buckets (e.g. for maxTasks=3: [3], [2,1], [1,1,1]) -- not a single (nhosts, slots_per_host) pair, which cannot express uneven splits. allocate_primary() must therefore satisfy every bucket in a plan against a distinct host before the plan counts as satisfied; satisfying only the first bucket and stopping (as a single-pass try_alloc_on_host loop would) is not a valid implementation of Requirement 4 -- it degenerates into the single-host plan and never actually exercises the multi-host case.

Host exclusivity within a plan is enforced by the caller, not xce_host_iter_t itself: allocate_primary() skips any host already present in claimed before calling try_alloc_on_host -- Requirement 3's init(plan, workers) signature is unchanged, host_iter stays a pure select+availability filter. This distinction matters: a host offered and rejected for one bucket (too small for that bucket's slot count) must remain eligible for a later, smaller bucket in the same plan -- only a host actually claimed by a bucket is permanently excluded for the rest of that plan attempt.

Correction (this review): a fresh host_iter per bucket, not one shared across the whole plan. An earlier version of this pseudocode created one host_iter before the bucket loop and let every bucket's while pull from the same cursor. Since next() is a forward-only cursor with no rewind, that silently breaks the "offered-and-rejected stays eligible for a later bucket" guarantee just stated: a host bucket 0 tries and rejects (too small for it) is gone from the cursor forever, even if it exactly satisfies bucket 1. Concretely, for plan [2,1] with host A (1 free slot) and host B (2 free slots): bucket 0 ("need 2") pulls A first, rejects it, pulls B, succeeds -- cursor now exhausted; bucket 1 ("need 1") gets nothing back even though A was sitting there the whole time. The fix: host_iter_init(iter, plan, sched->workers) is called fresh at the top of each bucket iteration (same plan, same host_filter every time -- only is_claimed()'s growing exclusion set differs call to call), and fini()'d at the end of that same bucket's search. This is a three-level nested loop (plan → bucket/partition → worker, the third level always starting fresh), not a two-level one with an inner index.

Bucket order is not merely a fail-fast heuristic -- for v1's single-resource-dimension model, descending (largest bucket first) can succeed where ascending would fail, with the identical set of available hosts. Same example: with buckets tried ascending [1,2] instead, a fresh scan for "need 1" could greedily claim host B (2 free slots -- trivially satisfies 1, but wastefully overqualified), leaving only A (1 free slot) for "need 2", which fails. Descending order avoids this by construction: the largest ask is checked against the full pool before a smaller ask has a chance to consume a host only the larger one could have used. This is why the v1 default (descending, "Data Structure Changes Summary" below) is a real correctness property here, not just an optimization -- and why generating the reverse order as a second candidate plan for the same partition isn't needed on top of it. Which specific host gets picked among multiple hosts that satisfy the same bucket (host_iter's own internal candidate order, HOST_ORDER_ANY/HOST_ORDER_MOST_FREE etc.) is a separate, softer question already deferred as future optimization -- it affects efficiency under adversarial cases, not whether a solvable case gets missed outright the way the shared-cursor bug did.

Rollback scope. ctx->uncomitted (sched_ctx_init/sched_ctx_fini, schedobj.c) is per-job-allocation-attempt scratch state, torn down at the end of one job's sched_make_primary_allocation()/sched_make_compound_allocation() call -- it never touches the real, session-wide resInUse books, which are only charged later, separately, once a job actually dispatches (job->accounts/sched_account_jobs()). So rollback_plan_allocation() can never corrupt cluster-wide resource state. It is still required, though: without it, a bucket that succeeded before a later bucket exhausted the host iterator leaves its provisional debit in ctx->uncomitted.hosts[idx] for the rest of that job's allocation attempt -- a later plan considering the same host would see it as falsely less available, purely from a phantom debit an earlier, abandoned plan left behind, which could wrongly pend a job a later plan could actually have satisfied. The reversal itself has no existing precedent to reuse (sched_make_compound_allocation()'s own on-failure path, scheduler.c:206-238, does not actually clean up partial state on a mid-loop failure -- a separate, pre-existing gap outside this spec's scope, tracked separately, not modeled here). The underlying subtract primitive already exists (xvec_op(v1, v2, XVEC_SUB_FLOAT), xvec.c); it just needs a small new "uncommit" helper wired to it, mirroring the existing XVEC_ADD_FLOAT commit call in sched_try_alloc_on_host_(). For this to work, each claimed[] entry must retain the exact alloc_req.host snapshot captured when that bucket succeeded (needed for the merge step too -- see "Final allocation result" below), since ctx->alloc_req is reset and repopulated for every subsequent host tried.

Allocation convention (this review): iterators are stack-allocated by the caller, then initialized in place -- matching sched_ctx_init(&ctx, ...)'s existing convention in this codebase, NOT a heap-allocating factory. init()'s own declared signature (int (*init)(xce_alloc_plan_iter_t* iter, ...), "Data Structure Changes Summary" below) takes the iterator's own address as its first argument for exactly this reason. An earlier version of this pseudocode wrote plan_iter = plan_iter_init(primary_resreq, sched) as if it were a 2-arg factory function returning a new iterator -- inconsistent with the 3-arg in-place-init method the struct actually declares. Corrected below.

// === PRIMARY leaf: iterator composition ===
// Returns >0 on success, 0 on failure
int allocate_primary(sched, primary_resreq, ctx) {
    xce_alloc_plan_iter_t plan_iter_storage;
    xce_alloc_plan_iter_t* plan_iter = &plan_iter_storage;
    if (plan_iter->init(plan_iter, primary_resreq, sched) != 0) {
        return ALLOC_FAIL;  // exact error-propagation contract: design.md
    }

    while ((plan = plan_iter->next(plan_iter)) != NULL) {
        // Correction (kiro-validate-design pass): sized by maxTasks, not an
        // undefined fixed constant -- nbuckets can never exceed maxTasks
        // (each bucket needs >=1 slot), so this is the true, tight upper
        // bound. A fixed compile-time guess risks either overflow for a
        // large parallel job or wasted stack for a small one.
        xce_sched_worker_t* claimed[primary_resreq->maxTasks];
        int nclaimed = 0;
        int plan_ok = 1;

        // Three levels of iteration: plan (outer while, above) -> bucket/
        // partition (this for) -> worker (host_iter, fresh per bucket, below).
        for (int b = 0; b < plan->nbuckets; b++) {
            // Correction (this review): a FRESH host_iter per bucket, not
            // one shared across the whole plan -- next() is a forward-only
            // cursor, so sharing it would drop any host an earlier, larger
            // bucket tried and rejected, even if it exactly satisfies THIS,
            // smaller bucket. Same plan (same select-expression host_filter)
            // every call; only is_claimed()'s growing exclusion set differs.
            xce_host_iter_t host_iter_storage;
            xce_host_iter_t* host_iter = &host_iter_storage;
            xce_sched_worker_t* w;
            int bucket_ok = 0;

            if (host_iter->init(host_iter, plan, sched->workers) == 0) {
                while ((w = host_iter->next(host_iter)) != NULL) {
                    if (is_claimed(claimed, nclaimed, w)) {
                        continue;
                    }
                    if (try_alloc_on_host(w, plan->bucket_slots[b], ctx) > 0) {
                        // snapshot ctx->alloc_req.host here (it is reset on
                        // the next plugin_prepare_allocation() cycle) --
                        // needed for both rollback below and the merge step
                        // in commit_plan_allocation().
                        claimed[nclaimed++] = capture_claim(w, ctx);
                        bucket_ok = 1;
                        break;
                    }
                }
                host_iter->fini(host_iter);
            }
            // host_iter->init() failing is treated the same as "no
            // candidates" for this bucket -- an infrastructure error, not a
            // resource-availability determination; see design.md for the
            // exact error-propagation contract.

            if (!bucket_ok) {
                plan_ok = 0;
                break;  // this bucket has no satisfying host -- abandon the plan
            }
        }

        if (plan_ok) {
            commit_plan_allocation(claimed, nclaimed, ctx);
            plan_iter->fini(plan_iter);
            return ALLOC_SUCCESS;
        }

        // Partial success within this plan attempt must not leak into the
        // next plan attempt for the SAME job (ctx->uncomitted persists across
        // plan attempts until sched_ctx_fini() -- see "Rollback scope" above):
        // for each claimed host, subtract back exactly the debit
        // sched_try_alloc_on_host_() added via XVEC_ADD_FLOAT, using the
        // snapshot captured above (new xvec_op(..., XVEC_SUB_FLOAT)-based
        // helper -- no existing "uncommit" primitive to call into).
        rollback_plan_allocation(claimed, nclaimed, ctx);
    }
    plan_iter->fini(plan_iter);
    return ALLOC_FAIL;
}

// === COMPOUND: AND of primaries ===
// ALL primaries must allocate. On any failure, roll back prior allocations
// and return failure. NOTE: today's sched_make_compound_allocation() does
// NOT actually do this rollback (see bug-1766, found during this spec's
// review) -- rollback_compound_allocations() below is new/corrective
// behavior this spec should introduce, not a port of existing logic. Out of
// this spec's own boundary per Req 9 AC2 (COMPOUND/ALTER dispatch
// unchanged) unless the user elects to fold bug-1766's fix in here.
int allocate_compound(sched, compound_resreq, ctx) {
    for (int i = 0; i < compound_resreq->primaries.count; i++) {
        primary = compound_resreq->primaries[i];
        if (allocate_primary(sched, primary, ctx) <= 0) {
            // Roll back all primaries allocated so far in this compound
            rollback_compound_allocations(ctx, i);
            return ALLOC_FAIL;
        }
    }
    return ALLOC_SUCCESS;
}

// === ALTER: OR of compounds ===
// First compound that fully allocates wins. Same logic as today's
// sched_make_alter_allocation(). On failure, no rollback needed
// (prior compounds were never committed -- they failed their own checks).
int allocate_alter(sched, alter_resreq, ctx) {
    for (int i = 0; i < alter_resreq->compounds.count; i++) {
        compound = alter_resreq->compounds[i];
        if (allocate_compound(sched, compound, ctx) > 0) {
            return ALLOC_SUCCESS;
        }
        // This compound failed -- advance to next alternative
    }
    return ALLOC_FAIL;
}

// === Top-level dispatch ===
// Unchanged from today's sched_make_allocation() dispatch on obj->tag.
int allocate_job(sched, job, ctx) {
    switch (job->resreq_type) {
    case PRIMARY:
        return allocate_primary(sched, job->primary_resreq, ctx);
    case COMPOUND:
        return allocate_compound(sched, job->compound_resreq, ctx);
    case ALTER:
        return allocate_alter(sched, job->alter_resreq, ctx);
    }
}

COMPOUND rollback note: If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 should be rolled back -- the compound is meant to be atomic. Correction (this review): today's sched_make_compound_allocation() does not actually do this -- see bug-1766. It shares one ctx across the whole primaries loop, breaks on the first failing primary without undoing earlier successes, and its unconditional post-loop if (ctx.alloc != NULL) MOVE_PTR(obj->alloc, ctx.alloc) can attach a partial, orphaned allocation to a compound job reported as failed. This spec's iterator rework does not touch that path (Req 9 AC2 keeps COMPOUND/ALTER dispatch unchanged) -- the pre-existing gap is tracked separately in bug-1766, not fixed here, unless scope is deliberately expanded to include it.

Who Uses This Composition

Allocator	maxTasks	Plan iterator	Host iterator	Behavior
seqjob_allocator	== 1	Trivial: single plan (1 host x 1 slot)	Full: filters hosts by select expression	Unchanged from today -- first matching host wins
parjob_allocator	> 1	Full combinatorial: 1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1	Full: filters hosts with enough available slots	New -- walks plans until one succeeds
Future allocators	any	Custom strategy (e.g., topology-aware, bin-packing)	Reuses same xce_host_iter_t	Pluggable via job_alloc_fn

Key Design Properties

• Lazy: plan_iter->next() generates one plan at a time. host_iter->next() returns one host at a time. No full materialization.
• Cancel-safe: both next() calls have pthread_cancel cancellation points (Req 2 AC 6 / Req 3 AC 6 -- a testable requirement, not just a design intent, since neither loop has a natural cancellation point of its own).
• Independent lifecycles: a host_iter is created from a plan, used, and destroyed -- and, for a multi-bucket plan, this happens once per bucket, not once per plan (a fresh instance each time, per Req 3 AC 3/4). The plan_iter is advanced to the next plan only after every bucket's host_iter for the current plan is done. fini() on one does not affect the other.
• COMPOUND/ALTER unchanged: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside allocate_primary().
• Seqjob rewrap: seqjob_allocator is rewrapped onto the iterator interfaces as a trivial case -- its plan iterator always returns the single plan (1 host x 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case.
• Pluggable plan generation, shared host iteration: plan generation is allocator-specific — each job_alloc_fn brings its own strategy (trivial, combinatorial, topology-aware, bin-packing) and creates its own xce_alloc_plan_iter_t internally. Host iteration is shared infrastructure — xce_host_iter_t has one implementation reused by all allocators, parameterized by sort order (HOST_ORDER_ANY, HOST_ORDER_MOST_FREE, etc.). Swapping allocation strategy = swapping the plugin; no core loop change.

Requirements

Requirement 2: Lazy allocation plan iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front.

Acceptance Criteria

1. The Scheduler shall define an xce_alloc_plan_iter_t interface with init(resreq, sched), next() -> plan, and fini() operations. The init() signature passes the full xce_primary_resreq_t (including span, same, affinity) and scheduler state — sufficient for any allocator to derive its plan space without future interface changes.

   Plan-shaping fields (on xce_primary_resreq_t, consumed by plan iterator internally):

   • span: bounds host count (nhosts_min/nhosts_max) and per-host task cap (max_per_host, analogous to LSF ptile). v1 combinatorial generation ignores these (empty structs); fill when ptile/nhosts support is needed.
   • same: co-location constraints ("all tasks on same rack"). v1 ignores; future.
   • affinity: restrict to named host group. v1 ignores; future.
   • minTasks/maxTasks: task count bounds — the primary driver of plan space. Used in v1.
   • select: host filter expression — consumed by host iterator (Req 3), not plan iterator.
   • rusages: resource requests — consumed by try_alloc_on_host, not plan iterator.

   v1 implements simple combinatorial generation: 1host×N, 2hosts×ceil(N/2), ..., Nhosts×1. Future allocators read span/same/affinity to constrain the plan space without touching the init() signature.

2. Each call to next() shall generate exactly one candidate plan -- a partition of maxTasks into per-host buckets (xce_alloc_plan_t.bucket_slots[], see "Composition Algorithm" above; NOT a uniform (nhosts, slots_per_host) pair, which cannot express uneven splits) -- without generating or materializing any other plan in the same call.

3. The Scheduler shall order plans returned by successive next() calls from most-preferred to least-preferred, per the owning allocator's preference policy.

4. next() shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion.

5. Calling fini() shall release all resources held by the iterator, and the interface shall define no other required cleanup step.

6. next() shall call pthread_testcancel() (or otherwise provide an equivalent, real cancellation point) at least once per invocation, matching the existing convention sched_job_picker()/sched_allocator() already establish at the top of their own loops (scheduler.c). This is required because, unlike those threads' blocking cbuf_get() calls (a POSIX cancellation point via pthread_cond_wait), combinatorial plan generation is fast, non-blocking pure computation with no cancellation point of its own -- an implementation satisfying AC 1-5 without this would silently violate the "Cancel-safe" design property stated for this iterator without failing any other criterion.

Requirement 3: Lazy host candidate iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction for finding a host satisfying one plan bucket at a time, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering. (Correction, this review: NOT "one xce_host_iter_t per plan" -- a plan with multiple buckets requires a fresh instance per bucket, per Req 4 AC 2.)

Acceptance Criteria

1. The Scheduler shall define an xce_host_iter_t interface with init(plan, workers), next() -> worker, and fini() operations.

2. Each call to next() shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.

3. next() shall return NULL once no further candidate host exists for the current bucket's search, and the caller shall not need to call next() again on that same host_iter instance after receiving NULL. This does NOT mean the caller is done with the plan: a plan with more buckets remaining requires a fresh host_iter instance (a new init() call) for the next bucket, per Req 4 AC 2 -- exhaustion is scoped to one host_iter instance, not to the plan as a whole.

4. The host iterator shall compose with the plan iterator: a new xce_host_iter_t shall be constructible from any plan produced by Requirement 2's xce_alloc_plan_iter_t.next() -- and, for a multi-bucket plan, constructible repeatedly from the same plan, once per bucket (Req 4 AC 2), each a fully independent instance.

5. Calling fini() shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.

6. next() shall call pthread_testcancel() (or otherwise provide an equivalent, real cancellation point) at least once per invocation, for the same reason as Requirement 2 AC 6: scanning/filtering candidate hosts is non-blocking pure computation with no cancellation point of its own, so this must be explicit rather than assumed.

Requirement 4: Parallel job combinatorial allocation

Objective: As a cluster operator submitting a job that needs multiple task slots, I want parjob_allocator to actually allocate across hosts, so that jobs with maxTasks > 1 are no longer silently unallocatable.

Acceptance Criteria

1. When a job's xce_primary_resreq_t.maxTasks > 1 reaches parjob_allocator's job_alloc_fn (my_alloc() in parjob_allocator.c), the Scheduler shall generate candidate plans covering at minimum the host-count sequence 1..N hosts, where each plan's bucket sizes are a partition of maxTasks into that many buckets (e.g. for N=3: [3], [2,1], [1,1,1] -- balanced ceil/floor split, not a uniform slots_per_host that cannot express uneven splits), using Requirement 2's plan iterator.

2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to find, for every bucket in the plan, a distinct host with enough available slots to satisfy that bucket -- accumulating claimed hosts one bucket at a time -- and shall attempt allocation before advancing to the next plan. A plan is satisfied only when every one of its buckets has a claimed host; if any bucket exhausts the host iterator without a match, the plan fails as a whole and any buckets already matched within that same plan attempt are released before advancing to the next plan (see Composition Algorithm's rollback_plan_allocation).

   A new xce_host_iter_t instance shall be constructed for each bucket (three levels of iteration: plan -> bucket/partition -> worker, the third always starting fresh) -- never one shared, continuously-advancing instance reused across every bucket of a plan. Reusing one instance is an incorrect implementation: next()'s forward-only cursor would permanently drop any host an earlier, larger bucket tried and rejected as too small, even when that same host exactly satisfies a later, smaller bucket -- silently failing plans a correct implementation would have satisfied. Only is_claimed() (Criterion 2's own "distinct host" exclusion, applied by the caller across each bucket's fresh scan) may exclude a host across buckets; rejection by an earlier bucket's size check must not.

3. When every bucket of a plan is satisfied, the Scheduler shall commit that allocation as a single xce_job_alloc_t merging one xce_host_alloc_t per claimed bucket/host (nhosts == the plan's bucket count; see "Final allocation result" under parjob_allocator changes, below) and shall not evaluate any remaining plan for that job.

4. When no plan among those generated in Criterion 1 can be satisfied, my_alloc() shall return the same not-satisfied signal (0) that the current stub returns today, so pend-reason handling in sched_make_primary_alloc_() is unaffected.

5. parjob_allocator's job_alloc_fn shall no longer unconditionally return 0 regardless of ctx contents -- the current stub behavior -- for any job with maxTasks > 1.

6. Before each candidate host is checked against a plan's bucket, the Scheduler shall set ctx->nslots (and ctx->nhosts, for the shared/global-resource pass) to that bucket's slot count so that parjob_prepare_allocation() computes the per-host resource want via the existing calc_wanted_host_resvals(..., ctx->nhosts, ctx->nslots) call -- the same mechanism seqjob_prepare_allocation() already uses, currently hardcoded to 1, 1. Concurrently, seqjob_prepare_allocation() shall gain a primary->maxTasks == 1 guard (it currently has none, unlike parjob_prepare_allocation(), which already gates on maxTasks). This guard is required, not optional cleanup: plugin_prepare_allocation() (plugin.c:591) unconditionally calls every registered plugin's prepare_allocation_hook, and xvec_idval_store() (xvec.c:247) appends (idx, val) pairs with no dedup by resource idx -- so once parjob_prepare_allocation() stores real values, an unguarded seqjob_prepare_allocation() would also fire for the same maxTasks > 1 job and append a second, wrong (nslots=1) entry for the same resource idx into ctx->alloc_req, corrupting the availability check that Criterion 2 depends on. This is a latent defect in the existing fan-out, invisible today only because parjob_prepare_allocation() is currently a no-op.

7. Shared-resource satisfaction (sched_check_shared_satisfy(), ctx->alloc_req.shared) shall be computed and checked once per primary resreq -- independent of which plan or bucket-shape is currently being tried -- exactly as it already is today for seqjob_allocator. Plan/bucket iteration (Criteria 1-3) governs only per-host slot allocation; it does not change when or how often the job-level shared-resource check runs.

Requirement 7: Session-level unsatisfiable resreq cache

Objective: As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.

Acceptance Criteria

1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on xce_scheduler_t, keyed by xce_cache_key_t — a value type composing xce_resreq_fingerprint_t (pre-computed {minTasks, maxTasks, select_node_id, rusage_hash}) and identity ({user, queue, project} as xstring pointers). The resreq fingerprint is stored on the scheduling object (xce_primary_schedobj_t) and computed once on job ingest. The full cache key is constructed on the stack at lookup time and looked up via a single key_find() call against a kvtab_t bound to a new cache_key_ops (kv_key_ops_t) — this codebase's real hash-table API binds hash/cmp/assign/destroy to the table at creation, not per call (see "New types" below; no free-function kvtab_lookup() taking hash/eq params exists). Lookup itself is O(1): no list walks, no string traversals; key_find() does not allocate (only key_add(), on insert, allocates the heap-owned key copy). This captures everything the allocator and limit plugin consume.

2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (begin_session), by placing the clear inside core_begin_session_internal() (core_session.c) alongside the existing, unlocked core_reset_shared_resval()/core_reset_workers() calls -- a core-level function both driver.c (normal mode) and sim_loop.c/sim.c (simulation mode) reach identically through xce_core_begin_session(), so no driver-specific clearing code is needed for either mode. No explicit lock is required: this matches the existing, already-safe pattern those two sibling resets use, which relies on xce_core_begin_session()'s own documented ordering guarantee -- the driver completes all begin-session reset work strictly before xce_core_signal_go() unblocks the picker thread (the one cbuf_put() that lets the picker/allocator touch anything for the new session), so the mutex/condvar inside that cbuf_put()/cbuf_get() pair provides the memory-barrier the reset needs, the same way it already does for worker resvals/sharedInUse. Neither driver touches the cache object anywhere else -- lookup (AC 4) and insert (AC 3) happen exclusively on the allocator thread, inside allocate_primary().

   Not part of the snapshot/replay wire format. The cache is pure in-memory, session-scoped state -- sched_snapshot_data_t (snapshot.c/typemap.h) does not capture it, and none should be added. Since AC 2 unconditionally clears it at the next begin_session regardless of what a loaded snapshot contained, any captured cache state would be discarded before a single lookup against it could occur -- persisting it would be dead weight with no observable effect.

3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. In v1, cache population is preemption-agnostic: it fires on allocation failure alone. When the scheduler-preemption spec ships, preemption fires between allocation failure and cache population — if preemption succeeds, no cache entry is created; if preemption fails or is inapplicable, the cache entry is populated as normal. The allocation layer does not call into preemption; the caller (scheduling loop) orchestrates the sequence.

4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption.

5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.

Note on resreq_fp and snapshot restore (informational, not an AC): resreq_fp is computed at job-ingest time (core_build_schedobj()/core_events.c) but not by snapshot_load.c's independent local schedobj reconstruction (scheduler-snapshot-job-allocation spec) -- so a session restored via xce_scheduler -i -s <replay> would run with a permanently-cold cache for this Requirement. Confirmed (xce_sched_core_load_snapshot()'s 12 call sites are all in sim.c; none in driver.c) that this can only ever affect offline simulation/investigation sessions, never live production scheduling throughput -- so this is not elevated to a formal Acceptance Criterion. Tracked in bug-1767 alongside the broader recurring-duplication pattern in that reconstruction path, should someone want to close it later.

Requirement 9: No regression to existing allocators or HA soak

Objective: As a maintainer, I want the iterator-driven rework scoped so that existing single-slot allocation and COMPOUND/ALTER dispatch are unaffected, so that this framework change is safe to ship independently of any new allocator.

Acceptance Criteria

1. Seqjob regression: seqjob_allocator's externally observable allocation behavior (which host a maxTasks == 1 job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the xce_host_iter_t interface.

2. COMPOUND/ALTER unchanged: The COMPOUND and ALTER resreq allocation paths (sched_make_compound_allocation, sched_make_alter_allocation in scheduler.c) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today.

3. Unit tests: Existing scheduler unit tests (scheduler meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the eager-one-shot behavior or stub return value this feature replaces.

4. HA soak: A full HA soak (sequential and chaotic /xce-test-ha regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.

5. Allocation-mode extension point: The Scheduler shall add an xce_alloc_mode_t enum (XCE_ALLOC_NORMAL = 0, XCE_ALLOC_PREEMPT) and an alloc_mode field on xce_sched_ctx_t, defaulting to XCE_ALLOC_NORMAL. This spec's own plan/host iterators and try_alloc_on_host() shall not branch on any value other than XCE_ALLOC_NORMAL -- implementing what XCE_ALLOC_PREEMPT (or any future mode) actually does to resource-availability checks is scheduler-preemption's job, not this spec's. The field exists here, in this spec's own scope, purely so scheduler-preemption and later resource-pool modes (e.g. reservation-aware backfill) can extend xce_sched_ctx_t and the availability-check call sites without a further interface change to types this spec already owns -- avoiding a sibling spec's first task being "go modify scheduler-allocation's struct." Adding further enum values beyond the two declared here is explicitly out of this spec's scope.

Note (informational, not an AC): Requirement 7's resreq_fp field is only ever populated on the live job-ingest path in this spec's v1 -- snapshot_load.c's independent restore path does not compute it (see Requirement 7's note above and bug-1767). Since that path is exclusively simulation/investigation tooling (sim.c, never driver.c), this does not warrant a dedicated /xce-audit-invariants scheduler pass as a formal AC here; ordinary unit-test coverage of Requirement 7's cache hit/miss behavior is sufficient for this spec's scope.

Data Structure Changes Summary

xce_alloc_plan_iter_t (new, Req 2)

typedef struct xce_alloc_plan_iter_s xce_alloc_plan_iter_t;

struct xce_alloc_plan_iter_s {
    // Initialize the iterator for a given primary resreq and scheduler state.
    // Returns 0 on success, nonzero on error.
    int  (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq,
                 xce_scheduler_t* sched);

    // Return the next candidate plan, or NULL when exhausted.
    // The returned plan pointer is valid until the next next() or fini() call.
    xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter);

    // Release all resources held by the iterator.
    void (*fini)(xce_alloc_plan_iter_t* iter);

    // Private state: current position, strategy-specific data
    void* private;
};

// A plan is a partition of maxTasks into per-host buckets, e.g. for
// maxTasks=3: [3], [2,1], [1,1,1] -- NOT a uniform (nhosts, slots_per_host)
// pair, which cannot express uneven splits.
typedef struct {
    int  nbuckets;       // number of hosts this plan needs (== len(bucket_slots))
    int* bucket_slots;   // partition of maxTasks; bucket_slots[i] = slots
                         // needed on the i-th distinct host claimed for this plan
    // host_filter: derived from primary_resreq->select, plus slot availability
} xce_alloc_plan_t;

// Generating which host-counts to try, in what order (plan-shape policy) and
// how to partition maxTasks for a given host-count (partition policy) are
// deliberately left as internal, swappable policy inside the plan iterator's
// reference implementation -- not fixed by this interface. v1 ships ONE
// concrete, fully-specified, deterministic default of each -- "swappable
// later" does not mean "unspecified now":
//
//   Plan-shape order (v1 default): ascending host-count, k = 1, 2, ..., N
//   (N = maxTasks). Most-consolidated (fewest hosts) tried first, most-spread
//   (N hosts x 1 slot) tried last -- the brief's original "1hostxN, 2hostsx
//   ceil(N/2), ..., Nhostsx1" sequence, generalized to real partitions below.
//
//   Partition order (v1 default), for a given host-count k: the canonical
//   BALANCED split -- (N mod k) buckets get ceil(N/k) slots, the remaining
//   (k - N mod k) buckets get floor(N/k) slots. E.g. N=7, k=3 -> [3,2,2].
//   No other partition of N into k parts is generated (bounds plan count to
//   exactly N, not the full integer-partition combinatorial space).
//
//   Bucket order within one plan (v1 default): descending by slot count --
//   largest bucket first, so a plan fails fast on its hardest-to-place chunk
//   rather than discovering that late. A simplest-deterministic tie-break,
//   not itself claimed as an optimization.
//
// This is deliberately "blind": v1's plan-shape and partition choices are a
// pure function of maxTasks alone -- they never inspect sched->workers'
// actual free-slot counts, topology, or load when deciding WHAT to generate
// or in WHAT order (only try_alloc_on_host/Req 3's host_iter ever look at
// live worker state, to decide whether a generated plan succeeds). Accepted
// consequence: a job with a very large maxTasks (e.g. thousands) on a modest
// cluster generates up to N candidate plans/host-counts in v1, most of which
// exceed the live worker count and fail immediately once host_iter finds too
// few candidates -- cheap per attempt, but not free, and not capped by
// anything cluster-state-aware. Real optimization -- e.g. skipping a
// host-count already known infeasible from current worker counts, or
// ordering by observed cluster state -- is explicitly future work, not v1;
// ordering MAY consult the full
// primary_resreq already passed to init() (span/same/affinity) without an
// interface change, matching AC1's existing forward-compatibility note.
// Deferred to a later enhancement, not this spec: a formal pluggable-strategy
// API for swapping either policy independently. v1 hardcodes both inside the
// combinatorial plan iterator; only allocator swap-out (Req 2 AC1's existing
// "future allocators bring their own iterator" path) is required now.

xce_host_iter_t (new, Req 3)

typedef struct xce_host_iter_s xce_host_iter_t;

// Correction (found during task 1.1 implementation): next() returns
// xce_sched_worker_t* (sched/types.h -- the scheduler-internal type
// carrying resvals/resInUse/host_eval_ctx), NOT xce_worker_t* (xce/types.h
// -- the wire/config type make_sched_worker() converts FROM). This
// Composition Algorithm's own pseudocode already uses
// `xce_sched_worker_t* w = host_iter->next(host_iter)` -- this struct
// declaration was simply out of sync with it. sched_check_host_satisfy()/
// sched_meet_host_selection() (resource.c), which any real host_iter
// implementation must call, both take xce_sched_worker_t*, not
// xce_worker_t*.
struct xce_host_iter_s {
    // Initialize the iterator for a given plan and worker set.
    // Returns 0 on success, nonzero on error.
    int  (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan,
                 var workers);

    // Return the next candidate worker matching the plan's host filter,
    // or NULL when exhausted. The returned worker pointer is valid until
    // the next next() or fini() call.
    xce_sched_worker_t* (*next)(xce_host_iter_t* iter);

    // Release all resources held by the iterator.
    void (*fini)(xce_host_iter_t* iter);

    // Private state: current position, filtering state
    void* private;
};

New types (Req 7)

// Resreq portion of the cache key — pre-computed, shareable across jobs
// with identical resreqs. Fits in a single cache line (24 bytes).
typedef struct {
    int      min_tasks;
    int      max_tasks;
    uint32_t select_node_id;   // CSE-interned root; 0 if select is NULL
    uint64_t rusage_hash;      // computed at parse time, stable for life of resreq
} xce_resreq_fingerprint_t;

// Full cache key — resreq fingerprint + identity.
// Identity fields are xstring pointers (not owned, not freed).
// Pointer equality = content equality for xstring-interned strings.
typedef struct {
    xce_resreq_fingerprint_t resreq;
    const char*              user;     // xstring pointer
    const char*              queue;    // xstring pointer
    const char*              project;  // xstring pointer
} xce_cache_key_t;

// Correction (this review): this codebase's real kvtab API (kv.h/kv.c) does
// NOT take a hash/eq pair per lookup call -- a kvtab_t binds ONE fixed
// kv_key_ops_t (.hash/.assign/.cmp/.destroy) at table-CREATION time
// (kv_params_t), and every lookup/insert goes through the generic
// key_find()/key_add()/key_del() using whatever keyops that table was built
// with. int_key_ops/string_key_ops are the only existing kv_key_ops_t
// instances -- a composite struct key needs a THIRD one, new to this spec:
static uint32_t cache_key_hash_(void* k) {
    const xce_cache_key_t* key = (const xce_cache_key_t*)k;
    uint32_t                h  = 2166136261u;  /* FNV-1a 32-bit -- no
                                                  * existing hash-update
                                                  * helper in this codebase;
                                                  * this is new, self-
                                                  * contained code. */
    const unsigned char* p = (const unsigned char*)&key->resreq;
    for (size_t i = 0; i < sizeof(key->resreq); i++) {
        h = (h ^ p[i]) * 16777619u;
    }
    h = (h ^ (uint32_t)(uintptr_t)key->user)    * 16777619u;
    h = (h ^ (uint32_t)(uintptr_t)key->queue)   * 16777619u;
    h = (h ^ (uint32_t)(uintptr_t)key->project) * 16777619u;
    return h;
}

// cmp -- three-way comparator, 0 == equal (confirmed against kv.c's real
// int_key_ops.cmp -> cmp_int_key: returns -1/0/1, and kvtab_get() checks
// `cmp(...) == 0` for a match). Correction (design-phase discovery): an
// earlier version of this function returned a BOOLEAN (1 for match, via
// &&-chained == comparisons) -- exactly inverted from the real contract.
// As originally written, every lookup would have been a false miss,
// silently disabling the entire cache (Req 7) without failing any test
// that didn't specifically assert a hit. memcmp() on the resreq portion
// already returns a 3-way result; the identity pointers just need the
// same shape.
static int cache_key_cmp_(void* a, void* b) {
    const xce_cache_key_t* ka = (const xce_cache_key_t*)a;
    const xce_cache_key_t* kb = (const xce_cache_key_t*)b;
    int rc = memcmp(&ka->resreq, &kb->resreq, sizeof(ka->resreq));
    if (rc != 0) {
        return rc;
    }
    if (ka->user != kb->user) {
        return (ka->user < kb->user) ? -1 : 1;
    }
    if (ka->queue != kb->queue) {
        return (ka->queue < kb->queue) ? -1 : 1;
    }
    if (ka->project != kb->project) {
        return (ka->project < kb->project) ? -1 : 1;
    }
    return 0;
}

// assign -- kvtab keys are heap-owned by the table (see string_key_ops'
// strdup-on-insert precedent); *dest must be a fresh heap copy of the
// stack-constructed key passed at the lookup site, not an alias of it.
static int cache_key_assign_(void** dest, void* src) {
    xce_cache_key_t* copy = (xce_cache_key_t*)malloc(sizeof(xce_cache_key_t));
    if (copy == NULL) {
        return -ENOMEM;
    }
    memcpy(copy, src, sizeof(xce_cache_key_t));
    *dest = copy;
    return 0;
}

// destroy -- frees only the key struct itself; user/queue/project are
// borrowed xstring pointers (not owned, not freed here).
static void cache_key_destroy_(void* k) {
    free(k);
}

static kv_key_ops_t cache_key_ops = {
    .hash    = cache_key_hash_,
    .assign  = cache_key_assign_,
    .cmp     = cache_key_cmp_,
    .destroy = cache_key_destroy_,
};

// Table construction: standard kvtab_type RTTI pattern (see sched->core.plugins/
// sched->core.resmgr for a live, already-working example of this exact
// construction call site in this codebase) with
// kv_params_t{ .nslots = N, .keyops = &cache_key_ops, .datakiller = NULL }
// (NULL datakiller: a cache "value" is a hit/miss sentinel, nothing to free).

xce_primary_resreq_t addition (Req 7)

One new field, set once at parse time:

// In xce_primary_resreq_t (resreq.h):
uint64_t rusage_hash;  // set by parse_resreq_primary(), stable for life of resreq

xce_sched_obj_t / xce_primary_schedobj_t addition (Req 7)

Store the resreq portion of the fingerprint on the scheduling object so it is computed once and reused for every cache lookup of that job:

// On xce_primary_schedobj_t (or xce_sched_obj_t):
xce_resreq_fingerprint_t resreq_fp;  // populated on job ingest, read by cache lookup

Lookup site (pseudocode)

// Before allocation: construct key on stack, single hash + lookup via the
// real kvtab API (key_find/key_add -- NOT a free-function kvtab_lookup()
// taking hash/eq params, which doesn't exist in this codebase).
xce_cache_key_t key = {
    .resreq  = obj->u.primary->resreq_fp,
    .user    = obj->user,
    .queue   = obj->queue,
    .project = obj->project,
};
void* unused = NULL;
if (key_find(sched->unsatisfiable_cache, &key, &unused) > 0) {
    pend_job(obj, original_reason);
    return;
}
// on exhaustion (Req 7 AC 3): key_add(sched->unsatisfiable_cache, &key, SENTINEL)

Key design decisions

• struct-based: hash/cmp/assign/destroy live together in one kv_key_ops_t, matching this codebase's existing int_key_ops/string_key_ops idiom rather than an ad-hoc per-call hash/eq pair.
• memcmp on resreq portion: all fields are fixed-size integers/IDs — no pointers, no deep compare.
• pointer equality on identity: xstring guarantees identical content → identical pointer.
• select via node_id: CSE intern table deduplicates identical select expressions. O(1).
• rusages via pre-computed hash: set once at parse time, stable for life of resreq. O(1).
• span/same/affinity deferred: currently empty structs; add to xce_resreq_fingerprint_t when they gain fields.

xce_sched_ctx_t addition (allocation mode, Req 9 AC 5)

A mode field that controls which resource pool the allocator draws from. Defaults to XCE_ALLOC_NORMAL. Consumed by try_alloc_on_host() and any resource-availability check — applies to all resource types (slots, shared resources, limit instances, licenses, etc.), not just CPU slots.

// In xce_sched_ctx_t (sched/types.h):
typedef enum {
    XCE_ALLOC_NORMAL  = 0,  // only truly free resources
    XCE_ALLOC_PREEMPT,       // count preemptible job resources as available
    // Future:
    // XCE_ALLOC_RESERVE,    // count reserved-but-idle resources as available
    // XCE_ALLOC_BACKFILL,   // count reserved + preemptible resources
} xce_alloc_mode_t;

xce_alloc_mode_t alloc_mode;  // XCE_ALLOC_NORMAL by default

When alloc_mode != XCE_ALLOC_NORMAL, resource-availability checks count the full resource footprint of victim-candidate jobs (not just slots) as borrowable. The allocator logic is otherwise identical — no fork in try_alloc_on_host().

xce_preempt_victim_iter_t (new, scheduler-preemption spec)

Same lazy-iterator pattern as plan iterator, applied to victim selection:

typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t;

struct xce_preempt_victim_iter_s {
    int  (*init)(xce_preempt_victim_iter_t*, xce_sched_job_t* job, xce_scheduler_t*);
    xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t*);
    void (*fini)(xce_preempt_victim_iter_t*);
    void* private;
};

The default policy (lowest_priority_first) generates victim sets lazily: smallest sufficient set of lowest-priority running jobs from preemptible_queues that would free enough resources. Listed here because the preemption flow wraps allocate_primary() and consumes xce_alloc_mode_t — both defined in this spec.

parjob_allocator changes (Req 4)

• my_alloc() is replaced from a stub (return 0) to a real implementation that creates a plan iterator, walks candidate plans, and commits on first success.
• Correction (this review): "only job_alloc_fn changes" is wrong -- an earlier pass of this document said so before Req 4 AC 6 was added; parjob_prepare_allocation() (the plugin's separate prepare_allocation_hook) also changes, per the next bullet. What IS unchanged is the plugin's registration struct itself (name/init_fn/fini_fn/job_elect_fn/data, register_parjob_allocator()) -- only the bodies of its two hook implementations change.
• parjob_prepare_allocation() (currently also a stub -- see Req 4 AC 6) must be fixed alongside my_alloc(): it is the hook that populates ctx->alloc_req.host/ctx->alloc_req.shared (via calc_wanted_host_resvals/calc_wanted_global_resvals, the same calls seqjob_prepare_allocation() already makes) for whichever bucket is currently being tried. xce_sched_ctx_t already carries nhosts/nslots fields for exactly this; the per-bucket loop sets ctx->nslots = bucket_slots[b] (ctx->nhosts = 1) before each host try, and the fixed hook reads them instead of the hardcoded 1, 1 seqjob's version uses. Because ctx->alloc_req is a single scratch buffer reset and repopulated per candidate host (plugin_prepare_allocation(), plugin.c:591, xvec_idval_fini+inits it before each fan-out), this is safe to recompute per host-try -- see Req 4 AC 6 for the corollary seqjob_prepare_allocation() gate this depends on.
• Do not bypass plugin_prepare_allocation()'s fan-out to precompute a bucket's resource want once and reuse it across every candidate host. The rusage-derived portion (calc_wanted_host_resvals) is host-independent and in principle cacheable per bucket, but the same function's tail call to sched_fill_limit_requests(sched, &ctx->alloc_req.host, hlimits) reads hlimits = zarray_get(&obj->hlimits, ctx->worker->idx) -- genuinely host-specific per-host limit-tracking state. A full per-bucket bypass would silently stop enforcing per-host limits for jobs allocated through the new iterator path. Caching only the rusage-derived slice (still running the limit-fill per host) is a valid future optimization but is not required for this spec.

Final allocation result -- merged from each claimed bucket, no new type needed. xce_job_alloc_t (xce/types.h) already generalizes past one host: nhosts + xce_host_alloc_t** allocs. Today only seqjob_allocator exercises it, always with nhosts=1 via the variadic make_host_allocs(1, halloc). For a successful multi-bucket plan, commit_plan_allocation() builds one xce_host_alloc_t per claimed host (each from that bucket's own snapshotted alloc_req.host/uncomit, captured at the moment try_alloc_on_host succeeded for it -- alloc_req.host is overwritten on the next bucket's prepare/check cycle, so it cannot be read lazily after the loop moves on), collects them into an xce_host_alloc_t** hallocs array of length nbuckets (built directly with calloc+loop, since make_host_allocs()'s variadic signature only fits a compile-time-fixed argument list, not a runtime-sized claimed set), and calls the existing make_job_alloc(jobid, nbuckets, hallocs) unchanged. Shared-resource allocation (allocs_shared/nres_shared) stays job-level and computed once for the whole primary resreq, exactly as it is today -- it does not vary per bucket and is not part of the merge.

seqjob_allocator changes (Req 9 AC 1)

• Internal rewrap: my_alloc() is restructured to use the plan and host iterator interfaces internally as a trivial case (1 host x 1 slot plan, then host iteration matching today's sched_collect_workers() + xlist_find() logic).
• Externally observable behavior is unchanged -- same host selection given same state.
• seqjob_prepare_allocation() gains a primary->maxTasks == 1 guard (Req 4 AC 6). This is a no-op for seqjob's own observable behavior -- it never handles maxTasks > 1 jobs regardless -- but is required for parjob_allocator correctness once its own prepare hook stops being a stub; see Req 4 AC 6 for why the unguarded fan-out is unsafe.

Architecture Diagram

Allocation Flow (PRIMARY leaf)

┌──────────────────────────────────────────────────────────┐
│              ALLOCATE PRIMARY (Req 2+3+4)                  │
│                                                            │
│  plan_iter_init(primary_resreq)                            │
│       │                                                    │
│       ▼                                                    │
│  ┌──────────────────────┐                                  │
│  │ plan_iter->next()     │  ←─ generates per-host bucket   │
│  │ returns plan or NULL  │    partitions (e.g. [2,1])      │
│  └──────────┬───────────┘    most-preferred first          │
│             │                                              │
│       plan  │   NULL (exhausted)                           │
│             ▼              ▼                               │
│  ┌──────────────────┐   return ALLOC_FAIL                  │
│  │ host_iter_init()  │                                      │
│  └────────┬─────────┘                                      │
│           │                                                │
│           ▼                                                │
│  ┌──────────────────────┐                                  │
│  │ host_iter->next()     │  ←─ filters workers by          │
│  │ returns worker or NULL│    select expression +          │
│  └──────────┬───────────┘    slot availability             │
│             │                                              │
│     worker  │   NULL (no host for plan)                    │
│             ▼              ▼                               │
│  ┌──────────────────┐   host_iter->fini()                  │
│  │ try_alloc(worker, │   plan_iter->next()  (next plan)    │
│  │   plan, ctx)      │                                      │
│  └────────┬─────────┘                                      │
│           │                                                │
│   success │ fail                                           │
│           ▼        ▼                                       │
│  ┌────────────┐  host_iter->next() (next worker for plan)  │
│  │ commit      │                                            │
│  │ return      │                                            │
│  │ ALLOC_SUCCESS│                                           │
│  └────────────┘                                            │
└──────────────────────────────────────────────────────────┘

Simplified -- see "Composition Algorithm" above for the authoritative version. This diagram shows one worker being tried per plan, which is exactly right for seqjob_allocator (nbuckets == 1) but is NOT the full picture for parjob_allocator's multi-bucket plans: a real plan with nbuckets > 1 requires an inner loop over each bucket, accumulating a distinct claimed host per bucket before the plan counts as satisfied (host-exclusivity, per-bucket resource snapshotting, and rollback-on-partial-failure all live in that inner loop, none of which fits cleanly into this box diagram). Treat this diagram as the single-bucket special case; the pseudocode under "Composition Algorithm" is the one to implement against.

Unsatisfiable Resreq Cache (Req 7)

┌──────────────────────────────────────────────────────────┐
│              SESSION START (begin_session)                 │
│                                                            │
│  Clear unsatisfiable resreq fingerprint cache              │
│                                                            │
│  For each job from picker:                                │
│       │                                                    │
│       ▼                                                    │
│  ┌──────────────────────────────┐                         │
│  │ Lookup fingerprint in cache   │                         │
│  │ key = hash(minTasks,          │                         │
│  │   maxTasks, select_node_id,   │                         │
│  │   rusage_hash, user, queue,   │                         │
│  │   project)                    │                         │
│  └──────────┬───────────────────┘                         │
│             │                                              │
│     hit     │   miss                                       │
│      ▼      │     ▼                                        │
│  ┌────────┐ │  ┌──────────────────┐                       │
│  │ PEND    │ │  │ Run full          │                       │
│  │ immedi- │ │  │ allocation        │                       │
│  │ ately   │ │  │ (Req 2+3+4)       │                       │
│  └────────┘ │  └────────┬─────────┘                       │
│             │           │                                  │
│             │   success │ fail (exhausted)                 │
│             │           ▼        ▼                         │
│             │  ┌────────────┐ ┌──────────────────────┐    │
│             │  │ DISPATCH    │ │ Populate cache with   │    │
│             │  │ (no cache   │ │ resreq fingerprint    │    │
│             │  │  entry)     │ │ then PEND job         │    │
│             │  └────────────┘ └──────────────────────┘    │
└──────────────────────────────────────────────────────────┘

Integration Note: Preemption Interaction

Preemption (scheduler-preemption spec) fires when allocation fails for a preemptible job. It follows the same lazy-iterator pattern as allocation, applied to victim selection instead of host selection.

Preemption Iterator

Same shape as plan iterator, different domain:

// Preemption victim iterator — lazy generation of victim sets
typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t;

struct xce_preempt_victim_iter_s {
    int  (*init)(xce_preempt_victim_iter_t* iter, xce_sched_job_t* job,
                 xce_scheduler_t* sched);
    xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t* iter);
    void (*fini)(xce_preempt_victim_iter_t* iter);
    void* private;
};

The default policy (lowest_priority_first) generates victim sets in order: smallest sufficient set of lowest-priority running jobs from preemptible_queues that would free enough resources for the preempting job.

Allocation Mode

A xce_alloc_mode_t enum on xce_sched_ctx_t controls which resource pool the allocator draws from — applies to all resource types (slots, shared resources, limit instances, licenses), not just CPU slots:

typedef enum {
    XCE_ALLOC_NORMAL  = 0,  // only truly free resources
    XCE_ALLOC_PREEMPT,       // count preemptible job resources as available
} xce_alloc_mode_t;

When alloc_mode != XCE_ALLOC_NORMAL, resource-availability checks count the full resource footprint of victim-candidate jobs as borrowable. The allocator logic is otherwise identical — no fork in try_alloc_on_host().

Full Preemption Flow

Cache placement, corrected: the cache is keyed by resreq_fp, which lives on xce_primary_schedobj_t -- a PRIMARY-level concept, not a whole-job one (a COMPOUND/ALTER job has no single resreq shape of its own to fingerprint). So the cache check wraps allocate_primary() itself, not allocate_job()'s top-level dispatch. For a bare PRIMARY job this is the whole story; for a COMPOUND or ALTER job, each nested primary gets its own cache check for free, once for every call allocate_compound()/allocate_alter() already makes into allocate_primary() -- no job-level cache entry is needed or possible.

// Wraps allocate_primary() -- called for EVERY primary reached, whether the
// job's own resreq IS a primary, or this is one primary nested inside a
// COMPOUND (allocate_compound()'s loop) or an ALTER's compound (transitively).
int allocate_primary_with_preempt(sched, primary_resreq, ctx, job) {
    // 1. Cache check (Req 7) -- keyed on THIS primary's resreq_fp, not job-level
    if (cache_lookup(primary_resreq, job)) return PEND;

    // 2. Normal allocation
    if (allocate_primary(sched, primary_resreq, ctx) > 0) return ALLOC_SUCCESS;

    // 3. Preemption (if applicable)
    if (job->queue->preemptible_queues == NULL) {
        cache_insert(primary_resreq, job);
        return PEND;
    }

    victim_iter_init(&viter, job, sched);
    while ((victims = viter.next(&viter)) != NULL) {
        ctx.alloc_mode = XCE_ALLOC_PREEMPT;
        if (allocate_primary(sched, primary_resreq, &ctx) > 0) {
            viter.fini(&viter);
            return commit_preempt(victims);
        }
        ctx.alloc_mode = XCE_ALLOC_NORMAL;
    }
    viter.fini(&viter);

    cache_insert(primary_resreq, job);
    return PEND;
}

Key properties:

• v1 cache is preemption-agnostic (populated on allocation failure alone).
• When preemption ships, it fires between allocation failure and cache population.
• If preemption succeeds → no cache entry, preempting job is re-pended for next round.
• If preemption fails/exhausted → cache entry populated, job pended normally.
• Plan and host iterators are not preemption-aware — they simply report success/failure.
• xce_alloc_mode_t alloc_mode is the only new allocator surface needed.

Implementation Order

Scheduler sub-specs are designed for sequential delivery:

Phase 1: scheduler-allocation     (this spec)
  ├── plan_iter + host_iter       foundation iterators
  ├── seqjob rewrap               validates abstraction on simple case
  ├── parjob combinatorial        real parallel allocation
  └── resreq cache                performance optimization

Phase 2: scheduler-core
  ├── core/driver split           extracts xce_sched_core_t from xce_scheduler_t
  ├── picker interface            xce_job_picker_iface_t + builtin_fcfs
  └── session loop wires          core_go() calls allocate_job()

Phase 3: scheduler-preemption
  ├── victim_iter                 lazy victim-set generator
  ├── XCE_ALLOC_PREEMPT mode      tells allocator to count borrowed resources
  └── preempt_policy plugin       lowest_priority_first default

Phase	Why there
Allocation first	Self-contained. Unit-testable with fake workers. No core, no preemption needed. Iterators are the building block everything else sits on.
Core second	Wires allocation into a real scheduling loop. Picker feeds jobs → allocator places them → decisions flow. Without allocation, core has nothing to call.
Preemption last	Wraps allocation. Needs allocate_primary() to exist. Needs core session loop to re-pend preempted job. Is a "retry with borrowing" layer on top of normal allocation.