myxce/.kiro/specs/scheduler-allocation/requirements.md

Requirements Document

Introduction

The XCE scheduler has a plugin architecture (plugin.h/plugin.c, xce_sched_plugin_t) but three gaps keep it from being a real framework. The first gap -- job selection policy is hardcoded (sched_job_picker() sets job_pick_plugin = "xce_sched_fcfs" as a literal) -- is addressed in the sibling scheduler-core spec.

This spec closes the remaining two allocation gaps:

1. Parallel job allocation is a stub: parjob_allocator.c's my_alloc() takes (plugin, ctx) and immediately return 0 -- any job with maxTasks > 1 (routed there by sched_make_primary_alloc_() at scheduler.c:93-103) produces no allocation at all.

2. Allocation is eager and one-shot: the sibling seqjob_allocator.c's my_alloc() (used when maxTasks == 1) calls sched_collect_workers() then xlist_find() for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.

This spec introduces an iterator-driven allocation architecture with these components:

• A lazy xce_alloc_plan_iter_t -- candidate (nhosts, slots_per_host) plans, most-preferred first
• A lazy xce_host_iter_t -- candidate hosts for a given plan, filtered by the existing select-expression evaluator
• A parjob_allocator that walks the plan iterator to do real combinatorial multi-slot allocation
• A session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session
• A seqjob_allocator rewrap onto the iterator interfaces as a trivial 1-plan/1-slot case, validating the abstraction before applying it to the complex parjob case

Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in .kiro/specs/scheduler-allocation/brief.md.

Boundary Context

• In scope: xce_alloc_plan_iter_t and xce_host_iter_t interfaces (lazy, cancel-safe, composable); seqjob_allocator rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, maxTasks == 1 only); parjob_allocator's my_alloc() implementing real combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) for maxTasks > 1; xce_alloc_mode_t on xce_sched_ctx_t for allocation mode (NORMAL/PREEMPT); session-level unsatisfiable-resreq fingerprint cache keyed by xce_cache_key_t (O(1) content hash from parsed resreq + identity); unit tests for plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence; integration test for end-to-end multi-slot parallel job allocation.

• Out of scope: Pluggable job picker interface (xce_job_picker_iface_t) -- in scheduler-core; builtin_fcfs picker -- in scheduler-core; configurable picker plugin with per-queue override -- in scheduler-core; preemption (Req 6 of scheduler-framework) -- in scheduler-preemption; core/driver separation (Req 10 of scheduler-framework) -- in scheduler-core; Queue vs Project roles -- in scheduler-core; new job selection policies beyond the builtin FCFS picker; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins).

• Adjacent expectations: depends on the existing plugin framework (plugin.h/plugin.c), the existing select-expression evaluator (resreq.h, memoized via select_eval_ctx_t), and existing shared-resource tracking (xce_scheduler_t.sharedInst/sharedInUse) -- none of these change. Depends on scheduler-core for the core event interface that pushes worker and resource state; the iterators are independently unit-testable with constructed state. ha-consistency-invariants' commit barrier is a future consumer of scheduling decisions; this spec does not wire it.

Allocation Architecture (Reqs 2-4)

Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER):

Resreq Type Hierarchy

ALTER     ->  OR-of-compounds: first compound that fully allocates wins
  └─ COMPOUND ->  AND-of-primaries: ALL primaries must allocate for success
       └─ PRIMARY  ->  plan_iter -> host_iter -> try_alloc  (leaf level)

The iterator composition (xce_alloc_plan_iter_t + xce_host_iter_t) operates at the PRIMARY leaf. COMPOUND and ALTER are composition rules above the iterators -- they call into the same primary allocator, unchanged from today's sched_make_compound_allocation() and sched_make_alter_allocation() dispatch.

Iterator Composition (PRIMARY leaf)

                    ┌──────────────────────────┐
                    │  xce_alloc_plan_iter_t    │  Req 2
                    │                            │
                    │  init(primary_resreq, sched)  │
                    │  next() -> plan            │  generates (nhosts, slots_per_host)
                    │  fini()                    │  candidates, most-preferred first
                    └────────────┬───────────────┘
                                 │ plan
                                 ▼
                    ┌──────────────────────────┐
                    │  xce_host_iter_t          │  Req 3
                    │                            │
                    │  init(plan, workers)       │  for a given plan, iterate
                    │  next() -> worker           │  candidate hosts matching
                    │  fini()                    │  select expression + availability
                    └────────────┬───────────────┘
                                 │ worker
                                 ▼
                    ┌──────────────────────────┐
                    │  try_alloc(worker, plan)  │  Req 4 (parjob) or
                    │                            │  existing seqjob logic
                    │  check slots, shared res,  │
                    │  commit or advance         │
                    └────────────────────────────┘

Composition Algorithm (All Three Types)

// === PRIMARY leaf: iterator composition ===
// Returns >0 on success, 0 on failure
int allocate_primary(sched, primary_resreq, ctx) {
    plan_iter = plan_iter_init(primary_resreq, sched);
    while ((plan = plan_iter->next(plan_iter)) != NULL) {
        host_iter = host_iter_init(plan, sched->workers);
        while ((worker = host_iter->next(host_iter)) != NULL) {
            if (try_alloc_on_host(worker, plan, ctx) > 0) {
                host_iter->fini(host_iter);
                plan_iter->fini(plan_iter);
                return ALLOC_SUCCESS;
            }
        }
        host_iter->fini(host_iter);
    }
    plan_iter->fini(plan_iter);
    return ALLOC_FAIL;
}

// === COMPOUND: AND of primaries ===
// ALL primaries must allocate. On any failure, roll back prior allocations
// and return failure. Same logic as today's sched_make_compound_allocation().
int allocate_compound(sched, compound_resreq, ctx) {
    for (int i = 0; i < compound_resreq->primaries.count; i++) {
        primary = compound_resreq->primaries[i];
        if (allocate_primary(sched, primary, ctx) <= 0) {
            // Roll back all primaries allocated so far in this compound
            rollback_compound_allocations(ctx, i);
            return ALLOC_FAIL;
        }
    }
    return ALLOC_SUCCESS;
}

// === ALTER: OR of compounds ===
// First compound that fully allocates wins. Same logic as today's
// sched_make_alter_allocation(). On failure, no rollback needed
// (prior compounds were never committed -- they failed their own checks).
int allocate_alter(sched, alter_resreq, ctx) {
    for (int i = 0; i < alter_resreq->compounds.count; i++) {
        compound = alter_resreq->compounds[i];
        if (allocate_compound(sched, compound, ctx) > 0) {
            return ALLOC_SUCCESS;
        }
        // This compound failed -- advance to next alternative
    }
    return ALLOC_FAIL;
}

// === Top-level dispatch ===
// Unchanged from today's sched_make_allocation() dispatch on obj->tag.
int allocate_job(sched, job, ctx) {
    switch (job->resreq_type) {
    case PRIMARY:
        return allocate_primary(sched, job->primary_resreq, ctx);
    case COMPOUND:
        return allocate_compound(sched, job->compound_resreq, ctx);
    case ALTER:
        return allocate_alter(sched, job->alter_resreq, ctx);
    }
}

COMPOUND rollback note: If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back -- the compound is atomic. Today this is handled by sched_make_compound_allocation() tracking allocations in ctx and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract.

Who Uses This Composition

Allocator	maxTasks	Plan iterator	Host iterator	Behavior
seqjob_allocator	== 1	Trivial: single plan (1 host x 1 slot)	Full: filters hosts by select expression	Unchanged from today -- first matching host wins
parjob_allocator	> 1	Full combinatorial: 1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1	Full: filters hosts with enough available slots	New -- walks plans until one succeeds
Future allocators	any	Custom strategy (e.g., topology-aware, bin-packing)	Reuses same xce_host_iter_t	Pluggable via job_alloc_fn

Key Design Properties

• Lazy: plan_iter->next() generates one plan at a time. host_iter->next() returns one host at a time. No full materialization.
• Cancel-safe: both next() calls have pthread_cancel cancellation points.
• Independent lifecycles: a host_iter is created from a plan, used, and destroyed; the plan_iter is advanced to the next plan only after the host_iter is done. fini() on one does not affect the other.
• COMPOUND/ALTER unchanged: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside allocate_primary().
• Seqjob rewrap: seqjob_allocator is rewrapped onto the iterator interfaces as a trivial case -- its plan iterator always returns the single plan (1 host x 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case.
• Pluggable plan generation, shared host iteration: plan generation is allocator-specific — each job_alloc_fn brings its own strategy (trivial, combinatorial, topology-aware, bin-packing) and creates its own xce_alloc_plan_iter_t internally. Host iteration is shared infrastructure — xce_host_iter_t has one implementation reused by all allocators, parameterized by sort order (HOST_ORDER_ANY, HOST_ORDER_MOST_FREE, etc.). Swapping allocation strategy = swapping the plugin; no core loop change.

Requirements

Requirement 2: Lazy allocation plan iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front.

Acceptance Criteria

1. The Scheduler shall define an xce_alloc_plan_iter_t interface with init(resreq, sched), next() -> plan, and fini() operations. The init() signature passes the full xce_primary_resreq_t (including span, same, affinity) and scheduler state — sufficient for any allocator to derive its plan space without future interface changes.

   Plan-shaping fields (on xce_primary_resreq_t, consumed by plan iterator internally):

   • span: bounds host count (nhosts_min/nhosts_max) and per-host task cap (max_per_host, analogous to LSF ptile). v1 combinatorial generation ignores these (empty structs); fill when ptile/nhosts support is needed.
   • same: co-location constraints ("all tasks on same rack"). v1 ignores; future.
   • affinity: restrict to named host group. v1 ignores; future.
   • minTasks/maxTasks: task count bounds — the primary driver of plan space. Used in v1.
   • select: host filter expression — consumed by host iterator (Req 3), not plan iterator.
   • rusages: resource requests — consumed by try_alloc_on_host, not plan iterator.

   v1 implements simple combinatorial generation: 1host×N, 2hosts×ceil(N/2), ..., Nhosts×1. Future allocators read span/same/affinity to constrain the plan space without touching the init() signature.

2. Each call to next() shall generate exactly one candidate (nhosts, slots_per_host, host_filter) plan without generating or materializing any other plan in the same call.

3. The Scheduler shall order plans returned by successive next() calls from most-preferred to least-preferred, per the owning allocator's preference policy.

4. next() shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion.

5. Calling fini() shall release all resources held by the iterator, and the interface shall define no other required cleanup step.

Requirement 3: Lazy host candidate iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering.

Acceptance Criteria

1. The Scheduler shall define an xce_host_iter_t interface with init(plan, workers), next() -> worker, and fini() operations.

2. Each call to next() shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.

3. next() shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call next() again after receiving NULL for that plan.

4. The host iterator shall compose with the plan iterator: a new xce_host_iter_t shall be constructible from any plan produced by Requirement 2's xce_alloc_plan_iter_t.next().

5. Calling fini() shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.

Requirement 4: Parallel job combinatorial allocation

Objective: As a cluster operator submitting a job that needs multiple task slots, I want parjob_allocator to actually allocate across hosts, so that jobs with maxTasks > 1 are no longer silently unallocatable.

Acceptance Criteria

1. When a job's xce_primary_resreq_t.maxTasks > 1 reaches parjob_allocator's job_alloc_fn (my_alloc() in parjob_allocator.c), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host x N slots, 2 hosts x ceil(N/2) slots, ..., N hosts x 1 slot, using Requirement 2's plan iterator.

2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan.

3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job.

4. When no plan among those generated in Criterion 1 can be satisfied, my_alloc() shall return the same not-satisfied signal (0) that the current stub returns today, so pend-reason handling in sched_make_primary_alloc_() is unaffected.

5. parjob_allocator's job_alloc_fn shall no longer unconditionally return 0 regardless of ctx contents -- the current stub behavior -- for any job with maxTasks > 1.

Requirement 7: Session-level unsatisfiable resreq cache

Objective: As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.

Acceptance Criteria

1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on xce_scheduler_t, keyed by xce_cache_key_t — a value type composing xce_resreq_fingerprint_t (pre-computed {minTasks, maxTasks, select_node_id, rusage_hash}) and identity ({user, queue, project} as xstring pointers). The resreq fingerprint is stored on the scheduling object (xce_primary_schedobj_t) and computed once on job ingest. The full cache key is constructed on the stack at lookup time — a single kvtab_lookup() with cache_key_hash() and cache_key_eq(). Fully O(1): no list walks, no string traversals, no allocations. This captures everything the allocator and limit plugin consume.

2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (begin_session).

3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. In v1, cache population is preemption-agnostic: it fires on allocation failure alone. When the scheduler-preemption spec ships, preemption fires between allocation failure and cache population — if preemption succeeds, no cache entry is created; if preemption fails or is inapplicable, the cache entry is populated as normal. The allocation layer does not call into preemption; the caller (scheduling loop) orchestrates the sequence.

4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption.

5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.

Requirement 9: No regression to existing allocators or HA soak

Objective: As a maintainer, I want the iterator-driven rework scoped so that existing single-slot allocation and COMPOUND/ALTER dispatch are unaffected, so that this framework change is safe to ship independently of any new allocator.

Acceptance Criteria

1. Seqjob regression: seqjob_allocator's externally observable allocation behavior (which host a maxTasks == 1 job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the xce_host_iter_t interface.

2. COMPOUND/ALTER unchanged: The COMPOUND and ALTER resreq allocation paths (sched_make_compound_allocation, sched_make_alter_allocation in scheduler.c) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today.

3. Unit tests: Existing scheduler unit tests (scheduler meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the eager-one-shot behavior or stub return value this feature replaces.

4. HA soak: A full HA soak (sequential and chaotic /xce-test-ha regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.

Data Structure Changes Summary

xce_alloc_plan_iter_t (new, Req 2)

typedef struct xce_alloc_plan_iter_s xce_alloc_plan_iter_t;

struct xce_alloc_plan_iter_s {
    // Initialize the iterator for a given primary resreq and scheduler state.
    // Returns 0 on success, nonzero on error.
    int  (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq,
                 xce_scheduler_t* sched);

    // Return the next candidate plan, or NULL when exhausted.
    // The returned plan pointer is valid until the next next() or fini() call.
    xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter);

    // Release all resources held by the iterator.
    void (*fini)(xce_alloc_plan_iter_t* iter);

    // Private state: current position, strategy-specific data
    void* private;
};

// A plan describes how many hosts and slots to attempt
typedef struct {
    int nhosts;          // number of hosts needed
    int slots_per_host;  // slots required on each host
    // host_filter: derived from primary_resreq->select, plus slot availability
} xce_alloc_plan_t;

xce_host_iter_t (new, Req 3)

typedef struct xce_host_iter_s xce_host_iter_t;

struct xce_host_iter_s {
    // Initialize the iterator for a given plan and worker set.
    // Returns 0 on success, nonzero on error.
    int  (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan,
                 var workers);

    // Return the next candidate worker matching the plan's host filter,
    // or NULL when exhausted. The returned worker pointer is valid until
    // the next next() or fini() call.
    xce_worker_t* (*next)(xce_host_iter_t* iter);

    // Release all resources held by the iterator.
    void (*fini)(xce_host_iter_t* iter);

    // Private state: current position, filtering state
    void* private;
};

New types (Req 7)

// Resreq portion of the cache key — pre-computed, shareable across jobs
// with identical resreqs. Fits in a single cache line (24 bytes).
typedef struct {
    int      min_tasks;
    int      max_tasks;
    uint32_t select_node_id;   // CSE-interned root; 0 if select is NULL
    uint64_t rusage_hash;      // computed at parse time, stable for life of resreq
} xce_resreq_fingerprint_t;

// Full cache key — resreq fingerprint + identity.
// Identity fields are xstring pointers (not owned, not freed).
// Pointer equality = content equality for xstring-interned strings.
typedef struct {
    xce_resreq_fingerprint_t resreq;
    const char*              user;     // xstring pointer
    const char*              queue;    // xstring pointer
    const char*              project;  // xstring pointer
} xce_cache_key_t;

// Hash — single call site, no ad-hoc field assembly
static inline uint64_t cache_key_hash(const xce_cache_key_t* k) {
    uint64_t h = FNV64_INIT;
    hash_update(&h, &k->resreq, sizeof(k->resreq));
    hash_update_ptr(&h, k->user);
    hash_update_ptr(&h, k->queue);
    hash_update_ptr(&h, k->project);
    return h;
}

// Equality — memcmp on fixed-size resreq portion + pointer equality on identity
static inline bool cache_key_eq(const xce_cache_key_t* a, const xce_cache_key_t* b) {
    return memcmp(&a->resreq, &b->resreq, sizeof(a->resreq)) == 0
        && a->user    == b->user
        && a->queue   == b->queue
        && a->project == b->project;
}

xce_primary_resreq_t addition (Req 7)

One new field, set once at parse time:

// In xce_primary_resreq_t (resreq.h):
uint64_t rusage_hash;  // set by parse_resreq_primary(), stable for life of resreq

xce_sched_obj_t / xce_primary_schedobj_t addition (Req 7)

Store the resreq portion of the fingerprint on the scheduling object so it is computed once and reused for every cache lookup of that job:

// On xce_primary_schedobj_t (or xce_sched_obj_t):
xce_resreq_fingerprint_t resreq_fp;  // populated on job ingest, read by cache lookup

Lookup site (pseudocode)

// Before allocation: construct key on stack, single hash + lookup
xce_cache_key_t key = {
    .resreq  = obj->u.primary->resreq_fp,
    .user    = obj->user,
    .queue   = obj->queue,
    .project = obj->project,
};
if (kvtab_lookup(sched->unsatisfiable_cache, &key, cache_key_hash, cache_key_eq)) {
    pend_job(obj, original_reason);
    return;
}

Key design decisions

• struct-based: hash and equality live next to the type. Single call site, no scattered field assembly.
• memcmp on resreq portion: all fields are fixed-size integers/IDs — no pointers, no deep compare.
• pointer equality on identity: xstring guarantees identical content → identical pointer.
• select via node_id: CSE intern table deduplicates identical select expressions. O(1).
• rusages via pre-computed hash: set once at parse time, stable for life of resreq. O(1).
• span/same/affinity deferred: currently empty structs; add to xce_resreq_fingerprint_t when they gain fields.

xce_sched_ctx_t addition (allocation mode)

A mode field that controls which resource pool the allocator draws from. Defaults to XCE_ALLOC_NORMAL. Consumed by try_alloc_on_host() and any resource-availability check — applies to all resource types (slots, shared resources, limit instances, licenses, etc.), not just CPU slots.

// In xce_sched_ctx_t (sched/types.h):
typedef enum {
    XCE_ALLOC_NORMAL  = 0,  // only truly free resources
    XCE_ALLOC_PREEMPT,       // count preemptible job resources as available
    // Future:
    // XCE_ALLOC_RESERVE,    // count reserved-but-idle resources as available
    // XCE_ALLOC_BACKFILL,   // count reserved + preemptible resources
} xce_alloc_mode_t;

xce_alloc_mode_t alloc_mode;  // XCE_ALLOC_NORMAL by default

When alloc_mode != XCE_ALLOC_NORMAL, resource-availability checks count the full resource footprint of victim-candidate jobs (not just slots) as borrowable. The allocator logic is otherwise identical — no fork in try_alloc_on_host().

xce_preempt_victim_iter_t (new, scheduler-preemption spec)

Same lazy-iterator pattern as plan iterator, applied to victim selection:

typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t;

struct xce_preempt_victim_iter_s {
    int  (*init)(xce_preempt_victim_iter_t*, xce_sched_job_t* job, xce_scheduler_t*);
    xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t*);
    void (*fini)(xce_preempt_victim_iter_t*);
    void* private;
};

The default policy (lowest_priority_first) generates victim sets lazily: smallest sufficient set of lowest-priority running jobs from preemptible_queues that would free enough resources. Listed here because the preemption flow wraps allocate_primary() and consumes xce_alloc_mode_t — both defined in this spec.

parjob_allocator changes (Req 4)

• my_alloc() is replaced from a stub (return 0) to a real implementation that creates a plan iterator, walks candidate plans, and commits on first success.
• The allocator's registration in the plugin framework is unchanged -- only the job_alloc_fn implementation changes.

seqjob_allocator changes (Req 9 AC 1)

• Internal rewrap: my_alloc() is restructured to use the plan and host iterator interfaces internally as a trivial case (1 host x 1 slot plan, then host iteration matching today's sched_collect_workers() + xlist_find() logic).
• Externally observable behavior is unchanged -- same host selection given same state.

Architecture Diagram

Allocation Flow (PRIMARY leaf)

┌──────────────────────────────────────────────────────────┐
│              ALLOCATE PRIMARY (Req 2+3+4)                  │
│                                                            │
│  plan_iter_init(primary_resreq)                            │
│       │                                                    │
│       ▼                                                    │
│  ┌──────────────────────┐                                  │
│  │ plan_iter->next()     │  ←─ generates (nhosts,          │
│  │ returns plan or NULL  │    slots_per_host) candidates   │
│  └──────────┬───────────┘    most-preferred first          │
│             │                                              │
│       plan  │   NULL (exhausted)                           │
│             ▼              ▼                               │
│  ┌──────────────────┐   return ALLOC_FAIL                  │
│  │ host_iter_init()  │                                      │
│  └────────┬─────────┘                                      │
│           │                                                │
│           ▼                                                │
│  ┌──────────────────────┐                                  │
│  │ host_iter->next()     │  ←─ filters workers by          │
│  │ returns worker or NULL│    select expression +          │
│  └──────────┬───────────┘    slot availability             │
│             │                                              │
│     worker  │   NULL (no host for plan)                    │
│             ▼              ▼                               │
│  ┌──────────────────┐   host_iter->fini()                  │
│  │ try_alloc(worker, │   plan_iter->next()  (next plan)    │
│  │   plan, ctx)      │                                      │
│  └────────┬─────────┘                                      │
│           │                                                │
│   success │ fail                                           │
│           ▼        ▼                                       │
│  ┌────────────┐  host_iter->next() (next worker for plan)  │
│  │ commit      │                                            │
│  │ return      │                                            │
│  │ ALLOC_SUCCESS│                                           │
│  └────────────┘                                            │
└──────────────────────────────────────────────────────────┘

Unsatisfiable Resreq Cache (Req 7)

┌──────────────────────────────────────────────────────────┐
│              SESSION START (begin_session)                 │
│                                                            │
│  Clear unsatisfiable resreq fingerprint cache              │
│                                                            │
│  For each job from picker:                                │
│       │                                                    │
│       ▼                                                    │
│  ┌──────────────────────────────┐                         │
│  │ Lookup fingerprint in cache   │                         │
│  │ key = hash(minTasks,          │                         │
│  │   maxTasks, select_node_id,   │                         │
│  │   rusage_hash, user, queue,   │                         │
│  │   project)                    │                         │
│  └──────────┬───────────────────┘                         │
│             │                                              │
│     hit     │   miss                                       │
│      ▼      │     ▼                                        │
│  ┌────────┐ │  ┌──────────────────┐                       │
│  │ PEND    │ │  │ Run full          │                       │
│  │ immedi- │ │  │ allocation        │                       │
│  │ ately   │ │  │ (Req 2+3+4)       │                       │
│  └────────┘ │  └────────┬─────────┘                       │
│             │           │                                  │
│             │   success │ fail (exhausted)                 │
│             │           ▼        ▼                         │
│             │  ┌────────────┐ ┌──────────────────────┐    │
│             │  │ DISPATCH    │ │ Populate cache with   │    │
│             │  │ (no cache   │ │ resreq fingerprint    │    │
│             │  │  entry)     │ │ then PEND job         │    │
│             │  └────────────┘ └──────────────────────┘    │
└──────────────────────────────────────────────────────────┘

Integration Note: Preemption Interaction

Preemption (scheduler-preemption spec) fires when allocation fails for a preemptible job. It follows the same lazy-iterator pattern as allocation, applied to victim selection instead of host selection.

Preemption Iterator

Same shape as plan iterator, different domain:

// Preemption victim iterator — lazy generation of victim sets
typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t;

struct xce_preempt_victim_iter_s {
    int  (*init)(xce_preempt_victim_iter_t* iter, xce_sched_job_t* job,
                 xce_scheduler_t* sched);
    xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t* iter);
    void (*fini)(xce_preempt_victim_iter_t* iter);
    void* private;
};

The default policy (lowest_priority_first) generates victim sets in order: smallest sufficient set of lowest-priority running jobs from preemptible_queues that would free enough resources for the preempting job.

Allocation Mode

A xce_alloc_mode_t enum on xce_sched_ctx_t controls which resource pool the allocator draws from — applies to all resource types (slots, shared resources, limit instances, licenses), not just CPU slots:

typedef enum {
    XCE_ALLOC_NORMAL  = 0,  // only truly free resources
    XCE_ALLOC_PREEMPT,       // count preemptible job resources as available
} xce_alloc_mode_t;

When alloc_mode != XCE_ALLOC_NORMAL, resource-availability checks count the full resource footprint of victim-candidate jobs as borrowable. The allocator logic is otherwise identical — no fork in try_alloc_on_host().

Full Preemption Flow

int allocate_job(sched, job) {
    // 1. Cache check
    if (cache_lookup(job)) return pend(job);

    // 2. Normal allocation
    if (allocate_primary(sched, job->resreq, ctx) > 0) return dispatch(job);

    // 3. Preemption (if applicable)
    if (job->queue->preemptible_queues == NULL) {
        cache_insert(job);
        return pend(job);
    }

    victim_iter_init(&viter, job, sched);
    while ((victims = viter.next(&viter)) != NULL) {
        ctx.alloc_mode = XCE_ALLOC_PREEMPT;
        if (allocate_primary(sched, job->resreq, &ctx) > 0) {
            viter.fini(&viter);
            return commit_preempt(victims, job);
        }
        ctx.alloc_mode = XCE_ALLOC_NORMAL;
    }
    viter.fini(&viter);

    cache_insert(job);
    return pend(job);
}

Key properties:

• v1 cache is preemption-agnostic (populated on allocation failure alone).
• When preemption ships, it fires between allocation failure and cache population.
• If preemption succeeds → no cache entry, preempting job is re-pended for next round.
• If preemption fails/exhausted → cache entry populated, job pended normally.
• Plan and host iterators are not preemption-aware — they simply report success/failure.
• xce_alloc_mode_t alloc_mode is the only new allocator surface needed.

Implementation Order

Scheduler sub-specs are designed for sequential delivery:

Phase 1: scheduler-allocation     (this spec)
  ├── plan_iter + host_iter       foundation iterators
  ├── seqjob rewrap               validates abstraction on simple case
  ├── parjob combinatorial        real parallel allocation
  └── resreq cache                performance optimization

Phase 2: scheduler-core
  ├── core/driver split           extracts xce_sched_core_t from xce_scheduler_t
  ├── picker interface            xce_job_picker_iface_t + builtin_fcfs
  └── session loop wires          core_go() calls allocate_job()

Phase 3: scheduler-preemption
  ├── victim_iter                 lazy victim-set generator
  ├── XCE_ALLOC_PREEMPT mode      tells allocator to count borrowed resources
  └── preempt_policy plugin       lowest_priority_first default

Phase	Why there
Allocation first	Self-contained. Unit-testable with fake workers. No core, no preemption needed. Iterators are the building block everything else sits on.
Core second	Wires allocation into a real scheduling loop. Picker feeds jobs → allocator places them → decisions flow. Without allocation, core has nothing to call.
Preemption last	Wraps allocation. Needs allocate_primary() to exist. Needs core session loop to re-pend preempted job. Is a "retry with borrowing" layer on top of normal allocation.