myxce/.kiro/specs/scheduler-allocation/brief.md

Brief: scheduler-allocation

Problem

The XCE scheduler has a plugin architecture (plugin.h/plugin.c, xce_sched_plugin_t) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: sched_job_picker() sets job_pick_plugin = "xce_sched_fcfs" as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: parjob_allocator.c's my_alloc() takes (plugin, ctx) and immediately return 0 -- any job with maxTasks > 1 (routed there by sched_make_primary_alloc_()) produces no allocation at all. Third, allocation is eager and one-shot: the sibling seqjob_allocator.c's my_alloc() (used when maxTasks == 1) calls sched_collect_workers() then xlist_find() for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.

The first gap (hardcoded picker) is addressed in the sibling scheduler-core spec. This spec, scheduler-allocation, closes the other two: the parjob stub and one-shot allocation.

Current State

What exists (allocation surface)

• Seqjob allocator (seqjob_allocator.c): handles maxTasks == 1. Checks shared resources, collects workers via sched_collect_workers(), tries one host, returns first match. No retry with alternative plans.
• Parjob allocator stub (parjob_allocator.c): registered as a plugin but my_alloc() is empty (return 0). Any job with maxTasks > 1 silently fails allocation.
• Resource request types (resreq.h): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds). The type hierarchy is well-defined but allocation at the PRIMARY leaf lacks iteration abstractions.
• Select expression evaluator (resreq.h): full parser/evaluator with per-session memoization (select_eval_ctx_t) and inter-session GC. Already reusable for host filtering.
• Shared resources (xce_scheduler_t.sharedInst/sharedInUse): zarray of resource values tracked per scheduling session.
• Allocation dispatch (scheduler.c:sched_make_allocation(), sched_make_primary_alloc_()): routes to seqjob or parjob allocator based on maxTasks.

What's missing

• Lazy plan generation: no xce_alloc_plan_iter_t -- plans are not generated at all, let alone lazily
• Lazy host iteration: no xce_host_iter_t -- host collection is eager via sched_collect_workers(), materializing the full list before trying any
• Parallel job allocation: parjob_allocator is a stub. No combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1), no multi-slot host matching
• Unsatisfiable resreq caching: a batch of similarly-shaped unsatisfiable jobs each re-pays the full allocation cost within a session

Desired Outcome

Iterator-Driven Allocation Architecture

A lazy xce_alloc_plan_iter_t (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with a lazy xce_host_iter_t (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a parjob_allocator implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session.

Three-Layer Iterator Composition

ALTER     ->  OR-of-compounds: first compound that fully allocates wins (unchanged)
  └─ COMPOUND ->  AND-of-primaries: ALL primaries must allocate (unchanged)
       └─ PRIMARY  ->  plan_iter -> host_iter -> try_alloc  (NEW iterator layer)

The iterator composition operates at the PRIMARY leaf. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today.

Iterator Pipeline

xce_alloc_plan_iter_t    -- init(resreq) -> next() -> plan -> next() -> ... -> NULL -> fini()
       │ plan
       ▼
xce_host_iter_t          -- init(plan, workers) -> next() -> worker -> ... -> NULL -> fini()
       │ worker
       ▼
try_alloc(worker, plan)  -- check slots, shared res, commit or advance

Key Properties

• Lazy: plan_iter->next() generates one plan at a time. host_iter->next() returns one host at a time. No full materialization.
• Cancel-safe: both next() calls have pthread_cancel cancellation points.
• Independent lifecycles: a host_iter is created from a plan, used, and destroyed; the plan_iter is advanced to the next plan only after the host_iter is done.
• COMPOUND/ALTER unchanged: the composition rules above the PRIMARY leaf are unchanged from today.
• Seqjob rewrap: seqjob_allocator is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host x 1 slot). This validates the abstraction before applying it to the complex parjob case.

Alternatives Considered

Approach A: Incremental Enhancement (rejected)

Aspect	Detail
Plan generation	Pre-computed plan array. Eager generation for parjob.
Host iteration	No new abstraction. Simple for-loop over sched_collect_workers() result.
Unsatisfiable resreq	xlist_t unsatisfiable_resreqs in session state

Rejected because: Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.

Approach B: Iterator-Driven Allocation (chosen)

Aspect	Detail
Plan iterator	xce_alloc_plan_iter_t — lazy, cancel-safe, generates one plan per next() call
Host iterator	xce_host_iter_t — lazy per-plan host filtering via existing select-expression evaluator
Parallel jobs	Combinatorial plan generation: 1host x N -> 2hosts x ceil(N/2) -> ... -> Nhhosts x 1
Unsatisfiable resreq	Session-level fingerprint cache: {select, rusage, user, queue, project}
Seqjob path	Trivial rewrap — single plan (1x1) through the same iterator interfaces

Chosen because: Lazy generation avoids memory pressure. Iterator composition matches the resreq type hierarchy naturally. Seqjob rewrap validates the abstraction on the simple case before applying it to the complex parjob case. The cache eliminates redundant work for batches of unsatisfiable jobs.

Decision Rationale

Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The two-iterator composition (plan -> host -> try_alloc) is a proven pattern: the plan iterator answers "how many hosts, how many slots each?", the host iterator answers "which specific hosts can fulfill this plan?", and the allocator commits or advances. This separation of concerns makes each layer independently testable.

Approach

Iterator-Driven Allocation:

PRIMARY leaf allocation pseudocode

int allocate_primary(sched, primary_resreq, ctx) {
    plan_iter = plan_iter_init(primary_resreq, sched);
    while ((plan = plan_iter->next(plan_iter)) != NULL) {
        host_iter = host_iter_init(plan, sched->workers);
        while ((worker = host_iter->next(host_iter)) != NULL) {
            if (try_alloc_on_host(worker, plan, ctx) > 0) {
                host_iter->fini(host_iter);
                plan_iter->fini(plan_iter);
                return ALLOC_SUCCESS;
            }
        }
        host_iter->fini(host_iter);
    }
    plan_iter->fini(plan_iter);
    return ALLOC_FAIL;
}

Who uses this composition

Allocator	maxTasks	Plan iterator	Host iterator	Behavior
seqjob_allocator	== 1	Trivial: single plan (1 host x 1 slot)	Full: filters hosts by select expression	Unchanged from today — first matching host wins
parjob_allocator	> 1	Full combinatorial: 1hostxN, 2hosts x ceil(N/2), ..., Nhosts x 1	Full: filters hosts with enough available slots	New — walks plans until one succeeds
Future allocators	any	Custom strategy (e.g., topology-aware, bin-packing)	Reuses same xce_host_iter_t	Pluggable via job_alloc_fn

Unsatisfiable resreq cache

// Per-session cache on xce_scheduler_t
// Key: fingerprint of {select, rusage, user, queue, project}
// Cleared at begin_session()
// Populated when a job's plan iterator exhausts without success
// Checked before starting allocation for each job — hit = immediate pend

What stays unchanged

• seqjob_allocator externally observable behavior — rewrap is internal
• COMPOUND and ALTER resreq allocation paths — dispatch is unchanged
• Select expression system — no changes
• Shared resource tracking — no changes
• Existing plugin framework — no changes

What's new

• xce_alloc_plan_iter_t — lazy plan iterator interface
• xce_host_iter_t — lazy host iterator interface
• parjob_allocator combinatorial plan generation
• Session-level unsatisfiable resreq fingerprint cache
• Seqjob rewrap onto iterator interfaces (trivial case, behavior unchanged)

Scope

In

• xce_alloc_plan_iter_t interface: init(resreq, sched), next() -> plan, fini()
• xce_host_iter_t interface: init(plan, workers), next() -> worker, fini()
• parjob_allocator combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) via plan iterator
• seqjob_allocator rewrap onto iterator interfaces (trivial 1-host/1-slot case, behavior unchanged)
• Session-level unsatisfiable resreq fingerprint cache (keyed on {select, rusage, user, queue, project})
• COMPOUND and ALTER resreq allocation paths unchanged — they compose over the PRIMARY leaf allocator
• Unit tests: plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence
• Integration test: end-to-end multi-slot parallel job allocation

Out

• Pluggable job picker interface (xce_job_picker_iface_t) — in scheduler-core
• builtin_fcfs picker — in scheduler-core
• Configurable picker plugin with per-queue override — in scheduler-core
• Pickers in general (Req 1, 5, 8 of scheduler-framework) — in scheduler-core
• Preemption (Req 6 of scheduler-framework) — in scheduler-preemption
• Core/driver separation (Req 10 of scheduler-framework) — in scheduler-core
• Queue vs Project section — in scheduler-core
• New job selection policies beyond what the picker interface enables — deferred
• Resreq template registry (named/shared resreq objects) — deferred
• Select expression language or evaluator changes
• GPU/topology/affinity-aware allocation — left pluggable via future allocator plugins
• Controller-side scheduling-state replication / commit-barrier wiring

Upstream / Downstream

Upstream (dependencies)

• scheduler-core — the plan and host iterators consume the core event interface (xce_core_event_t) for worker/resource state updates, but are independently unit-testable with constructed state
• Existing plugin framework (plugin.h, plugin.c) — NO changes needed
• Existing select expression evaluator (resreq.h) — NO changes needed
• Existing shared resource tracking (sharedInst/sharedInUse) — NO changes needed
• Existing resreq type hierarchy (PRIMARY/COMPOUND/ALTER) — NO changes needed

Downstream (future consumers)

• Affinity/GPU allocators — register as allocator plugins, implement custom plan iterators for topology-aware placement, reuse xce_host_iter_t
• Bin-packing allocator — custom plan iterator strategy
• Resreq template registry — the fingerprint cache's key can incorporate template identity when sharing is added

Existing Spec Touchpoints

Depends on

• scheduler-core — the iterators operate within the core's scheduling session, consuming worker and resource state pushed through the core event interface. The iterators do NOT depend on the core's driver or I/O layer — they are purely computational, operating on in-memory state.

Adjacent

• scheduler-preemption — preemption fires after allocation failure. The unsatisfiable resreq cache is populated after both allocation and preemption fail (or preemption is inapplicable). The iterator interfaces are not preemption-aware.
• ha-consistency-invariants — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.

Constraints

• C (gnu17) — no C++, no external libraries beyond existing (pthreads)
• Cancel-safe — plan_iter->next() and host_iter->next() must have pthread_cancel cancellation points
• Session isolation — iterator state and resreq cache are per-session, cleared on begin_session
• Backward compatible — existing seqjob allocation behavior unchanged. COMPOUND/ALTER dispatch unchanged. New functionality is additive.
• Memory — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M x N candidate evaluations (acceptably bounded).
• No controller dependency — iterators are purely computational, operating on in-memory worker/resource state. They do not perform I/O or call into the controller.