myxce/.kiro/specs/scheduler-allocation/brief.md

# Brief: scheduler-allocation

## Problem

The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: `sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: `parjob_allocator.c`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` -- any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()`) produces no allocation at all. Third, allocation is eager and one-shot: the sibling `seqjob_allocator.c`'s `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.

The first gap (hardcoded picker) is addressed in the sibling **scheduler-core** spec. This spec, **scheduler-allocation**, closes the other two: the parjob stub and one-shot allocation.

## Current State

### What exists (allocation surface)

- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources, collects workers via `sched_collect_workers()`, tries one host, returns first match. No retry with alternative plans.
- **Parjob allocator stub** (`parjob_allocator.c`): registered as a plugin but `my_alloc()` is empty (`return 0`). Any job with `maxTasks > 1` silently fails allocation.
- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds). The type hierarchy is well-defined but allocation at the PRIMARY leaf lacks iteration abstractions.
- **Select expression evaluator** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC. Already reusable for host filtering.
- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session.
- **Allocation dispatch** (`scheduler.c:sched_make_allocation()`, `sched_make_primary_alloc_()`): routes to seqjob or parjob allocator based on `maxTasks`.

### What's missing

- **Lazy plan generation**: no `xce_alloc_plan_iter_t` -- plans are not generated at all, let alone lazily
- **Lazy host iteration**: no `xce_host_iter_t` -- host collection is eager via `sched_collect_workers()`, materializing the full list before trying any
- **Parallel job allocation**: `parjob_allocator` is a stub. No combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1), no multi-slot host matching
- **Unsatisfiable resreq caching**: a batch of similarly-shaped unsatisfiable jobs each re-pays the full allocation cost within a session

## Desired Outcome

### Iterator-Driven Allocation Architecture

A lazy `xce_alloc_plan_iter_t` (candidate plans, each a partition of the job's task count into per-host slot buckets -- e.g. `[2,1]` for 3 tasks across 2 hosts, not a uniform `(nhosts, slots_per_host)` pair, which cannot express uneven splits -- most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session.

### Three-Layer Iterator Composition

```
ALTER     ->  OR-of-compounds: first compound that fully allocates wins (unchanged)
  └─ COMPOUND ->  AND-of-primaries: ALL primaries must allocate (unchanged)
       └─ PRIMARY  ->  plan_iter -> host_iter -> try_alloc  (NEW iterator layer)
```

The iterator composition operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today.

### Iterator Pipeline

```
xce_alloc_plan_iter_t    -- init(resreq) -> next() -> plan -> next() -> ... -> NULL -> fini()
       │ plan (a partition into per-host buckets, e.g. [2,1])
       ▼
for each bucket in plan.bucket_slots[]     -- NOT a 2-level pipeline: a plan
       │                                       with N>1 buckets needs a DISTINCT
       │ (fresh host_iter per bucket --      host per bucket, each found via
       │  see "Corrected below")             its own full host_iter scan
       ▼
xce_host_iter_t          -- init(plan, workers) -> next() -> worker -> ... -> NULL -> fini()
       │ worker
       ▼
try_alloc(worker, bucket_slots[i])  -- check slots, shared res, commit or advance
```

Three levels of iteration, not two: plan -> bucket/partition -> worker. A single-bucket plan (seqjob's trivial case) collapses the middle level to one iteration, which is why it's easy to under-specify this as a 2-level pipeline -- see "PRIMARY leaf allocation pseudocode" below for why the middle level must be a real loop with its own fresh `host_iter` per bucket, not folded into the plan/host levels.

### Key Properties

- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization.
- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points.
- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done.
- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf are unchanged from today.
- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host x 1 slot). This validates the abstraction before applying it to the complex parjob case.

## Alternatives Considered

### Approach A: Incremental Enhancement (rejected)

| Aspect | Detail |
|--------|--------|
| **Plan generation** | Pre-computed plan array. Eager generation for parjob. |
| **Host iteration** | No new abstraction. Simple for-loop over `sched_collect_workers()` result. |
| **Unsatisfiable resreq** | `xlist_t unsatisfiable_resreqs` in session state |

**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.

### Approach B: Iterator-Driven Allocation (chosen)

| Aspect | Detail |
|--------|--------|
| **Plan iterator** | `xce_alloc_plan_iter_t` — lazy, cancel-safe, generates one plan per `next()` call |
| **Host iterator** | `xce_host_iter_t` — lazy per-plan host filtering via existing select-expression evaluator |
| **Parallel jobs** | Combinatorial plan generation: 1host x N -> 2hosts x ceil(N/2) -> ... -> Nhhosts x 1 |
| **Unsatisfiable resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` |
| **Seqjob path** | Trivial rewrap — single plan (1x1) through the same iterator interfaces |

**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the resreq type hierarchy naturally. Seqjob rewrap validates the abstraction on the simple case before applying it to the complex parjob case. The cache eliminates redundant work for batches of unsatisfiable jobs.

### Decision Rationale

Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The two-iterator composition (plan -> host -> try_alloc) is a proven pattern: the plan iterator answers "how many hosts, how many slots each?", the host iterator answers "which specific hosts can fulfill this plan?", and the allocator commits or advances. This separation of concerns makes each layer independently testable.

## Approach

**Iterator-Driven Allocation**:

### PRIMARY leaf allocation pseudocode

**Corrected (this review) -- superseded from an earlier 2-level version of this pseudocode**, which tried one worker per plan and returned on the first success. That's only valid for a single-bucket plan (seqjob's trivial case); a plan with `nbuckets > 1` needs a distinct host per bucket, so the algorithm needs a third, explicit loop level. It also needs a *fresh* `host_iter` per bucket rather than one shared across the whole plan: `next()` is a forward-only cursor, so sharing it would permanently drop a host an earlier, larger bucket tried and rejected as too small -- even when that same host exactly satisfies a later, smaller bucket. Iterators are stack-allocated by the caller and initialized in place (matching this codebase's existing `sched_ctx_init(&ctx, ...)` convention), not returned by a heap-allocating factory. Full rationale (rollback scope, host exclusivity semantics, why descending bucket order matters for correctness not just speed): `requirements.md`'s "Composition Algorithm" section.

```c
int allocate_primary(sched, primary_resreq, ctx) {
    xce_alloc_plan_iter_t plan_iter_storage;
    xce_alloc_plan_iter_t* plan_iter = &plan_iter_storage;
    if (plan_iter->init(plan_iter, primary_resreq, sched) != 0) {
        return ALLOC_FAIL;
    }

    while ((plan = plan_iter->next(plan_iter)) != NULL) {
        // plan is a partition of maxTasks into per-host buckets, e.g. [2,1]
        // for 3 tasks across 2 hosts -- NOT a uniform (nhosts, slots_per_host).
        xce_sched_worker_t* claimed[primary_resreq->maxTasks];  // sized by maxTasks, the true upper bound on bucket count
        int nclaimed = 0;
        int plan_ok = 1;

        // Three levels of iteration: plan (above) -> bucket/partition (this
        // for) -> worker (host_iter, fresh per bucket, below).
        for (int b = 0; b < plan->nbuckets; b++) {
            xce_host_iter_t host_iter_storage;
            xce_host_iter_t* host_iter = &host_iter_storage;
            xce_sched_worker_t* w;
            int bucket_ok = 0;

            if (host_iter->init(host_iter, plan, sched->workers) == 0) {
                while ((w = host_iter->next(host_iter)) != NULL) {
                    if (is_claimed(claimed, nclaimed, w)) {
                        continue;
                    }
                    if (try_alloc_on_host(w, plan->bucket_slots[b], ctx) > 0) {
                        claimed[nclaimed++] = w;
                        bucket_ok = 1;
                        break;
                    }
                }
                host_iter->fini(host_iter);
            }

            if (!bucket_ok) {
                plan_ok = 0;
                break;  // this bucket has no satisfying host -- abandon the plan
            }
        }

        if (plan_ok) {
            commit_plan_allocation(claimed, nclaimed, ctx);
            plan_iter->fini(plan_iter);
            return ALLOC_SUCCESS;
        }

        rollback_plan_allocation(claimed, nclaimed, ctx);  // undo partial claims
    }
    plan_iter->fini(plan_iter);
    return ALLOC_FAIL;
}
```

### Who uses this composition

| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior |
|-----------|------------|---------------|---------------|----------|
| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host x 1 slot) | Full: filters hosts by select expression | Unchanged from today — first matching host wins |
| `parjob_allocator` | `> 1` | Full combinatorial: 1hostxN, 2hosts x ceil(N/2), ..., Nhosts x 1 | Full: filters hosts with enough available slots | New — walks plans until one succeeds |
| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` |

### Unsatisfiable resreq cache

```c
// Per-session cache on xce_scheduler_t
// Key: fingerprint of {select, rusage, user, queue, project}
// Cleared at begin_session()
// Populated when a job's plan iterator exhausts without success
// Checked before starting allocation for each job — hit = immediate pend
```

### What stays unchanged

- `seqjob_allocator` externally observable behavior — rewrap is internal
- COMPOUND and ALTER resreq allocation paths — dispatch is unchanged
- Select expression system — no changes
- Shared resource tracking — no changes
- Existing plugin framework — no changes

### What's new

- `xce_alloc_plan_iter_t` — lazy plan iterator interface
- `xce_host_iter_t` — lazy host iterator interface
- `parjob_allocator` combinatorial plan generation
- Session-level unsatisfiable resreq fingerprint cache
- Seqjob rewrap onto iterator interfaces (trivial case, behavior unchanged)

## Scope

### In

- `xce_alloc_plan_iter_t` interface: `init(resreq, sched)`, `next() -> plan`, `fini()`
- `xce_host_iter_t` interface: `init(plan, workers)`, `next() -> worker`, `fini()`
- `parjob_allocator` combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) via plan iterator
- `seqjob_allocator` rewrap onto iterator interfaces (trivial 1-host/1-slot case, behavior unchanged)
- Session-level unsatisfiable resreq fingerprint cache (keyed on `{select, rusage, user, queue, project}`)
- COMPOUND and ALTER resreq allocation paths unchanged — they compose over the PRIMARY leaf allocator
- Unit tests: plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence
- Integration test: end-to-end multi-slot parallel job allocation

### Out

- Pluggable job picker interface (`xce_job_picker_iface_t`) — in **scheduler-core**
- `builtin_fcfs` picker — in **scheduler-core**
- Configurable picker plugin with per-queue override — in **scheduler-core**
- Pickers in general (Req 1, 5, 8 of scheduler-framework) — in **scheduler-core**
- Preemption (Req 6 of scheduler-framework) — in **scheduler-preemption**
- Core/driver separation (Req 10 of scheduler-framework) — in **scheduler-core**
- Queue vs Project section — in **scheduler-core**
- New job selection policies beyond what the picker interface enables — deferred
- Resreq template registry (named/shared resreq objects) — deferred
- Select expression language or evaluator changes
- GPU/topology/affinity-aware allocation — left pluggable via future allocator plugins
- Controller-side scheduling-state replication / commit-barrier wiring

## Upstream / Downstream

### Upstream (dependencies)

- **scheduler-core** — the plan and host iterators consume the core event interface (`xce_core_event_t`) for worker/resource state updates, but are independently unit-testable with constructed state
- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed
- Existing select expression evaluator (`resreq.h`) — NO changes needed
- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed
- Existing resreq type hierarchy (PRIMARY/COMPOUND/ALTER) — NO changes needed

### Downstream (future consumers)

- **Affinity/GPU allocators** — register as allocator plugins, implement custom plan iterators for topology-aware placement, reuse `xce_host_iter_t`
- **Bin-packing allocator** — custom plan iterator strategy
- **Resreq template registry** — the fingerprint cache's key can incorporate template identity when sharing is added

## Existing Spec Touchpoints

### Depends on
- **scheduler-core** — the iterators operate within the core's scheduling session, consuming worker and resource state pushed through the core event interface. The iterators do NOT depend on the core's driver or I/O layer — they are purely computational, operating on in-memory state.

### Adjacent
- **scheduler-preemption** — preemption fires after allocation failure. The unsatisfiable resreq cache is populated after both allocation and preemption fail (or preemption is inapplicable). The iterator interfaces are not preemption-aware.
- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.

## Constraints

- **C (gnu17)** — no C++, no external libraries beyond existing (pthreads)
- **Cancel-safe** — `plan_iter->next()` and `host_iter->next()` must have `pthread_cancel` cancellation points
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`
- **Backward compatible** — existing seqjob allocation behavior unchanged. COMPOUND/ALTER dispatch unchanged. New functionality is additive.
- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M x N candidate evaluations (acceptably bounded).
- **No controller dependency** — iterators are purely computational, operating on in-memory worker/resource state. They do not perform I/O or call into the controller.