myxce/.kiro/specs/scheduler-allocation/design.md

# Design Document

## Overview

**Purpose**: This feature turns the scheduler's allocation layer from a partially-stubbed, one-shot mechanism into a real, retry-capable, combinatorial allocator, and adds a session-scoped cache so batches of identically-unsatisfiable jobs don't each re-pay full allocation cost.

**Users**: Cluster operators submitting parallel (`maxTasks > 1`) jobs, who today get silent, permanent allocation failure regardless of actual cluster capacity; and scheduler maintainers who today cannot add a new allocation strategy without reimplementing host collection/filtering from scratch.

**Impact**: `parjob_allocator.c`'s `my_alloc()` goes from an unconditional `return 0` stub to a real combinatorial allocator. `seqjob_allocator.c`'s `my_alloc()` is rewrapped onto the same iterator interfaces as a trivial case, with no change to its externally observable host-selection behavior. `scheduler.c` gains a session-scoped unsatisfiable-resreq cache wrapping the existing PRIMARY-resreq allocator dispatch.

### Goals
- `parjob_allocator` allocates real multi-host, multi-slot jobs via lazy combinatorial plan generation (Requirement 4).
- A reusable lazy plan-iterator (Requirement 2) and host-iterator (Requirement 3) interface, shared by every current and future allocator plugin.
- A session-scoped unsatisfiable-resreq fingerprint cache (Requirement 7) that skips full allocation for a job whose resreq this session already proved unsatisfiable.
- `seqjob_allocator`'s externally observable behavior is provably unchanged after rewrap (Requirement 9.1).

### Non-Goals
- Pluggable job picker, `builtin_fcfs`, per-queue picker override — **scheduler-core** (already implementation-complete).
- Preemption, victim selection, `XCE_ALLOC_PREEMPT`'s actual resource-borrowing behavior — **scheduler-preemption** (this spec only declares the `xce_alloc_mode_t` extension point it needs).
- Core/driver separation — **scheduler-core** (already implementation-complete).
- `span`/`same`/`affinity`-aware plan generation, topology/GPU-aware allocation, resreq template registry, select-expression changes — deferred; the iterator interfaces are shaped to accept these later without a signature change.
- Controller-side commit-barrier wiring (`ha-consistency-invariants`) — this spec produces allocation decisions the barrier will eventually consume, does not wire it.
- Fixing `sched_make_compound_allocation()`'s pre-existing missing-rollback-on-partial-failure defect (bug-1766) or the recurring schedobj-reconstruction-duplication pattern in `snapshot_load.c` (bug-1767) — both discovered during this spec's own review, both out of boundary, both logged for independent follow-up.

## Boundary Commitments

### This Spec Owns
- The `xce_alloc_plan_iter_t` interface and every concrete plan-iterator implementation (`parjob_allocator`'s combinatorial one, `seqjob_allocator`'s trivial one).
- The `xce_host_iter_t` interface and its one shared implementation, reused by every allocator.
- `parjob_allocator.c`'s `my_alloc()`/`parjob_prepare_allocation()` bodies — real implementations, replacing the stubs.
- `seqjob_allocator.c`'s `my_alloc()`/`seqjob_prepare_allocation()` bodies — rewrapped onto the shared iterator pattern, behavior-preserving.
- The session-scoped unsatisfiable-resreq cache: `xce_resreq_fingerprint_t`, `xce_cache_key_t`, `cache_key_ops` (`kv_key_ops_t`), and the `xce_scheduler_t.unsatisfiable_cache` field's full lifecycle (populate, lookup, clear).
- The `xce_alloc_mode_t` enum and `xce_sched_ctx_t.alloc_mode` field — declared here as an extension point; only `XCE_ALLOC_NORMAL` behavior is implemented.
- `rusage_hash` on `xce_primary_resreq_t` and `resreq_fp` on `xce_primary_schedobj_t`, and every construction site that must populate them.

### Out of Boundary
- `COMPOUND`/`ALTER` resreq dispatch (`sched_make_compound_allocation`, `sched_make_alter_allocation`) — calls into this spec's per-primary allocator, otherwise untouched (Requirement 9.2).
- The existing select-expression evaluator, shared-resource tracking (`sharedInst`/`sharedInUse`), and plugin framework registration mechanics — consumed, not modified.
- Preemption's actual victim-selection logic and the resource-borrowing semantics `XCE_ALLOC_PREEMPT` will eventually trigger — **scheduler-preemption**'s job.
- `sched_make_compound_allocation()`'s missing partial-failure rollback (bug-1766) and the `snapshot_load.c` schedobj-reconstruction duplication pattern (bug-1767) — both pre-existing, both logged, neither fixed here.
- `resreq_fp` population on the snapshot-restore path (`snapshot_load.c`) — confirmed cosmetic (affects only offline simulation sessions, `sim.c`, never `driver.c`); not a requirement of this spec (see `requirements.md` Requirement 7's informational note).

### Allowed Dependencies
- **scheduler-core** (implementation-complete) — core event interface pushing worker/resource state; this feature's iterators consume it but do not depend on the driver/I/O layer.
- **scheduler-dispatch-commit** (implementation-complete, bug-1624 fixed) — `resInUse`/`sharedInUse` totals this feature's `try_alloc_on_host()` reads must already be correct; this was the spec's own sequencing blocker, now resolved.
- Existing `plugin.h`/`plugin.c` plugin framework, `resreq.h`'s select-expression evaluator (`select_eval_ctx_t`), `xvec.c` (`xvec_op(..., XVEC_SUB_FLOAT)` for rollback), `kv.c`/`kv.h` (`kv_key_ops_t`, `kvtab_type`, `key_find`/`key_add`) — all consumed as-is, no changes to their own contracts.
- `xce_job_alloc_t`/`make_job_alloc()`/`xce_host_alloc_t` (`xce/types.h`, `xcealloc.c`) — already multi-host-capable; this feature is the first caller to exercise `nhosts > 1`.

### Revalidation Triggers
- Any signature change to `xce_alloc_plan_iter_t`, `xce_host_iter_t`, or `xce_alloc_plan_t` (`bucket_slots[]` shape) — every allocator plugin consuming them must be re-checked.
- Any change to `xce_sched_ctx_t`'s `nslots`/`nhosts`/`alloc_mode` fields — **scheduler-preemption** depends on `alloc_mode` existing exactly as declared here.
- Any change to `xce_cache_key_t`'s field set or `plugin_prepare_allocation()`'s fan-out semantics.
- `scheduler-preemption` reaching implementation — its victim-selection flow wraps this spec's `allocate_primary()`-shaped logic and consumes `xce_alloc_mode_t`; a mismatch here blocks that spec.

## Architecture

### Existing Architecture Analysis

- `sched_make_primary_alloc_()` (`scheduler.c:167-177`) is the existing choke point every `PRIMARY`-tagged job passes through: it dispatches to `sched_make_seqjob_allocation()`/`sched_make_parjob_allocation()` purely on `primary->maxTasks == 1`. This dispatch itself is **unchanged** by this spec — only what happens inside each allocator's `job_alloc_fn` changes.
- `plugin_prepare_allocation()` (`plugin.c:591-661`) unconditionally fans out to **every** registered plugin's `prepare_allocation_hook` before each host-satisfaction check, resetting `ctx->alloc_req.host`/`.shared` first. `seqjob_prepare_allocation()` currently has no `maxTasks` guard, so it already fires (harmlessly, since it's a no-op today) on every job regardless of allocator — this becomes load-bearing once `parjob_prepare_allocation()` stops being a no-op (Requirement 4.6).
- `xce_sched_ctx_t` (`sched/types.h`) already carries `nhosts`/`nslots` scratch fields, currently hardcoded to `1, 1` by `seqjob_prepare_allocation()`; this spec is the first to vary them per allocation attempt.
- `core_begin_session_internal()` (`core_session.c:152-168`) already resets other shared, per-session state (`core_reset_shared_resval()`, `core_reset_workers()`) with **zero explicit locking**, relying on `xce_core_begin_session()`'s own documented ordering guarantee: all begin-session reset work completes strictly before `xce_core_signal_go()` unblocks the picker/allocator threads. This spec's cache clear reuses that exact placement and safety argument rather than introducing new synchronization.
- `xce_job_alloc_t`/`make_job_alloc()` (`xce/types.h`, `xcealloc.c`) already generalize past one host (`nhosts` + `xce_host_alloc_t**`), but only `seqjob_allocator` exercises them today, always with `nhosts=1` via the variadic `make_host_allocs(1, halloc)`. This spec is the first caller needing a runtime-sized host array, which the existing variadic helper cannot provide.

**Project Invariant Cross-Check** (per `design-review-gate.md`): `src/scheduler/CLAUDE.md`'s core-isolation invariant (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c` carry zero controller I/O, mechanically enforced by `test_core_isolation.c`) was checked against every new/modified piece of this design. All new code (`xce_alloc_plan_iter_t`/`xce_host_iter_t` implementations, the cache, both allocator plugins) is pure in-memory computation — no I/O, no blocking calls — so it does not violate the invariant regardless of which file it lands in. The one touch inside the isolated set (`core_begin_session_internal()`'s cache clear, `core_session.c`) is a plain kvtab clear, the same shape as the two existing calls already there.

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    Allocator[sched_make_primary_alloc_] --> CacheCheck{Cache lookup}
    CacheCheck -->|hit| Pend[Pend job immediately]
    CacheCheck -->|miss| Dispatch{maxTasks}
    Dispatch -->|equals 1| Seqjob[seqjob_allocator my_alloc]
    Dispatch -->|greater than 1| Parjob[parjob_allocator my_alloc]
    Seqjob --> PlanIterTrivial[Trivial plan iterator one bucket]
    Parjob --> PlanIterCombo[Combinatorial plan iterator]
    PlanIterTrivial --> HostIter[Shared host_iter]
    PlanIterCombo --> HostIter
    HostIter --> TryAlloc[try_alloc_on_host]
    TryAlloc -->|success| Commit[Commit job_alloc]
    TryAlloc -->|all plans exhausted| CacheInsert[Insert fingerprint into cache]
    CacheInsert --> Pend
```

**Architecture Integration**:
- **Selected pattern**: two-layer lazy iterator composition (plan → host) at the `PRIMARY` leaf, unchanged `COMPOUND`/`ALTER` composition above it — see `requirements.md`'s "Composition Algorithm" for the full pseudocode (bucket-loop, host-exclusivity, rollback) this design does not restate.
- **Domain/feature boundaries**: plan generation is allocator-owned (each `job_alloc_fn` builds its own plan-iterator instance); host iteration is shared, reused infrastructure; the cache wraps the dispatch choke point, not either allocator individually.
- **Existing patterns preserved**: `sched_ctx_init`/`sched_ctx_fini`'s stack-allocate-then-init-in-place convention (extended to the new iterators); `plugin_prepare_allocation()`'s fan-out (extended, not replaced); `core_begin_session_internal()`'s lock-free per-session reset pattern (extended to the cache).
- **New components rationale**: `xce_host_iter_t` is new because host collection was previously eager and allocator-specific (`sched_collect_workers()` + `xlist_find()`, inlined in `seqjob_allocator.c`); the cache is new because no unsatisfiable-resreq memoization existed at all.
- **Steering compliance**: no new controller I/O (`ha-async-io.md`'s thread-role contract); no new persistence format, and the one persistence-adjacent question (should the cache or `resreq_fp` be snapshot-captured) was resolved as "no" with a stated reason (`persistence.md`-style completeness review, applied inline in `requirements.md`'s Requirement 7 notes).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Language | C (gnu17) | All new code | No C++, no new external libraries |
| Hashing/KV | `src/lib/kv.c` (`kvtab_type`, `kv_key_ops_t`) | Backs the unsatisfiable-resreq cache | Existing infra; `cache_key_ops` is a new, third `kv_key_ops_t` instance alongside `int_key_ops`/`string_key_ops` |
| Arithmetic primitives | `src/lib/xvec.c` (`xvec_op`, `XVEC_SUB_FLOAT`) | Rollback's resource-debit reversal | Existing primitive, first caller to use the subtract op for this purpose |
| Concurrency | `src/lib/mylock.c`-style implicit ordering (no new locks) | Cache safety | Reuses `xce_core_begin_session()`/`xce_core_signal_go()`'s existing happens-before guarantee — no new mutex introduced |

## File Structure Plan

### Directory Structure
```
src/scheduler/
├── include/sched/
│   ├── alloc_iter.h        # NEW — xce_alloc_plan_t, xce_alloc_plan_iter_t,
│   │                       #   xce_host_iter_t interface declarations (Req 2, 3)
│   ├── alloc_cache.h       # NEW — xce_cache_key_t, xce_resreq_fingerprint_t,
│   │                       #   cache_key_ops declaration (Req 7)
│   └── types.h             # MODIFIED — xce_sched_ctx_t gains alloc_mode
│                           #   (Req 9.5); xce_primary_schedobj_t gains
│                           #   resreq_fp (Req 7)
├── src/
│   ├── host_iter.c          # NEW — the one shared xce_host_iter_t
│   │                       #   implementation (Req 3), reused by both
│   │                       #   allocators
│   ├── alloc_cache.c        # NEW — cache_key_hash_/cmp_/assign_/destroy_,
│   │                       #   cache lookup/insert/clear helpers (Req 7)
│   ├── scheduler.c          # MODIFIED — sched_make_primary_alloc_() wraps
│   │                       #   allocator dispatch with cache lookup/insert;
│   │                       #   xce_ctx_uncommit_host() rollback helper (in
│   │                       #   resource.c, see below) called from here
│   ├── resource.c           # MODIFIED — new sched_ctx_uncommit_host()
│   │                       #   helper wired to xvec_op(..., XVEC_SUB_FLOAT)
│   ├── core_session.c       # MODIFIED — core_begin_session_internal()
│   │                       #   clears the cache (Req 7.2)
│   ├── core_events.c        # MODIFIED — core_build_schedobj()/
│   │                       #   core_build_primary_schedobj() compute
│   │                       #   resreq_fp on ingest (Req 7.1)
│   ├── schedobj.c           # MODIFIED — sched_ctx_init() gains nothing new;
│   │                       #   referenced for the stack-allocate-then-init
│   │                       #   convention new iterators follow
│   └── plugins/
│       ├── parjob_allocator.c   # MODIFIED — real my_alloc() (combinatorial
│       │                       #   plan iterator + bucket loop over shared
│       │                       #   host_iter.c); parjob_prepare_allocation()
│       │                       #   fixed to use ctx->nslots (Req 4)
│       └── seqjob_allocator.c   # MODIFIED — my_alloc() rewrapped onto the
│                               #   same bucket-loop pattern as a trivial
│                               #   1-bucket case; seqjob_prepare_allocation()
│                               #   gains maxTasks==1 guard (Req 9.1, 4.6)
├── include/
│   └── scheduler.h          # MODIFIED — xce_scheduler_t gains
│                           #   unsatisfiable_cache field (Req 7.1)
└── testsuite/
    ├── test_alloc_plan_iter.c            # NEW (Req 2)
    ├── test_host_iter.c                  # NEW (Req 3)
    ├── test_alloc_cache.c                # NEW (Req 7)
    ├── test_parjob_combinatorial.c       # NEW (Req 4)
    ├── test_seqjob_rewrap_equivalence.c  # NEW (Req 9.1)
    ├── test_parjob_e2e_multislot.c       # NEW — integration (Req 4, brief.md scope)
    └── meson.build                       # MODIFIED — register the 6 new executables

src/include/
└── resreq.h                 # MODIFIED — xce_primary_resreq_t gains
                              #   rusage_hash (Req 7.1)
src/lib/
├── resreq.c                 # MODIFIED — parse_resreq_primary() computes
│                             #   rusage_hash (Req 7.1)
└── xcealloc.c                # MODIFIED — new merge helper building a
                              #   runtime-sized xce_host_alloc_t** array
                              #   directly (make_host_allocs()'s variadic
                              #   signature cannot take a runtime count),
                              #   used by parjob's commit step (Req 4.3)
```

### Modified Files
See inline annotations above — each modified file's change is scoped to exactly the requirement(s) noted; no file listed above is touched for any other reason.

## System Flows

### Combinatorial Allocation with Cache (sequence)

```mermaid
sequenceDiagram
    participant D as sched_make_primary_alloc_
    participant C as unsatisfiable_cache
    participant P as parjob_allocator
    participant PI as plan_iter
    participant HI as host_iter
    D->>C: lookup(resreq_fp)
    C-->>D: miss
    D->>P: my_alloc(ctx)
    P->>PI: init + next (plan 1)
    loop each bucket in plan
        P->>HI: init fresh, next until match or exhausted
        HI-->>P: worker or none
    end
    alt all buckets satisfied
        P-->>D: success, commit
    else a bucket exhausted
        P->>P: rollback claimed hosts
        P->>PI: next (plan 2)
        Note over P,PI: repeat until a plan succeeds or plan_iter exhausted
    end
    P-->>D: ALLOC_FAIL (all plans exhausted)
    D->>C: insert(resreq_fp)
    D-->>D: pend job
```

Full per-bucket accumulation/rollback logic (host-exclusivity via `is_claimed()`, fresh `host_iter` per bucket, `ctx->uncomitted` rollback scope) is specified in `requirements.md`'s "Composition Algorithm" — not restated here; this diagram shows only the cross-file call sequence design.md needs to fix component ownership.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 2.1–2.6 | Lazy plan iterator: interface, one-plan-per-call, ordering, exhaustion signal, cleanup, cancel-safety | `xce_alloc_plan_iter_t` (per-allocator impl) | `init`/`next`/`fini` | Combinatorial Allocation sequence |
| 3.1–3.6 | Lazy host iterator: interface, filtering, exhaustion, composition with plan iterator, cleanup, cancel-safety | `host_iter.c` (shared) | `init`/`next`/`fini` | Combinatorial Allocation sequence |
| 4.1–4.7 | Parjob combinatorial allocation: plan generation, per-bucket distinct-host accumulation, commit/merge, stub-return preserved on exhaustion, no unconditional stub, `ctx->nslots` resource-want plumbing, shared-resource check unchanged | `parjob_allocator`, `alloc_cache.c` (fingerprint only, not the check itself) | `job_alloc_fn`, `prepare_allocation_hook` | Combinatorial Allocation sequence |
| 7.1–7.5 | Unsatisfiable-resreq cache: key/fingerprint shape and O(1) lookup, begin-session clear placement, populate-on-exhaustion, lookup-before-allocation, pend-reason consistency | `alloc_cache.c`, `core_session.c`, `core_events.c` | `cache_key_ops` (`kv_key_ops_t`), `key_find`/`key_add` | Combinatorial Allocation sequence |
| 9.1–9.5 | No regression: seqjob behavior unchanged, COMPOUND/ALTER unchanged, existing unit tests pass, HA soak clean, `xce_alloc_mode_t` extension point | `seqjob_allocator`, `xce_sched_ctx_t.alloc_mode` | `job_alloc_fn` (seqjob), `xce_alloc_mode_t` enum | Combinatorial Allocation sequence (seqjob path = 1-bucket case) |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| `xce_alloc_plan_iter_t` (interface) | Scheduler / Allocation | Lazy per-allocator plan generation | 2.1–2.6 | `xce_primary_resreq_t` (P0) | Service |
| `host_iter.c` | Scheduler / Allocation | Shared lazy host-candidate filtering | 3.1–3.6 | select-expression evaluator (P0), `resource.c` (P0) | Service |
| `parjob_allocator` | Scheduler / Plugin | Real combinatorial multi-host allocation | 4.1–4.7 | `xce_alloc_plan_iter_t` (P0), `host_iter.c` (P0), `resource.c` (P0) | Service |
| `seqjob_allocator` | Scheduler / Plugin | Trivial single-host allocation, behavior-preserving rewrap | 9.1, 4.6 | `xce_alloc_plan_iter_t` (P0), `host_iter.c` (P0) | Service |
| `alloc_cache.c` | Scheduler / Allocation | Session-scoped unsatisfiable-resreq memoization | 7.1–7.5 | `kv.c` (P0), `core_session.c` (P0) | Service, State |
| `xce_alloc_mode_t` extension point | Scheduler / Allocation | Forward-compatible resource-pool-mode surface | 9.5 | none (v1: `XCE_ALLOC_NORMAL` only) | State |

### Scheduler / Allocation

#### `xce_alloc_plan_iter_t`

| Field | Detail |
|-------|--------|
| Intent | Lazy, cancel-safe generation of candidate `(bucket_slots[])` partitions, most-preferred first, one per allocator |
| Requirements | 2.1, 2.2, 2.3, 2.4, 2.5, 2.6 |

**Responsibilities & Constraints**
- Each concrete implementation (parjob's combinatorial, seqjob's trivial) owns its own plan-shape and partition policy; the interface itself is policy-agnostic.
- v1's combinatorial default (ascending host-count, canonical balanced ceil/floor partition, descending bucket order) is fully specified in `requirements.md`'s "Data Structure Changes Summary" — a pure function of `maxTasks`, never inspecting live worker state.
- Stack-allocated by the caller, initialized in place (`iter.init(&iter, resreq, sched)`) — never a heap-allocating factory (matches `sched_ctx_init(&ctx, ...)`'s existing convention).

**Dependencies**
- Inbound: `parjob_allocator`, `seqjob_allocator` (P0, sole consumers)
- Outbound: `xce_primary_resreq_t` (P0, read-only — `minTasks`/`maxTasks` used in v1; `span`/`same`/`affinity` read by future allocators without an interface change)

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
struct xce_alloc_plan_iter_s {
    int (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq, xce_scheduler_t* sched);
    xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter);
    void (*fini)(xce_alloc_plan_iter_t* iter);
    void* private;
};
```
- Preconditions: `resreq != NULL`; `init()` called exactly once before any `next()`.
- Postconditions: `next()` returns candidate plans strictly in most-preferred-first order until exhaustion (stable `NULL` thereafter); `fini()` releases all iterator-owned resources.
- Invariants: the pointer `next()` returns is valid only until the following `next()` or `fini()` call; `next()` calls `pthread_testcancel()` (or equivalent) at least once per invocation (Req 2.6).

#### `host_iter.c`

| Field | Detail |
|-------|--------|
| Intent | The one shared `xce_host_iter_t` implementation, filtering candidate hosts by select expression + resource availability for a given plan |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 |

**Responsibilities & Constraints**
- One implementation, reused by every allocator — no allocator-specific host-filtering logic anywhere else.
- Scoped **per plan bucket**, not per plan: a multi-bucket plan requires a fresh `xce_host_iter_t` instance for each bucket (`init()` called once per bucket), never one instance shared across a plan's buckets — see `requirements.md`'s Requirement 3.3/3.4 for why a shared, forward-only cursor is an incorrect implementation (it silently drops a host an earlier, larger bucket rejected even when it exactly satisfies a later, smaller one).
- Host exclusivity **across** buckets of the same plan (`is_claimed()`) is the caller's responsibility, not this component's — this component only ever answers "does this select expression + availability filter match," with no memory of prior buckets.

**Dependencies**
- Inbound: `parjob_allocator`, `seqjob_allocator` (P0)
- Outbound: existing select-expression evaluator (`resreq.h`, `select_eval_ctx_t`) (P0, unchanged); `resource.c`'s `sched_check_host_satisfy()` (P0, unchanged)

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
/* Correction (found during task 1.1 implementation): next() returns
 * xce_sched_worker_t* (sched/types.h -- resvals/resInUse/host_eval_ctx),
 * not xce_worker_t* (xce/types.h -- the wire/config type make_sched_worker()
 * converts FROM). requirements.md's own Composition Algorithm pseudocode
 * already used xce_sched_worker_t*; this struct declaration was out of
 * sync with it. */
struct xce_host_iter_s {
    int (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan, var workers);
    xce_sched_worker_t* (*next)(xce_host_iter_t* iter);
    void (*fini)(xce_host_iter_t* iter);
    void* private;
};
```
- Preconditions: `plan != NULL`; a fresh instance per bucket (see above).
- Postconditions: `next()` returns `NULL` once no further candidate exists **for this instance's bucket search**; a fresh instance for the next bucket is unaffected.
- Invariants: independent lifecycle from the owning `xce_alloc_plan_iter_t` — `fini()` on one never affects the other; `next()` calls `pthread_testcancel()` (or equivalent) at least once per invocation (Req 3.6).

**Implementation Notes**
- Integration: constructed fresh inside the per-bucket loop of both `parjob_allocator`'s and `seqjob_allocator`'s `my_alloc()` — see `requirements.md`'s "Composition Algorithm" for the exact loop shape.
- Validation: `test_host_iter.c` must include the exact adversarial case that motivated the fresh-per-bucket requirement (a host too small for an earlier, larger bucket remaining eligible for a later, smaller one).
- Risks: none new — this is the shared implementation of behavior `seqjob_allocator` already has today (`sched_collect_workers()` + `xlist_find()`), generalized.

#### `alloc_cache.c`

| Field | Detail |
|-------|--------|
| Intent | Session-scoped memoization of resreqs this session already proved unsatisfiable, keyed by a composite fingerprint |
| Requirements | 7.1, 7.2, 7.3, 7.4, 7.5 |

**Responsibilities & Constraints**
- Owns `xce_cache_key_t`, `xce_resreq_fingerprint_t`, and `cache_key_ops` (a third `kv_key_ops_t` instance alongside `int_key_ops`/`string_key_ops`). Full field-by-field hash/cmp/assign/destroy implementations are in `requirements.md`'s "New types (Req 7)" section — corrected during design-phase discovery (the `.cmp` return-value polarity was inverted in an earlier draft; fixed to the real three-way, 0-means-equal convention `kv.c`'s `int_key_ops.cmp` establishes).
- Lookup/insert happen exclusively on the allocator dispatch path (`sched_make_primary_alloc_()`, wrapping either allocator's call) — never inside `parjob_allocator.c`/`seqjob_allocator.c` themselves, so both allocators benefit identically without duplicated cache-handling code.
- Clear happens exactly once, inside `core_begin_session_internal()` (`core_session.c`), alongside the existing `core_reset_shared_resval()`/`core_reset_workers()` calls — no new lock, reusing `xce_core_begin_session()`'s existing happens-before guarantee (see Existing Architecture Analysis above).
- Not part of the snapshot/replay wire format — pure in-memory, unconditionally cleared at the next `begin_session` regardless of what a loaded snapshot contained.

**Dependencies**
- Inbound: `scheduler.c`'s `sched_make_primary_alloc_()` (P0, sole caller of lookup/insert); `core_session.c`'s `core_begin_session_internal()` (P0, sole caller of clear)
- Outbound: `kv.c`'s `kvtab_type`/`key_find`/`key_add` (P0); `core_events.c`'s `core_build_schedobj()` (P0, populates `resreq_fp` on ingest); `resreq.c`'s `parse_resreq_primary()` (P0, populates `rusage_hash`)

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x]

##### Service Interface
```c
/* Correction (kiro-validate-design pass): a bare hit/miss sentinel cannot
 * satisfy Req 7 AC 5 ("same pend reason category" on a hit) -- the value
 * stored per key is the original miss's reason CATEGORY (an int code,
 * e.g. xce_job_preason_t.main_reason), not a placeholder. Not the full
 * detailed reason object: that carries job/host-specific breakdown
 * (xce_host_reason_t*/xce_res_reason_t* arrays) that wouldn't transfer
 * meaningfully to a different job sharing the same fingerprint. */
int cache_lookup(xce_scheduler_t* sched, xce_sched_obj_t* obj, int* out_reason_category);
                                    /* 1 = hit (out_reason_category set), 0 = miss */
int cache_insert(xce_scheduler_t* sched, xce_sched_obj_t* obj, int reason_category);
                                    /* 0 = ok, <0 = error */
void cache_clear(xce_scheduler_t* sched);   /* called from core_begin_session_internal() */
```
- Preconditions: `obj->tag == XCE_RESREQ_PRIMARY`; `obj->u.primary->resreq_fp` already populated (job-ingest time); `cache_insert()`'s `reason_category` is the exhausted plan iterator's own final pend reason's `main_reason`.
- Postconditions: `cache_lookup()` returning a hit means the caller pends the job using `*out_reason_category` (Req 7.5) and never invokes the plan/host iterators or preemption for this attempt.
- Invariants: cache is cleared exactly once per scheduling session, strictly before the picker/allocator threads touch it for that session.

##### State Management
- State model: one `kvtab_t` per `xce_scheduler_t`, keyed by `xce_cache_key_t` (composing `xce_resreq_fingerprint_t` + `{user, queue, project}` xstring pointers), value is the reason-category `int` boxed as `(void*)(intptr_t)category` (no heap allocation needed for a single int; `NULL` datakiller, matching `int_key_ops`'s own value-handling convention).
- Persistence & consistency: none — explicitly not snapshot-captured (see Boundary Commitments).
- Concurrency strategy: no explicit lock; safety derives from `xce_core_begin_session()`'s documented ordering guarantee plus lookup/insert being confined to the single-threaded allocator path.

**Implementation Notes**
- Integration: `cache_key_cmp_()`'s three-way-comparator fix (this design pass) must land alongside the rest of `alloc_cache.c` — a functionally dead cache from day one would not fail any test that doesn't specifically assert a hit, so `test_alloc_cache.c` must include a same-key-twice hit assertion, not only a miss-then-insert-then-lookup round-trip that could pass even with the inverted polarity if written carelessly.
- Validation: `test_alloc_cache.c` must also assert the reason-category round-trip (insert with category X, hit later returns X) — the value-model fix from this validation pass.
- Risks: `.cmp` polarity bug already fixed (see `research.md`); the reason-category round-trip is the one new surface this fix adds, worth its own explicit test rather than assuming the hit/miss path alone proves it.

### Scheduler / Plugin

#### `parjob_allocator`

| Field | Detail |
|-------|--------|
| Intent | Real combinatorial multi-host, multi-slot job allocation |
| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7 |

**Responsibilities & Constraints**
- `my_alloc()` walks its own combinatorial plan iterator; for each plan, loops buckets, each bucket getting a fresh `host_iter.c` instance (host-exclusivity via `is_claimed()`, not iterator sharing).
- The per-plan `claimed[]` accumulator (tracking which host was claimed for each bucket, for both the merge step and rollback) is sized by `primary_resreq->maxTasks` — the true, tight upper bound on bucket count (each bucket needs ≥1 slot, so `nbuckets` can never exceed it) — not a fixed compile-time constant an implementer would otherwise have to guess (`kiro-validate-design` finding, corrected in `requirements.md`'s pseudocode).
- On full-plan success: merges claimed hosts into one `xce_job_alloc_t` via the new `xcealloc.c` helper (runtime-sized array, not `make_host_allocs()`'s variadic signature).
- On per-bucket failure: rolls back via `resource.c`'s new `sched_ctx_uncommit_host()`, scoped to `ctx->uncomitted` only (never touches real `resInUse`).
- On full exhaustion: returns `0`, identical to today's stub return value, so `sched_make_primary_alloc_()`'s pend-reason handling is unaffected (Req 4.4).
- `parjob_prepare_allocation()` sets `ctx->alloc_req` via `calc_wanted_host_resvals(..., ctx->nhosts, ctx->nslots)`, reading the current bucket's slot count from `ctx->nslots` (set by the bucket loop before each host try) rather than a hardcoded constant.
- Shared-resource satisfaction is checked once per primary resreq, independent of plan/bucket shape — unchanged from today's mechanism (Req 4.7).

**Dependencies**
- Inbound: `sched_make_primary_alloc_()` (P0)
- Outbound: `xce_alloc_plan_iter_t` (own combinatorial impl, P0), `host_iter.c` (P0), `resource.c` (`sched_check_host_satisfy`, `sched_ctx_uncommit_host`, P0), `xcealloc.c` (merge helper, P0)

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
static int my_alloc(xce_sched_plugin_t* plugin, xce_sched_ctx_t* ctx);              /* job_alloc_fn */
static int parjob_prepare_allocation(void* arg, xce_sched_ctx_t* ctx);              /* prepare_allocation_hook */
```
- Preconditions: `ctx->primary->maxTasks > 1` (routed here by `sched_make_primary_alloc_()`).
- Postconditions: return `>0` and `ctx->alloc` populated on success; return `0` with `ctx->reason` populated on exhaustion — same contract shape `seqjob_allocator`'s `my_alloc()` already has.
- Invariants: never claims the same host twice within one plan attempt; never leaves a phantom `ctx->uncomitted` debit after a failed plan attempt.

**Implementation Notes**
- Integration: registration in the plugin framework (`register_parjob_allocator()`) is unchanged — only the two hook bodies change.
- Validation: `test_parjob_combinatorial.c` must cover the plan-order-matters worked example from `requirements.md` (descending bucket order succeeding where ascending would fail, identical host pool) and the fresh-per-bucket `host_iter` requirement.
- Risks: the `seqjob_prepare_allocation()` fan-out corruption this fix depends on (Req 4.6) — mitigated by landing both changes together, never one without the other (see `research.md` Risks).

#### `seqjob_allocator`

| Field | Detail |
|-------|--------|
| Intent | Trivial single-host allocation, rewrapped onto the shared iterator pattern with provably unchanged external behavior |
| Requirements | 9.1, 4.6 |

**Responsibilities & Constraints**
- `my_alloc()`'s bucket loop degenerates to exactly one iteration (`nbuckets == 1`, `bucket_slots == [1]`), using the same shared `host_iter.c` and `is_claimed()`/rollback machinery `parjob_allocator` uses — the abstraction validated on the simple case before the complex one.
- `seqjob_prepare_allocation()` gains a `primary->maxTasks == 1` guard — a no-op for seqjob's own behavior (it never sees `maxTasks > 1` jobs), required only so it stops corrupting `parjob_allocator`'s fixed prepare hook via `plugin_prepare_allocation()`'s fan-out.

**Dependencies**
- Inbound: `sched_make_primary_alloc_()` (P0)
- Outbound: `xce_alloc_plan_iter_t` (own trivial impl, P0), `host_iter.c` (P0)

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
Same shape as `parjob_allocator`'s (`my_alloc`/`prepare_allocation_hook`) — see above; not restated.
- Preconditions: `ctx->primary->maxTasks == 1`.
- Postconditions: identical host selection to today's `sched_collect_workers()` + `xlist_find()` implementation, given identical worker/resource state (Req 9.1) — this is the requirement the rewrap must prove, not just aim for.
- Invariants: unchanged.

**Implementation Notes**
- Integration: registration (`register_seqjob_allocator()`) unchanged.
- Validation: `test_seqjob_rewrap_equivalence.c` must construct fixture states where the rewrap could plausibly diverge (multiple candidate hosts, at least one failing the select filter) and assert identical outcomes pre/post rewrap — a test that only exercises the single-obvious-host case would not actually prove equivalence.
- Risks: none beyond the shared `seqjob_prepare_allocation()` guard dependency already noted under `parjob_allocator`.

### Scheduler / Allocation (extension point)

#### `xce_alloc_mode_t`

| Field | Detail |
|-------|--------|
| Intent | Forward-compatible resource-pool-mode field on `xce_sched_ctx_t`, declared now so `scheduler-preemption` (and later modes) don't need to modify this spec's struct later |
| Requirements | 9.5 |

**Responsibilities & Constraints**
- v1 declares `XCE_ALLOC_NORMAL = 0` and `XCE_ALLOC_PREEMPT`; only `XCE_ALLOC_NORMAL` behavior is implemented — no branch anywhere in this spec's `try_alloc_on_host()`/availability-check call sites reads this field for any purpose beyond existing.
- Adding further enum values or the actual borrowing behavior `XCE_ALLOC_PREEMPT` implies is explicitly `scheduler-preemption`'s job, not this spec's.

**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x]

##### State Management
- State model: a single `xce_alloc_mode_t alloc_mode` field on `xce_sched_ctx_t`, defaulting to `XCE_ALLOC_NORMAL` (zero-value, matches `sched_ctx_init()`'s existing `memset(ctx, 0, sizeof(*ctx))`).
- Persistence & consistency: none — `xce_sched_ctx_t` is per-allocation-attempt scratch state, never persisted (see `requirements.md`'s "Rollback scope" discussion of the same struct).
- Concurrency strategy: none needed — single-threaded, per-attempt scratch.

## Data Models

### Domain Model
- **`xce_alloc_plan_t`**: a partition of a job's `maxTasks` into per-host slot buckets (`nbuckets`, `bucket_slots[]`) — the domain concept this whole feature revolves around. Not persisted; lives only for the duration of one plan attempt.
- **`xce_resreq_fingerprint_t`** / **`xce_cache_key_t`**: value objects, immutable once computed, fitting in a single cache line — see `requirements.md`'s exact field layout, not restated here.

### Logical Data Model
No new persistent entities. The only durable-adjacent addition is `rusage_hash` on `xce_primary_resreq_t` (in-memory, process-lifetime, set once at parse time) and `resreq_fp` on `xce_primary_schedobj_t` (same lifetime class) — both scratch/derived fields, never wire-serialized, confirmed in `requirements.md`'s Requirement 7 notes and this spec's own Boundary Commitments.

### Data Contracts & Integration
None — this feature introduces no new wire format, no new capnp schema, and (per the explicit design decision in `requirements.md`) no snapshot/replay format changes for the cache.

## Error Handling

### Error Strategy
Allocation failure is not an error condition in this domain — it is the expected, common "no plan/host satisfies this resreq right now" outcome, communicated via return value (`0`) and `ctx->reason`, exactly as today. Genuine errors (OOM, iterator `init()` failure) propagate as negative return codes and are treated the same as "no candidate for this bucket" at the allocation-decision level — a distinction `requirements.md`'s Composition Algorithm draws explicitly (host-iterator init failure treated as an infrastructure error, not a resource-availability determination).

### Error Categories and Responses
- **Resource unavailable** (expected, high-frequency): plan exhausted → pend job with reason; cache the fingerprint so subsequent identically-shaped jobs skip the cost (Req 7).
- **Iterator init failure** (rare, e.g. OOM): treated as "this bucket/plan has no candidates," not a distinct error path — avoids a second failure taxonomy the caller would need to handle.
- **Fan-out corruption** (latent, closed by this design): `seqjob_prepare_allocation()`'s missing `maxTasks` guard, closed by Req 4.6 — see Existing Architecture Analysis.

### Monitoring
No new logging/metrics surface beyond what `ctx->reason`/`xce_pendreason_t` already provides — this feature populates the same pend-reason machinery every allocator already uses, adding no new observability contract.

## Testing Strategy

- **Unit — plan iterator** (`test_alloc_plan_iter.c`, Req 2): one-plan-per-`next()`-call; most-preferred-first ordering for `maxTasks` values with and without a clean partition (e.g. `N=7` → `[3,2,2]` for `k=3`); stable exhaustion signal; `fini()` cleanup; `pthread_testcancel()` present.
- **Unit — host iterator** (`test_host_iter.c`, Req 3): select-expression + availability filtering unchanged from the existing evaluator; **the fresh-per-bucket adversarial case** (host too small for an earlier bucket remains eligible for a later, smaller one) — this is the single most important test in this spec, directly proving the bug found during design discussion is closed.
- **Unit — cache** (`test_alloc_cache.c`, Req 7): hit on identical fingerprint+identity; miss on any differing field; **same-key-twice hit** (not just insert-then-miss-different-key, which the inverted `.cmp` bug would still have passed); **reason-category round-trip** (insert with category X, later hit returns X, per Req 7.5 — the `kiro-validate-design` fix to the cache's value model); clear-at-begin-session.
- **Unit — parjob combinatorial** (`test_parjob_combinatorial.c`, Req 4): multi-bucket plan success and partial-failure-then-rollback; descending-vs-ascending bucket order worked example; `ctx->nslots` correctly varies the resource want per bucket.
- **Unit — seqjob rewrap equivalence** (`test_seqjob_rewrap_equivalence.c`, Req 9.1): identical host selection pre/post rewrap across fixture states including a failing-select-filter host, not just the single-obvious-host trivial case.
- **Integration — end-to-end multi-slot allocation** (`test_parjob_e2e_multislot.c`, brief.md scope): a real multi-host parallel job dispatched through the full `sched_make_primary_alloc_()` → `parjob_allocator` → commit path, asserting the final `xce_job_alloc_t.nhosts > 1` and correct per-host slot counts.
- **Regression — existing scheduler suite** (Req 9.3): full `scheduler` meson suite must pass unmodified in assertions, adjusted only where a test directly exercised the eager-one-shot or stub-return behavior this feature replaces.
- **HA soak** (Req 9.4): sequential and chaotic `/xce-test-ha` regression, confirming zero new job loss/divergence/liveness-stall attributable to this feature.

## Supporting References
- Full pseudocode (composition algorithm, cache key hash/cmp/assign/destroy, v1 plan-shape/partition defaults) lives in `requirements.md` — not duplicated here per this template's own deduplication guidance. `design.md`'s job is component/file ownership and contracts; `requirements.md` remains the algorithmic source of truth.
- `research.md` — discovery findings (the `.cmp` polarity fix, the core-isolation boundary check, the test-file convention) and the three most consequential design decisions with alternatives considered.