myxce/.kiro/specs/scheduler-allocation/design.md

Design Document

Overview

Purpose: This feature turns the scheduler's allocation layer from a partially-stubbed, one-shot mechanism into a real, retry-capable, combinatorial allocator, and adds a session-scoped cache so batches of identically-unsatisfiable jobs don't each re-pay full allocation cost.

Users: Cluster operators submitting parallel (maxTasks > 1) jobs, who today get silent, permanent allocation failure regardless of actual cluster capacity; and scheduler maintainers who today cannot add a new allocation strategy without reimplementing host collection/filtering from scratch.

Impact: parjob_allocator.c's my_alloc() goes from an unconditional return 0 stub to a real combinatorial allocator. seqjob_allocator.c's my_alloc() is rewrapped onto the same iterator interfaces as a trivial case, with no change to its externally observable host-selection behavior. scheduler.c gains a session-scoped unsatisfiable-resreq cache wrapping the existing PRIMARY-resreq allocator dispatch.

Goals

• parjob_allocator allocates real multi-host, multi-slot jobs via lazy combinatorial plan generation (Requirement 4).
• A reusable lazy plan-iterator (Requirement 2) and host-iterator (Requirement 3) interface, shared by every current and future allocator plugin.
• A session-scoped unsatisfiable-resreq fingerprint cache (Requirement 7) that skips full allocation for a job whose resreq this session already proved unsatisfiable.
• seqjob_allocator's externally observable behavior is provably unchanged after rewrap (Requirement 9.1).

Non-Goals

• Pluggable job picker, builtin_fcfs, per-queue picker override — scheduler-core (already implementation-complete).
• Preemption, victim selection, XCE_ALLOC_PREEMPT's actual resource-borrowing behavior — scheduler-preemption (this spec only declares the xce_alloc_mode_t extension point it needs).
• Core/driver separation — scheduler-core (already implementation-complete).
• span/same/affinity-aware plan generation, topology/GPU-aware allocation, resreq template registry, select-expression changes — deferred; the iterator interfaces are shaped to accept these later without a signature change.
• Controller-side commit-barrier wiring (ha-consistency-invariants) — this spec produces allocation decisions the barrier will eventually consume, does not wire it.
• Fixing sched_make_compound_allocation()'s pre-existing missing-rollback-on-partial-failure defect (bug-1766) or the recurring schedobj-reconstruction-duplication pattern in snapshot_load.c (bug-1767) — both discovered during this spec's own review, both out of boundary, both logged for independent follow-up.

Boundary Commitments

This Spec Owns

• The xce_alloc_plan_iter_t interface and every concrete plan-iterator implementation (parjob_allocator's combinatorial one, seqjob_allocator's trivial one).
• The xce_host_iter_t interface and its one shared implementation, reused by every allocator.
• parjob_allocator.c's my_alloc()/parjob_prepare_allocation() bodies — real implementations, replacing the stubs.
• seqjob_allocator.c's my_alloc()/seqjob_prepare_allocation() bodies — rewrapped onto the shared iterator pattern, behavior-preserving.
• The session-scoped unsatisfiable-resreq cache: xce_resreq_fingerprint_t, xce_cache_key_t, cache_key_ops (kv_key_ops_t), and the xce_scheduler_t.unsatisfiable_cache field's full lifecycle (populate, lookup, clear).
• The xce_alloc_mode_t enum and xce_sched_ctx_t.alloc_mode field — declared here as an extension point; only XCE_ALLOC_NORMAL behavior is implemented.
• rusage_hash on xce_primary_resreq_t and resreq_fp on xce_primary_schedobj_t, and every construction site that must populate them.

Out of Boundary

• COMPOUND/ALTER resreq dispatch (sched_make_compound_allocation, sched_make_alter_allocation) — calls into this spec's per-primary allocator, otherwise untouched (Requirement 9.2).
• The existing select-expression evaluator, shared-resource tracking (sharedInst/sharedInUse), and plugin framework registration mechanics — consumed, not modified.
• Preemption's actual victim-selection logic and the resource-borrowing semantics XCE_ALLOC_PREEMPT will eventually trigger — scheduler-preemption's job.
• sched_make_compound_allocation()'s missing partial-failure rollback (bug-1766) and the snapshot_load.c schedobj-reconstruction duplication pattern (bug-1767) — both pre-existing, both logged, neither fixed here.
• resreq_fp population on the snapshot-restore path (snapshot_load.c) — confirmed cosmetic (affects only offline simulation sessions, sim.c, never driver.c); not a requirement of this spec (see requirements.md Requirement 7's informational note).

Allowed Dependencies

• scheduler-core (implementation-complete) — core event interface pushing worker/resource state; this feature's iterators consume it but do not depend on the driver/I/O layer.
• scheduler-dispatch-commit (implementation-complete, bug-1624 fixed) — resInUse/sharedInUse totals this feature's try_alloc_on_host() reads must already be correct; this was the spec's own sequencing blocker, now resolved.
• Existing plugin.h/plugin.c plugin framework, resreq.h's select-expression evaluator (select_eval_ctx_t), xvec.c (xvec_op(..., XVEC_SUB_FLOAT) for rollback), kv.c/kv.h (kv_key_ops_t, kvtab_type, key_find/key_add) — all consumed as-is, no changes to their own contracts.
• xce_job_alloc_t/make_job_alloc()/xce_host_alloc_t (xce/types.h, xcealloc.c) — already multi-host-capable; this feature is the first caller to exercise nhosts > 1.

Revalidation Triggers

• Any signature change to xce_alloc_plan_iter_t, xce_host_iter_t, or xce_alloc_plan_t (bucket_slots[] shape) — every allocator plugin consuming them must be re-checked.
• Any change to xce_sched_ctx_t's nslots/nhosts/alloc_mode fields — scheduler-preemption depends on alloc_mode existing exactly as declared here.
• Any change to xce_cache_key_t's field set or plugin_prepare_allocation()'s fan-out semantics.
• scheduler-preemption reaching implementation — its victim-selection flow wraps this spec's allocate_primary()-shaped logic and consumes xce_alloc_mode_t; a mismatch here blocks that spec.

Architecture

Existing Architecture Analysis

• sched_make_primary_alloc_() (scheduler.c:167-177) is the existing choke point every PRIMARY-tagged job passes through: it dispatches to sched_make_seqjob_allocation()/sched_make_parjob_allocation() purely on primary->maxTasks == 1. This dispatch itself is unchanged by this spec — only what happens inside each allocator's job_alloc_fn changes.
• plugin_prepare_allocation() (plugin.c:591-661) unconditionally fans out to every registered plugin's prepare_allocation_hook before each host-satisfaction check, resetting ctx->alloc_req.host/.shared first. seqjob_prepare_allocation() currently has no maxTasks guard, so it already fires (harmlessly, since it's a no-op today) on every job regardless of allocator — this becomes load-bearing once parjob_prepare_allocation() stops being a no-op (Requirement 4.6).
• xce_sched_ctx_t (sched/types.h) already carries nhosts/nslots scratch fields, currently hardcoded to 1, 1 by seqjob_prepare_allocation(); this spec is the first to vary them per allocation attempt.
• core_begin_session_internal() (core_session.c:152-168) already resets other shared, per-session state (core_reset_shared_resval(), core_reset_workers()) with zero explicit locking, relying on xce_core_begin_session()'s own documented ordering guarantee: all begin-session reset work completes strictly before xce_core_signal_go() unblocks the picker/allocator threads. This spec's cache clear reuses that exact placement and safety argument rather than introducing new synchronization.
• xce_job_alloc_t/make_job_alloc() (xce/types.h, xcealloc.c) already generalize past one host (nhosts + xce_host_alloc_t**), but only seqjob_allocator exercises them today, always with nhosts=1 via the variadic make_host_allocs(1, halloc). This spec is the first caller needing a runtime-sized host array, which the existing variadic helper cannot provide.

Project Invariant Cross-Check (per design-review-gate.md): src/scheduler/CLAUDE.md's core-isolation invariant (core.c/core_events.c/core_session.c/core_clock.c carry zero controller I/O, mechanically enforced by test_core_isolation.c) was checked against every new/modified piece of this design. All new code (xce_alloc_plan_iter_t/xce_host_iter_t implementations, the cache, both allocator plugins) is pure in-memory computation — no I/O, no blocking calls — so it does not violate the invariant regardless of which file it lands in. The one touch inside the isolated set (core_begin_session_internal()'s cache clear, core_session.c) is a plain kvtab clear, the same shape as the two existing calls already there.

Architecture Pattern & Boundary Map

graph TB
    Allocator[sched_make_primary_alloc_] --> CacheCheck{Cache lookup}
    CacheCheck -->|hit| Pend[Pend job immediately]
    CacheCheck -->|miss| Dispatch{maxTasks}
    Dispatch -->|equals 1| Seqjob[seqjob_allocator my_alloc]
    Dispatch -->|greater than 1| Parjob[parjob_allocator my_alloc]
    Seqjob --> PlanIterTrivial[Trivial plan iterator one bucket]
    Parjob --> PlanIterCombo[Combinatorial plan iterator]
    PlanIterTrivial --> HostIter[Shared host_iter]
    PlanIterCombo --> HostIter
    HostIter --> TryAlloc[try_alloc_on_host]
    TryAlloc -->|success| Commit[Commit job_alloc]
    TryAlloc -->|all plans exhausted| CacheInsert[Insert fingerprint into cache]
    CacheInsert --> Pend

Architecture Integration:

• Selected pattern: two-layer lazy iterator composition (plan → host) at the PRIMARY leaf, unchanged COMPOUND/ALTER composition above it — see requirements.md's "Composition Algorithm" for the full pseudocode (bucket-loop, host-exclusivity, rollback) this design does not restate.
• Domain/feature boundaries: plan generation is allocator-owned (each job_alloc_fn builds its own plan-iterator instance); host iteration is shared, reused infrastructure; the cache wraps the dispatch choke point, not either allocator individually.
• Existing patterns preserved: sched_ctx_init/sched_ctx_fini's stack-allocate-then-init-in-place convention (extended to the new iterators); plugin_prepare_allocation()'s fan-out (extended, not replaced); core_begin_session_internal()'s lock-free per-session reset pattern (extended to the cache).
• New components rationale: xce_host_iter_t is new because host collection was previously eager and allocator-specific (sched_collect_workers() + xlist_find(), inlined in seqjob_allocator.c); the cache is new because no unsatisfiable-resreq memoization existed at all.
• Steering compliance: no new controller I/O (ha-async-io.md's thread-role contract); no new persistence format, and the one persistence-adjacent question (should the cache or resreq_fp be snapshot-captured) was resolved as "no" with a stated reason (persistence.md-style completeness review, applied inline in requirements.md's Requirement 7 notes).

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Language	C (gnu17)	All new code	No C++, no new external libraries
Hashing/KV	src/lib/kv.c (kvtab_type, kv_key_ops_t)	Backs the unsatisfiable-resreq cache	Existing infra; cache_key_ops is a new, third kv_key_ops_t instance alongside int_key_ops/string_key_ops
Arithmetic primitives	src/lib/xvec.c (xvec_op, XVEC_SUB_FLOAT)	Rollback's resource-debit reversal	Existing primitive, first caller to use the subtract op for this purpose
Concurrency	src/lib/mylock.c-style implicit ordering (no new locks)	Cache safety	Reuses xce_core_begin_session()/xce_core_signal_go()'s existing happens-before guarantee — no new mutex introduced

File Structure Plan

Directory Structure

src/scheduler/
├── include/sched/
│   ├── alloc_iter.h        # NEW — xce_alloc_plan_t, xce_alloc_plan_iter_t,
│   │                       #   xce_host_iter_t interface declarations (Req 2, 3)
│   ├── alloc_cache.h       # NEW — xce_cache_key_t, xce_resreq_fingerprint_t,
│   │                       #   cache_key_ops declaration (Req 7)
│   └── types.h             # MODIFIED — xce_sched_ctx_t gains alloc_mode
│                           #   (Req 9.5); xce_primary_schedobj_t gains
│                           #   resreq_fp (Req 7)
├── src/
│   ├── host_iter.c          # NEW — the one shared xce_host_iter_t
│   │                       #   implementation (Req 3), reused by both
│   │                       #   allocators
│   ├── alloc_cache.c        # NEW — cache_key_hash_/cmp_/assign_/destroy_,
│   │                       #   cache lookup/insert/clear helpers (Req 7)
│   ├── scheduler.c          # MODIFIED — sched_make_primary_alloc_() wraps
│   │                       #   allocator dispatch with cache lookup/insert;
│   │                       #   xce_ctx_uncommit_host() rollback helper (in
│   │                       #   resource.c, see below) called from here
│   ├── resource.c           # MODIFIED — new sched_ctx_uncommit_host()
│   │                       #   helper wired to xvec_op(..., XVEC_SUB_FLOAT)
│   ├── core_session.c       # MODIFIED — core_begin_session_internal()
│   │                       #   clears the cache (Req 7.2)
│   ├── core_events.c        # MODIFIED — core_build_schedobj()/
│   │                       #   core_build_primary_schedobj() compute
│   │                       #   resreq_fp on ingest (Req 7.1)
│   ├── schedobj.c           # MODIFIED — sched_ctx_init() gains nothing new;
│   │                       #   referenced for the stack-allocate-then-init
│   │                       #   convention new iterators follow
│   └── plugins/
│       ├── parjob_allocator.c   # MODIFIED — real my_alloc() (combinatorial
│       │                       #   plan iterator + bucket loop over shared
│       │                       #   host_iter.c); parjob_prepare_allocation()
│       │                       #   fixed to use ctx->nslots (Req 4)
│       └── seqjob_allocator.c   # MODIFIED — my_alloc() rewrapped onto the
│                               #   same bucket-loop pattern as a trivial
│                               #   1-bucket case; seqjob_prepare_allocation()
│                               #   gains maxTasks==1 guard (Req 9.1, 4.6)
├── include/
│   └── scheduler.h          # MODIFIED — xce_scheduler_t gains
│                           #   unsatisfiable_cache field (Req 7.1)
└── testsuite/
    ├── test_alloc_plan_iter.c            # NEW (Req 2)
    ├── test_host_iter.c                  # NEW (Req 3)
    ├── test_alloc_cache.c                # NEW (Req 7)
    ├── test_parjob_combinatorial.c       # NEW (Req 4)
    ├── test_seqjob_rewrap_equivalence.c  # NEW (Req 9.1)
    ├── test_parjob_e2e_multislot.c       # NEW — integration (Req 4, brief.md scope)
    └── meson.build                       # MODIFIED — register the 6 new executables

src/include/
└── resreq.h                 # MODIFIED — xce_primary_resreq_t gains
                              #   rusage_hash (Req 7.1)
src/lib/
├── resreq.c                 # MODIFIED — parse_resreq_primary() computes
│                             #   rusage_hash (Req 7.1)
└── xcealloc.c                # MODIFIED — new merge helper building a
                              #   runtime-sized xce_host_alloc_t** array
                              #   directly (make_host_allocs()'s variadic
                              #   signature cannot take a runtime count),
                              #   used by parjob's commit step (Req 4.3)

Modified Files

See inline annotations above — each modified file's change is scoped to exactly the requirement(s) noted; no file listed above is touched for any other reason.

System Flows

Combinatorial Allocation with Cache (sequence)

sequenceDiagram
    participant D as sched_make_primary_alloc_
    participant C as unsatisfiable_cache
    participant P as parjob_allocator
    participant PI as plan_iter
    participant HI as host_iter
    D->>C: lookup(resreq_fp)
    C-->>D: miss
    D->>P: my_alloc(ctx)
    P->>PI: init + next (plan 1)
    loop each bucket in plan
        P->>HI: init fresh, next until match or exhausted
        HI-->>P: worker or none
    end
    alt all buckets satisfied
        P-->>D: success, commit
    else a bucket exhausted
        P->>P: rollback claimed hosts
        P->>PI: next (plan 2)
        Note over P,PI: repeat until a plan succeeds or plan_iter exhausted
    end
    P-->>D: ALLOC_FAIL (all plans exhausted)
    D->>C: insert(resreq_fp)
    D-->>D: pend job

Full per-bucket accumulation/rollback logic (host-exclusivity via is_claimed(), fresh host_iter per bucket, ctx->uncomitted rollback scope) is specified in requirements.md's "Composition Algorithm" — not restated here; this diagram shows only the cross-file call sequence design.md needs to fix component ownership.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
2.1–2.6	Lazy plan iterator: interface, one-plan-per-call, ordering, exhaustion signal, cleanup, cancel-safety	xce_alloc_plan_iter_t (per-allocator impl)	init/next/fini	Combinatorial Allocation sequence
3.1–3.6	Lazy host iterator: interface, filtering, exhaustion, composition with plan iterator, cleanup, cancel-safety	host_iter.c (shared)	init/next/fini	Combinatorial Allocation sequence
4.1–4.7	Parjob combinatorial allocation: plan generation, per-bucket distinct-host accumulation, commit/merge, stub-return preserved on exhaustion, no unconditional stub, ctx->nslots resource-want plumbing, shared-resource check unchanged	parjob_allocator, alloc_cache.c (fingerprint only, not the check itself)	job_alloc_fn, prepare_allocation_hook	Combinatorial Allocation sequence
7.1–7.5	Unsatisfiable-resreq cache: key/fingerprint shape and O(1) lookup, begin-session clear placement, populate-on-exhaustion, lookup-before-allocation, pend-reason consistency	alloc_cache.c, core_session.c, core_events.c	cache_key_ops (kv_key_ops_t), key_find/key_add	Combinatorial Allocation sequence
9.1–9.5	No regression: seqjob behavior unchanged, COMPOUND/ALTER unchanged, existing unit tests pass, HA soak clean, xce_alloc_mode_t extension point	seqjob_allocator, xce_sched_ctx_t.alloc_mode	job_alloc_fn (seqjob), xce_alloc_mode_t enum	Combinatorial Allocation sequence (seqjob path = 1-bucket case)

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
xce_alloc_plan_iter_t (interface)	Scheduler / Allocation	Lazy per-allocator plan generation	2.1–2.6	xce_primary_resreq_t (P0)	Service
host_iter.c	Scheduler / Allocation	Shared lazy host-candidate filtering	3.1–3.6	select-expression evaluator (P0), resource.c (P0)	Service
parjob_allocator	Scheduler / Plugin	Real combinatorial multi-host allocation	4.1–4.7	xce_alloc_plan_iter_t (P0), host_iter.c (P0), resource.c (P0)	Service
seqjob_allocator	Scheduler / Plugin	Trivial single-host allocation, behavior-preserving rewrap	9.1, 4.6	xce_alloc_plan_iter_t (P0), host_iter.c (P0)	Service
alloc_cache.c	Scheduler / Allocation	Session-scoped unsatisfiable-resreq memoization	7.1–7.5	kv.c (P0), core_session.c (P0)	Service, State
xce_alloc_mode_t extension point	Scheduler / Allocation	Forward-compatible resource-pool-mode surface	9.5	none (v1: XCE_ALLOC_NORMAL only)	State

Scheduler / Allocation

xce_alloc_plan_iter_t

Field	Detail
Intent	Lazy, cancel-safe generation of candidate (bucket_slots[]) partitions, most-preferred first, one per allocator
Requirements	2.1, 2.2, 2.3, 2.4, 2.5, 2.6

Responsibilities & Constraints

• Each concrete implementation (parjob's combinatorial, seqjob's trivial) owns its own plan-shape and partition policy; the interface itself is policy-agnostic.
• v1's combinatorial default (ascending host-count, canonical balanced ceil/floor partition, descending bucket order) is fully specified in requirements.md's "Data Structure Changes Summary" — a pure function of maxTasks, never inspecting live worker state.
• Stack-allocated by the caller, initialized in place (iter.init(&iter, resreq, sched)) — never a heap-allocating factory (matches sched_ctx_init(&ctx, ...)'s existing convention).

Dependencies

• Inbound: parjob_allocator, seqjob_allocator (P0, sole consumers)
• Outbound: xce_primary_resreq_t (P0, read-only — minTasks/maxTasks used in v1; span/same/affinity read by future allocators without an interface change)

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

struct xce_alloc_plan_iter_s {
    int (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq, xce_scheduler_t* sched);
    xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter);
    void (*fini)(xce_alloc_plan_iter_t* iter);
    void* private;
};

• Preconditions: resreq != NULL; init() called exactly once before any next().
• Postconditions: next() returns candidate plans strictly in most-preferred-first order until exhaustion (stable NULL thereafter); fini() releases all iterator-owned resources.
• Invariants: the pointer next() returns is valid only until the following next() or fini() call; next() calls pthread_testcancel() (or equivalent) at least once per invocation (Req 2.6).

host_iter.c

Field	Detail
Intent	The one shared xce_host_iter_t implementation, filtering candidate hosts by select expression + resource availability for a given plan
Requirements	3.1, 3.2, 3.3, 3.4, 3.5, 3.6

Responsibilities & Constraints

• One implementation, reused by every allocator — no allocator-specific host-filtering logic anywhere else.
• Scoped per plan bucket, not per plan: a multi-bucket plan requires a fresh xce_host_iter_t instance for each bucket (init() called once per bucket), never one instance shared across a plan's buckets — see requirements.md's Requirement 3.3/3.4 for why a shared, forward-only cursor is an incorrect implementation (it silently drops a host an earlier, larger bucket rejected even when it exactly satisfies a later, smaller one).
• Host exclusivity across buckets of the same plan (is_claimed()) is the caller's responsibility, not this component's — this component only ever answers "does this select expression + availability filter match," with no memory of prior buckets.

Dependencies

• Inbound: parjob_allocator, seqjob_allocator (P0)
• Outbound: existing select-expression evaluator (resreq.h, select_eval_ctx_t) (P0, unchanged); resource.c's sched_check_host_satisfy() (P0, unchanged)

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

/* Correction (found during task 1.1 implementation): next() returns
 * xce_sched_worker_t* (sched/types.h -- resvals/resInUse/host_eval_ctx),
 * not xce_worker_t* (xce/types.h -- the wire/config type make_sched_worker()
 * converts FROM). requirements.md's own Composition Algorithm pseudocode
 * already used xce_sched_worker_t*; this struct declaration was out of
 * sync with it. */
struct xce_host_iter_s {
    int (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan, var workers);
    xce_sched_worker_t* (*next)(xce_host_iter_t* iter);
    void (*fini)(xce_host_iter_t* iter);
    void* private;
};

• Preconditions: plan != NULL; a fresh instance per bucket (see above).
• Postconditions: next() returns NULL once no further candidate exists for this instance's bucket search; a fresh instance for the next bucket is unaffected.
• Invariants: independent lifecycle from the owning xce_alloc_plan_iter_t — fini() on one never affects the other; next() calls pthread_testcancel() (or equivalent) at least once per invocation (Req 3.6).

Implementation Notes

• Integration: constructed fresh inside the per-bucket loop of both parjob_allocator's and seqjob_allocator's my_alloc() — see requirements.md's "Composition Algorithm" for the exact loop shape.
• Validation: test_host_iter.c must include the exact adversarial case that motivated the fresh-per-bucket requirement (a host too small for an earlier, larger bucket remaining eligible for a later, smaller one).
• Risks: none new — this is the shared implementation of behavior seqjob_allocator already has today (sched_collect_workers() + xlist_find()), generalized.

alloc_cache.c

Field	Detail
Intent	Session-scoped memoization of resreqs this session already proved unsatisfiable, keyed by a composite fingerprint
Requirements	7.1, 7.2, 7.3, 7.4, 7.5

Responsibilities & Constraints

• Owns xce_cache_key_t, xce_resreq_fingerprint_t, and cache_key_ops (a third kv_key_ops_t instance alongside int_key_ops/string_key_ops). Full field-by-field hash/cmp/assign/destroy implementations are in requirements.md's "New types (Req 7)" section — corrected during design-phase discovery (the .cmp return-value polarity was inverted in an earlier draft; fixed to the real three-way, 0-means-equal convention kv.c's int_key_ops.cmp establishes).
• Lookup/insert happen exclusively on the allocator dispatch path (sched_make_primary_alloc_(), wrapping either allocator's call) — never inside parjob_allocator.c/seqjob_allocator.c themselves, so both allocators benefit identically without duplicated cache-handling code.
• Clear happens exactly once, inside core_begin_session_internal() (core_session.c), alongside the existing core_reset_shared_resval()/core_reset_workers() calls — no new lock, reusing xce_core_begin_session()'s existing happens-before guarantee (see Existing Architecture Analysis above).
• Not part of the snapshot/replay wire format — pure in-memory, unconditionally cleared at the next begin_session regardless of what a loaded snapshot contained.

Dependencies

• Inbound: scheduler.c's sched_make_primary_alloc_() (P0, sole caller of lookup/insert); core_session.c's core_begin_session_internal() (P0, sole caller of clear)
• Outbound: kv.c's kvtab_type/key_find/key_add (P0); core_events.c's core_build_schedobj() (P0, populates resreq_fp on ingest); resreq.c's parse_resreq_primary() (P0, populates rusage_hash)

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x]

Service Interface

/* Correction (kiro-validate-design pass): a bare hit/miss sentinel cannot
 * satisfy Req 7 AC 5 ("same pend reason category" on a hit) -- the value
 * stored per key is the original miss's reason CATEGORY (an int code,
 * e.g. xce_job_preason_t.main_reason), not a placeholder. Not the full
 * detailed reason object: that carries job/host-specific breakdown
 * (xce_host_reason_t*/xce_res_reason_t* arrays) that wouldn't transfer
 * meaningfully to a different job sharing the same fingerprint. */
int cache_lookup(xce_scheduler_t* sched, xce_sched_obj_t* obj, int* out_reason_category);
                                    /* 1 = hit (out_reason_category set), 0 = miss */
int cache_insert(xce_scheduler_t* sched, xce_sched_obj_t* obj, int reason_category);
                                    /* 0 = ok, <0 = error */
void cache_clear(xce_scheduler_t* sched);   /* called from core_begin_session_internal() */

• Preconditions: obj->tag == XCE_RESREQ_PRIMARY; obj->u.primary->resreq_fp already populated (job-ingest time); cache_insert()'s reason_category is the exhausted plan iterator's own final pend reason's main_reason.
• Postconditions: cache_lookup() returning a hit means the caller pends the job using *out_reason_category (Req 7.5) and never invokes the plan/host iterators or preemption for this attempt.
• Invariants: cache is cleared exactly once per scheduling session, strictly before the picker/allocator threads touch it for that session.

State Management

• State model: one kvtab_t per xce_scheduler_t, keyed by xce_cache_key_t (composing xce_resreq_fingerprint_t + {user, queue, project} xstring pointers), value is the reason-category int boxed as (void*)(intptr_t)category (no heap allocation needed for a single int; NULL datakiller, matching int_key_ops's own value-handling convention).
• Persistence & consistency: none — explicitly not snapshot-captured (see Boundary Commitments).
• Concurrency strategy: no explicit lock; safety derives from xce_core_begin_session()'s documented ordering guarantee plus lookup/insert being confined to the single-threaded allocator path.

Implementation Notes

• Integration: cache_key_cmp_()'s three-way-comparator fix (this design pass) must land alongside the rest of alloc_cache.c — a functionally dead cache from day one would not fail any test that doesn't specifically assert a hit, so test_alloc_cache.c must include a same-key-twice hit assertion, not only a miss-then-insert-then-lookup round-trip that could pass even with the inverted polarity if written carelessly.
• Validation: test_alloc_cache.c must also assert the reason-category round-trip (insert with category X, hit later returns X) — the value-model fix from this validation pass.
• Risks: .cmp polarity bug already fixed (see research.md); the reason-category round-trip is the one new surface this fix adds, worth its own explicit test rather than assuming the hit/miss path alone proves it.

Scheduler / Plugin

parjob_allocator

Field	Detail
Intent	Real combinatorial multi-host, multi-slot job allocation
Requirements	4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7

Responsibilities & Constraints

• my_alloc() walks its own combinatorial plan iterator; for each plan, loops buckets, each bucket getting a fresh host_iter.c instance (host-exclusivity via is_claimed(), not iterator sharing).
• The per-plan claimed[] accumulator (tracking which host was claimed for each bucket, for both the merge step and rollback) is sized by primary_resreq->maxTasks — the true, tight upper bound on bucket count (each bucket needs ≥1 slot, so nbuckets can never exceed it) — not a fixed compile-time constant an implementer would otherwise have to guess (kiro-validate-design finding, corrected in requirements.md's pseudocode).
• On full-plan success: merges claimed hosts into one xce_job_alloc_t via the new xcealloc.c helper (runtime-sized array, not make_host_allocs()'s variadic signature).
• On per-bucket failure: rolls back via resource.c's new sched_ctx_uncommit_host(), scoped to ctx->uncomitted only (never touches real resInUse).
• On full exhaustion: returns 0, identical to today's stub return value, so sched_make_primary_alloc_()'s pend-reason handling is unaffected (Req 4.4).
• parjob_prepare_allocation() sets ctx->alloc_req via calc_wanted_host_resvals(..., ctx->nhosts, ctx->nslots), reading the current bucket's slot count from ctx->nslots (set by the bucket loop before each host try) rather than a hardcoded constant.
• Shared-resource satisfaction is checked once per primary resreq, independent of plan/bucket shape — unchanged from today's mechanism (Req 4.7).

Dependencies

• Inbound: sched_make_primary_alloc_() (P0)
• Outbound: xce_alloc_plan_iter_t (own combinatorial impl, P0), host_iter.c (P0), resource.c (sched_check_host_satisfy, sched_ctx_uncommit_host, P0), xcealloc.c (merge helper, P0)

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

static int my_alloc(xce_sched_plugin_t* plugin, xce_sched_ctx_t* ctx);              /* job_alloc_fn */
static int parjob_prepare_allocation(void* arg, xce_sched_ctx_t* ctx);              /* prepare_allocation_hook */

• Preconditions: ctx->primary->maxTasks > 1 (routed here by sched_make_primary_alloc_()).
• Postconditions: return >0 and ctx->alloc populated on success; return 0 with ctx->reason populated on exhaustion — same contract shape seqjob_allocator's my_alloc() already has.
• Invariants: never claims the same host twice within one plan attempt; never leaves a phantom ctx->uncomitted debit after a failed plan attempt.

Implementation Notes

• Integration: registration in the plugin framework (register_parjob_allocator()) is unchanged — only the two hook bodies change.
• Validation: test_parjob_combinatorial.c must cover the plan-order-matters worked example from requirements.md (descending bucket order succeeding where ascending would fail, identical host pool) and the fresh-per-bucket host_iter requirement.
• Risks: the seqjob_prepare_allocation() fan-out corruption this fix depends on (Req 4.6) — mitigated by landing both changes together, never one without the other (see research.md Risks).

seqjob_allocator

Field	Detail
Intent	Trivial single-host allocation, rewrapped onto the shared iterator pattern with provably unchanged external behavior
Requirements	9.1, 4.6

Responsibilities & Constraints

• my_alloc()'s bucket loop degenerates to exactly one iteration (nbuckets == 1, bucket_slots == [1]), using the same shared host_iter.c and is_claimed()/rollback machinery parjob_allocator uses — the abstraction validated on the simple case before the complex one.
• seqjob_prepare_allocation() gains a primary->maxTasks == 1 guard — a no-op for seqjob's own behavior (it never sees maxTasks > 1 jobs), required only so it stops corrupting parjob_allocator's fixed prepare hook via plugin_prepare_allocation()'s fan-out.

Dependencies

• Inbound: sched_make_primary_alloc_() (P0)
• Outbound: xce_alloc_plan_iter_t (own trivial impl, P0), host_iter.c (P0)

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

Same shape as parjob_allocator's (my_alloc/prepare_allocation_hook) — see above; not restated.

• Preconditions: ctx->primary->maxTasks == 1.
• Postconditions: identical host selection to today's sched_collect_workers() + xlist_find() implementation, given identical worker/resource state (Req 9.1) — this is the requirement the rewrap must prove, not just aim for.
• Invariants: unchanged.

Implementation Notes

• Integration: registration (register_seqjob_allocator()) unchanged.
• Validation: test_seqjob_rewrap_equivalence.c must construct fixture states where the rewrap could plausibly diverge (multiple candidate hosts, at least one failing the select filter) and assert identical outcomes pre/post rewrap — a test that only exercises the single-obvious-host case would not actually prove equivalence.
• Risks: none beyond the shared seqjob_prepare_allocation() guard dependency already noted under parjob_allocator.

Scheduler / Allocation (extension point)

xce_alloc_mode_t

Field	Detail
Intent	Forward-compatible resource-pool-mode field on xce_sched_ctx_t, declared now so scheduler-preemption (and later modes) don't need to modify this spec's struct later
Requirements	9.5

Responsibilities & Constraints

• v1 declares XCE_ALLOC_NORMAL = 0 and XCE_ALLOC_PREEMPT; only XCE_ALLOC_NORMAL behavior is implemented — no branch anywhere in this spec's try_alloc_on_host()/availability-check call sites reads this field for any purpose beyond existing.
• Adding further enum values or the actual borrowing behavior XCE_ALLOC_PREEMPT implies is explicitly scheduler-preemption's job, not this spec's.

Contracts: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x]

State Management

• State model: a single xce_alloc_mode_t alloc_mode field on xce_sched_ctx_t, defaulting to XCE_ALLOC_NORMAL (zero-value, matches sched_ctx_init()'s existing memset(ctx, 0, sizeof(*ctx))).
• Persistence & consistency: none — xce_sched_ctx_t is per-allocation-attempt scratch state, never persisted (see requirements.md's "Rollback scope" discussion of the same struct).
• Concurrency strategy: none needed — single-threaded, per-attempt scratch.

Data Models

Domain Model

• xce_alloc_plan_t: a partition of a job's maxTasks into per-host slot buckets (nbuckets, bucket_slots[]) — the domain concept this whole feature revolves around. Not persisted; lives only for the duration of one plan attempt.
• xce_resreq_fingerprint_t / xce_cache_key_t: value objects, immutable once computed, fitting in a single cache line — see requirements.md's exact field layout, not restated here.

Logical Data Model

No new persistent entities. The only durable-adjacent addition is rusage_hash on xce_primary_resreq_t (in-memory, process-lifetime, set once at parse time) and resreq_fp on xce_primary_schedobj_t (same lifetime class) — both scratch/derived fields, never wire-serialized, confirmed in requirements.md's Requirement 7 notes and this spec's own Boundary Commitments.

Data Contracts & Integration

None — this feature introduces no new wire format, no new capnp schema, and (per the explicit design decision in requirements.md) no snapshot/replay format changes for the cache.

Error Handling

Error Strategy

Allocation failure is not an error condition in this domain — it is the expected, common "no plan/host satisfies this resreq right now" outcome, communicated via return value (0) and ctx->reason, exactly as today. Genuine errors (OOM, iterator init() failure) propagate as negative return codes and are treated the same as "no candidate for this bucket" at the allocation-decision level — a distinction requirements.md's Composition Algorithm draws explicitly (host-iterator init failure treated as an infrastructure error, not a resource-availability determination).

Error Categories and Responses

• Resource unavailable (expected, high-frequency): plan exhausted → pend job with reason; cache the fingerprint so subsequent identically-shaped jobs skip the cost (Req 7).
• Iterator init failure (rare, e.g. OOM): treated as "this bucket/plan has no candidates," not a distinct error path — avoids a second failure taxonomy the caller would need to handle.
• Fan-out corruption (latent, closed by this design): seqjob_prepare_allocation()'s missing maxTasks guard, closed by Req 4.6 — see Existing Architecture Analysis.

Monitoring

No new logging/metrics surface beyond what ctx->reason/xce_pendreason_t already provides — this feature populates the same pend-reason machinery every allocator already uses, adding no new observability contract.

Testing Strategy

• Unit — plan iterator (test_alloc_plan_iter.c, Req 2): one-plan-per-next()-call; most-preferred-first ordering for maxTasks values with and without a clean partition (e.g. N=7 → [3,2,2] for k=3); stable exhaustion signal; fini() cleanup; pthread_testcancel() present.
• Unit — host iterator (test_host_iter.c, Req 3): select-expression + availability filtering unchanged from the existing evaluator; the fresh-per-bucket adversarial case (host too small for an earlier bucket remains eligible for a later, smaller one) — this is the single most important test in this spec, directly proving the bug found during design discussion is closed.
• Unit — cache (test_alloc_cache.c, Req 7): hit on identical fingerprint+identity; miss on any differing field; same-key-twice hit (not just insert-then-miss-different-key, which the inverted .cmp bug would still have passed); reason-category round-trip (insert with category X, later hit returns X, per Req 7.5 — the kiro-validate-design fix to the cache's value model); clear-at-begin-session.
• Unit — parjob combinatorial (test_parjob_combinatorial.c, Req 4): multi-bucket plan success and partial-failure-then-rollback; descending-vs-ascending bucket order worked example; ctx->nslots correctly varies the resource want per bucket.
• Unit — seqjob rewrap equivalence (test_seqjob_rewrap_equivalence.c, Req 9.1): identical host selection pre/post rewrap across fixture states including a failing-select-filter host, not just the single-obvious-host trivial case.
• Integration — end-to-end multi-slot allocation (test_parjob_e2e_multislot.c, brief.md scope): a real multi-host parallel job dispatched through the full sched_make_primary_alloc_() → parjob_allocator → commit path, asserting the final xce_job_alloc_t.nhosts > 1 and correct per-host slot counts.
• Regression — existing scheduler suite (Req 9.3): full scheduler meson suite must pass unmodified in assertions, adjusted only where a test directly exercised the eager-one-shot or stub-return behavior this feature replaces.
• HA soak (Req 9.4): sequential and chaotic /xce-test-ha regression, confirming zero new job loss/divergence/liveness-stall attributable to this feature.

Supporting References

• Full pseudocode (composition algorithm, cache key hash/cmp/assign/destroy, v1 plan-shape/partition defaults) lives in requirements.md — not duplicated here per this template's own deduplication guidance. design.md's job is component/file ownership and contracts; requirements.md remains the algorithmic source of truth.
• research.md — discovery findings (the .cmp polarity fix, the core-isolation boundary check, the test-file convention) and the three most consequential design decisions with alternatives considered.