myxce/.kiro/specs/scheduler-framework/brief.md

Brief: scheduler-framework

Problem

The XCE scheduler has a plugin architecture but three critical gaps prevent it from being a real framework:

1. Job selection policy is hardcoded — sched_job_picker() in scheduler.c:48 uses the literal string "xce_sched_fcfs". No way to swap policies without recompilation.
2. Parallel job allocation is a stub — parjob_allocator.c:my_alloc() returns 0 with no logic. Jobs requiring multiple slots (maxTasks > 1) produce no allocation.
3. Allocation is one-shot — sched_make_allocation() tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.

Operators need configurable scheduling policies and parallel job support. The scheduler must generate multiple candidate allocation plans and try them in preference order.

Current State

What exists

• Plugin framework (plugin.h, plugin.c): xce_sched_plugin_t with 9 hooks + job_elect_fn / job_alloc_fn. Plugins registered in key-value map by name.
• FCFS job picker (fcfs.c): dedicated jobpicker thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via jbuf cbuf.
• Seqjob allocator (seqjob_allocator.c): handles maxTasks == 1. Checks shared resources → collects workers → tries one host → returns first match.
• Parjob allocator stub (parjob_allocator.c): registered but my_alloc() is empty.
• Resource request types (resreq.h): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
• Select expression AST (resreq.h): full parser/evaluator with per-session memoization (select_eval_ctx_t) and inter-session GC.
• Shared resources (xce_scheduler_t.sharedInst/sharedInUse): zarray of resource values tracked per scheduling session.
• Scheduling loop (scheduler.c:sched_allocator()): pick job → sched_make_allocation() → pend or dispatch → next.

What's missing

• Configurable job picker: the plugin name is a hardcoded string literal
• Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
• Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
• Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session

Desired Outcome

1. Configurable job selection: operator can set job_pick_plugin in scheduler config to swap policies (FCFS, priority, custom) without recompilation. Plugin interface contract is documented (cancel-safe, session-aware).
2. Lazy allocation plan iterator: xce_alloc_plan_iter_t — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
3. Lazy host candidate iterator: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
4. Parallel job combinatorial allocation: parjob_allocator implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
5. Session-level unsatisfiable resreq cache: if a primary resreq cannot be satisfied in the current scheduling session (all plans exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.

Alternatives Considered

Approach A: Incremental Enhancement (rejected)

Aspect	Detail
Job selection	Add job_pick_plugin config field, same as chosen approach
Parallel jobs	Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try
Iterator	No new abstraction. Simple for-loop over plan array, inner for-loop over hosts
Shared resreq	xlist_t unsatisfiable_resreqs in session state, same as chosen approach

Rejected because: Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.

Approach B: Iterator-Driven Allocation Architecture ★ Chosen

Aspect	Detail
Job selection	Configurable plugin name via job_pick_plugin field
Iterator abstraction	xce_alloc_plan_iter_t + xce_host_iter_t — lazy, cancel-safe, composable
Parallel jobs	Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1
Host iterator	Per-plan lazy candidate host iteration using existing select expression eval
Shared resreq	Session-level fingerprint cache: {select, rusage, user, queue, project}

Chosen because: Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable.

Approach C: Full Scheduler Framework (deferred)

Aspect	Detail
Job selection	Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption
Iterator	Triple composition: job iterator → plan iterator → host iterator
Resreq registry	Named primary resreqs registered globally, referenced by name, reference-counted
Allocation plugins	Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware
Session state machine	Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions

Deferred (not rejected): The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces.

Decision rationale

Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). Approach C's additional layers (policy chains, resreq registry, state machine) are deferred until real use cases emerge — the iterator interfaces are designed to compose upward when needed.

Approach

Iterator-Driven Allocation Architecture (Approach B):

Core abstractions

xce_alloc_plan_iter_t          xce_host_iter_t
┌─────────────────────┐        ┌──────────────────────┐
│ init(resreq, sched)  │        │ init(plan, workers)   │
│ next() → plan        │───────▶│ next() → worker       │
│ fini()               │        │ fini()                │
└─────────────────────┘        └──────────────────────┘

• Plan iterator: owned by allocator plugin (parjob_allocator or seqjob_allocator). Generates (nhosts, slots_per_host, host_filter) plans lazily.
• Host iterator: per-plan. Wraps sched_collect_workers() + select expression filtering into a lazy interface. Each next() returns the next candidate host or NULL.

Scheduling loop (modified)

for each elected job:
    plan_iter = job.allocator.make_plan_iter(resreq)
    while (plan = plan_iter.next()):
        host_iter = make_host_iter(plan, all_workers)
        while (host = host_iter.next()):
            if try_alloc(host, plan) succeeds:
                dispatch(job, host, plan)
                goto next_job
            host_iter.advance()
        host_iter.fini()
    plan_iter.fini()
    pend(job, reason)

Configurable job picker

• Add char* job_pick_plugin to xce_scheduler_t (default "xce_sched_fcfs")
• sched_job_picker() reads this field instead of hardcoded string
• Set via scheduler config or command-line flag
• Plugin interface contract: job_elect_fn must be cancel-safe (respond to pthread_cancel), session-aware (reset state on end_session), and communicate via jbuf cbuf

Unsatisfiable resreq cache

• Per scheduling session (xce_scheduler_t), cleared on begin_session
• Key: hash of {select_string, rusage_string, user, queue, project}
• Before full allocation: quick lookup. If present → skip to next job immediately
• Populated when a job's plan iterator exhausts all plans without success

What stays unchanged

• Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
• seqjob_allocator — wrapped with iterator interface, internally still 1host×1slot
• fcfs.c — continues as default job picker, no changes needed
• Select expression system — no changes
• COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
• Controller communication (handler.c opcodes, decision cbuf) — unchanged

Scope

In

• xce_alloc_plan_iter_t interface (header + implementation)
• xce_host_iter_t interface (header + implementation)
• parjob_allocator combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
• Configurable job picker: job_pick_plugin field + config wiring
• Session-level unsatisfiable resreq fingerprint cache
• Plugin interface contract documentation (cancel-safe, session-aware)
• Unit tests for plan generation, host iteration, cache hit/miss
• Integration test: multi-slot parallel job allocation end-to-end

Out

• New job selection policies (priority-only, fair-share, preemption, backfill) — framework enables them but they ship separately
• Resreq template registry (named/shared resreq objects) — future enhancement
• Select expression language changes
• Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
• Compound/ALTER resreq allocation changes
• GPU/affinity-aware allocation (pluggable later via allocator plugins)

Boundary Candidates

The work naturally splits into these implementation phases (single spec, sequential):

1. Iterator interfaces — define xce_alloc_plan_iter_t and xce_host_iter_t, implement for seqjob path first (trivial wrapper), validate the abstraction
2. Parallel job allocator — implement combinatorial plan generation in parjob_allocator, wire through host iterator
3. Configurable job picker — add job_pick_plugin config, document plugin contract
4. Unsatisfiable resreq cache — session-level fingerprint cache, wire into scheduling loop
5. Integration + testing — end-to-end multi-slot job allocation, cache behavior, config swapping

Out of Boundary

• New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
• Resreq template sharing across jobs (future: resreq_registry spec)
• Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
• Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)

Upstream / Downstream

Upstream (dependencies)

• Existing plugin framework (plugin.h, plugin.c) — NO changes needed
• Existing select expression evaluator (resreq.h eval/Gc) — NO changes needed
• Existing shared resource tracking (sharedInst/sharedInUse) — NO changes needed
• Existing scheduler data structures (xce_scheduler_t, xce_sched_ctx_t) — minimal additions

Downstream (future consumers)

• New job selection policies — priority, fair-share, preemption plugins register via the configurable picker
• Affinity/GPU allocators — register as allocator plugins, implement plan iterator for topology-aware placement
• Commit barrier wiring — scheduling decisions flow through majority commit (separate hardening item)
• Resreq template registry — named resreqs shared across jobs, reference-counted

Existing Spec Touchpoints

Extends

• scheduler-controller-msg-gap — adjacent (message handling). The framework enhancement operates on the same xce_scheduler_t state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.

Adjacent

• ha-consistency-invariants — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
• worker-join-refactor — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.

Constraints

• C (gnu17) — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
• Non-blocking in plugin hooks — allocation happens on allocator thread, not iomgr. No new blocking concerns.
• Cancel-safe — job picker thread is pthread_cancel-able. Iterator next() calls must have cancellation points.
• Session isolation — iterator state and resreq cache are per-session, cleared on begin_session.
• Backward compatible — existing FCFS + seqjob behavior unchanged. New functionality is additive.
• Memory — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).