myxce/.kiro/specs/scheduler-framework/brief.md

Brief: scheduler-framework

Problem

The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:

1. Job selection policy is hardcoded — sched_job_picker() in scheduler.c:39 uses the literal string "xce_sched_fcfs". No way to swap policies without recompilation. The picker is monolithic — it both chooses which queue to serve and which job within the queue.
2. Parallel job allocation is a stub — parjob_allocator.c:my_alloc() returns 0 with no logic. Jobs requiring multiple slots (maxTasks > 1) produce no allocation.
3. Allocation is one-shot — sched_make_allocation() tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
4. No preemption — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
5. No project-level scheduling — xce_sched_project_t is just { char* name }. Projects have no resource ownership, quota, or scheduling policy. Queue and project roles are conflated.

Operators need configurable scheduling policies at multiple levels, parallel job support, and preemption. The scheduler must generate multiple candidate allocation plans and try them in preference order.

Current State

What exists

• Plugin framework (plugin.h, plugin.c): xce_sched_plugin_t with 9 hooks + job_elect_fn / job_alloc_fn. Plugins registered in key-value map by name.
• FCFS job picker (fcfs.c): dedicated jobpicker thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via jbuf cbuf.
• Seqjob allocator (seqjob_allocator.c): handles maxTasks == 1. Checks shared resources → collects workers → tries one host → returns first match.
• Parjob allocator stub (parjob_allocator.c): registered but my_alloc() is empty.
• Resource request types (resreq.h): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
• Select expression AST (resreq.h): full parser/evaluator with per-session memoization (select_eval_ctx_t) and inter-session GC.
• Shared resources (xce_scheduler_t.sharedInst/sharedInUse): zarray of resource values tracked per scheduling session.
• Scheduling loop (scheduler.c:sched_allocator()): pick job → sched_make_allocation() → pend or dispatch → next.

What's missing

• Configurable job picker: the plugin name is a hardcoded string literal; a monolithic picker conflates inter-queue and intra-queue selection
• Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
• Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
• Preemption: no mechanism to preempt lower-priority running jobs when a high-priority job cannot be scheduled
• Project-level scheduling: xce_sched_project_t is just { char* name } — no resource ownership, quota, or scheduling policy
• Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
• Picker composition: no way to assign different pickers to different queues or projects

Desired Outcome

Four-Level Scheduling Architecture

The scheduling loop is always project → queue → job, composing pickers at each level:

Level	Picker	Plugin slot	Default (v1)
0	Project picker	sched->project_pick_plugin	builtin_flat — single "default" project
1	Inter-queue picker	sched->queue_pick_plugin	builtin_strict_priority
2	Intra-queue picker	queue→project→sched chain	builtin_fcfs
3	Preemption policy	queue->preempt_policy	lowest_priority_first

1. Unified picker loop: A single scheduling loop that always follows project → queue → job. In v1 (flat project space), the project picker builtin_flat yields one "default" project containing all queues — identical to today's behavior. In v2 (future), the project picker becomes quota-aware and hierarchical, without changing the loop code.
2. Configurable job selection at every level: each picker slot is a plugin. Level 1 (inter-queue) picks which queue to serve next. Level 2 (intra-queue) picks which job within that queue, with per-queue customization via the queue → project → global precedence chain.
3. Lazy allocation plan iterator: xce_alloc_plan_iter_t — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
4. Lazy host candidate iterator: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
5. Parallel job combinatorial allocation: parjob_allocator implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
6. Preemption: when allocation fails for a high-priority job, select victim jobs from lower-priority queues (LSF-style per-queue preemptible_queues list), send SIGSTP via controller→worker pipeline, and retry allocation next round.
7. Session-level unsatisfiable resreq cache: if a primary resreq cannot be satisfied in the current scheduling session (all plans + preemption exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.

Queue vs Project — Roles

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered and constrained?	WHO owns resources and gets charged?
Scope	One hardware/scheduling profile	Cross-queue, cross-cluster
Key attributes	Priority, limits, time windows, picker, preemption config	Quota, usage, hierarchy, members (future)
Cardinality	1 job → 1 queue, many-to-many	1 job → 1 project, many-to-many

One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware).

Alternatives Considered

Approach A: Incremental Enhancement (rejected)

Aspect	Detail
Job selection	Add job_pick_plugin config field, same as chosen approach
Parallel jobs	Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try
Iterator	No new abstraction. Simple for-loop over plan array, inner for-loop over hosts
Shared resreq	xlist_t unsatisfiable_resreqs in session state, same as chosen approach

Rejected because: Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.

Approach B: Iterator-Driven Allocation Architecture ★ Chosen

Aspect	Detail
Job selection	Configurable plugin name via job_pick_plugin field
Iterator abstraction	xce_alloc_plan_iter_t + xce_host_iter_t — lazy, cancel-safe, composable
Parallel jobs	Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1
Host iterator	Per-plan lazy candidate host iteration using existing select expression eval
Shared resreq	Session-level fingerprint cache: {select, rusage, user, queue, project}

Chosen because: Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable.

Approach C: Full Scheduler Framework (deferred)

Aspect	Detail
Job selection	Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption
Iterator	Triple composition: job iterator → plan iterator → host iterator
Resreq registry	Named primary resreqs registered globally, referenced by name, reference-counted
Allocation plugins	Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware
Session state machine	Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions

Deferred (not rejected): The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces.

Decision rationale

Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). The four-level picker architecture (project → queue → job + preemption) extends Approach B with design elements from Approach C (policy composition, formalized level separation) that are needed NOW (per-queue pickers, preemption), while deferring the remaining Approach C layers (policy chains with scoring, resreq registry, formal state machine) until real use cases emerge. The iterator interfaces and picker slots are designed to compose upward when needed.

Updated decision (2026-07-10): The original Approach B treated the job picker as a single global plugin (job_pick_plugin on xce_scheduler_t). Design discussion identified that different queues/projects need different pickers, and that preemption is an essential missing mechanism. The architecture now specifies four composable levels (project, queue, job, preemption) with per-queue configuration, LSF-style cross-queue preemption, and a unified loop that works identically for v1 (flat projects) and v2 (hierarchical projects). The project picker level is deferred to a future spec but the architecture slot and interface contract are defined now.

Approach

Iterator-Driven Allocation Architecture with Four-Level Picker Composition (extended from original Approach B):

Core abstractions

┌──────────────────────────────────────────────────────────┐
│ Level 0: Project Picker    [sched->project_pick_plugin]   │
│                                                            │
│ v1 (builtin_flat): yields one "default" project            │
│ v2 (future): quota-aware project selection                 │
└──────────────────────┬───────────────────────────────────┘
                       │ project scope
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin]    │
│                                                            │
│ Iterates queues scoped to the selected project             │
│ Default: strict_priority (descending queue->priority)      │
└──────────────────────┬───────────────────────────────────┘
                       │ queue
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Level 2: Intra-Queue Picker  [queue→project→sched]        │
│                                                            │
│ Picks one job from the selected queue                      │
│ Default: builtin_fcfs (priority desc, queue_time asc)      │
└──────────────────────┬───────────────────────────────────┘
                       │ job
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Allocation                                               │
│                                                            │
│ xce_alloc_plan_iter_t          xce_host_iter_t            │
│ ┌─────────────────────┐        ┌──────────────────────┐   │
│ │ init(resreq, sched)  │        │ init(plan, workers)   │   │
│ │ next() → plan        │───────▶│ next() → worker       │   │
│ │ fini()               │        │ fini()                │   │
│ └─────────────────────┘        └──────────────────────┘   │
│                                                            │
│   success → dispatch                                      │
│   fail → Level 3 (preemption)                             │
└──────────────────────┬──────────────────────────────────┘
                       │ fail
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Level 3: Preemption                                      │
│                                                            │
│ if queue->preemptible_queues:                             │
│   victims = preempt_policy.select_victims(job)            │
│   if victims: commit XCE_DECISION_PREEMPT                 │
│     → controller → worker → SIGSTP victims               │
│     → victims → JOB_SUSPENDED, release resources          │
│     → pend job for next round                             │
│   else: pend(job, "Resources")                            │
│ else: pend(job, "Resources")                              │
└──────────────────────────────────────────────────────────┘

Picker plugin interface

Each picker level follows the same contract:

typedef struct {
    void* (*init)(xce_scheduler_t* sched, void* parent_scope);
    void* (*next)(void* iter);       // returns scope or NULL
    void  (*fini)(void* iter);
} picker_iface_t;

Unified scheduling loop

void* sched_allocator(void* arg) {
    while (1) {
        // Level 0: project scope
        scope = project_picker.next(sched);
        if (!scope) { sleep(1); continue; }

        // Level 1: queue (scoped to project)
        queue = queue_picker.next(sched, scope);
        if (!queue) { project_picker.advance(); continue; }

        // Level 2: job (within queue)
        // Precedence: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin
        picker = resolve_job_picker(sched, queue);
        job = picker->elect_one(queue);

        if (!job) { queue_picker.advance(); continue; }

        // Allocation + preemption
        if (try_allocate(sched, job)) {
            dispatch(sched, job);
        } else if (can_preempt(sched, job)) {
            preempt_and_pend(sched, job);
        } else {
            pend(sched, job, "Resources");
        }
    }
}

This loop is identical for v1 and v2. v1 uses builtin_flat for Level 0. v2 replaces it with a quota-aware project picker.

Preemption flow

scheduler                          controller                  worker
───────────                        ──────────                  ──────
preempt_policy.select_victims(J)
  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim
  .victim_jobids[]                  deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                  victim's worker               release resources
  .preempting_jobid = J

J pended → next round → retry alloc

V1 uses SIGSTP (terminal stop — process groups paused, job stays in queue). Future escalation to SIGKILL kill-and-requeue is deferred.

What stays unchanged

• Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
• seqjob_allocator — wrapped with iterator interface, internally still 1host×1slot
• Select expression system — no changes
• COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
• Controller communication (handler.c opcodes, decision cbuf) — extended, not replaced
• HOOK_DETAIL_JOB_STOP — reused for preemption stop notifications

What's new

• XCE_DECISION_PREEMPT decision type
• xce_preempt_decision_t payload
• Three builtin plugins: builtin_flat, builtin_strict_priority, builtin_fcfs
• project_name on queue structs
• job_pick_plugin on queue + project structs
• preemptible_queues, preemptible, preempt_policy on queue struct
• project_pick_plugin, queue_pick_plugin, job_pick_plugin on scheduler struct

Scope

In

• Unified picker loop: project → queue → job with plugin slots at each level
• Three builtin plugins: builtin_flat (Level 0), builtin_strict_priority (Level 1), builtin_fcfs (Level 2)
• Picker plugin interface: init/next/fini — cancel-safe, session-aware
• Picker precedence chain: queue → project → global
• xce_alloc_plan_iter_t interface (header + implementation)
• xce_host_iter_t interface (header + implementation)
• parjob_allocator combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
• Preemption: XCE_DECISION_PREEMPT decision type, xce_preempt_decision_t payload, LSF-style cross-queue victim selection, SIGSTP mechanism via controller→worker pipeline
• Per-queue preemption config: preemptible_queues, preemptible, preempt_policy
• Session-level unsatisfiable resreq fingerprint cache
• Data structure changes to xce_scheduler_t, xce_sched_queue_t, xce_sched_project_t, decision.h
• Plugin interface contract documentation (cancel-safe, session-aware)
• Unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection
• Integration test: multi-slot parallel job allocation and preemption flow end-to-end

Out

• New job selection policy implementations beyond the three builtins (priority, fair-share, deadline, preemption, backfill, etc.) — the configurable slots enable them, but none ship in this spec
• Hierarchical project resource quotas (Level 0 full) — deferred to project-resource-ownership spec
• Resreq template registry (named/shared resreq objects) — future enhancement
• Select expression language changes
• Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
• Compound/ALTER resreq allocation changes
• GPU/affinity-aware allocation (pluggable later via allocator plugins)
• Kill-and-requeue preemption escalation (v1 is SIGSTP only, suspend-and-retry)
• Deadline field on xce_sched_job_t (needed for deadline-based pickers — deferred)

Future: Project Resource Ownership

Deferred to a separate project-resource-ownership spec. When implemented:

• Projects form a tree with resource quota at each node
• Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
• Level 1 queue picker is scoped to the selected project's queues
• Level 2 job picker sees only jobs from the selected project within each queue
• Project quota enforcement: projects at or over quota are skipped by Level 0
• The unified picker loop (above) requires zero code changes — only a new project picker plugin replaces builtin_flat

The architecture is designed so that v1 (flat project space) and v2 (hierarchical projects) share the identical scheduling loop.

Boundary Candidates

The work naturally splits into these implementation phases (single spec, sequential):

1. Unified picker loop + builtin plugins — implement the project → queue → job loop with builtin_flat, builtin_strict_priority, builtin_fcfs; retire sched_job_picker(); add plugin slots and precedence chain to structs
2. Iterator interfaces — define xce_alloc_plan_iter_t and xce_host_iter_t, implement for seqjob path first (trivial wrapper), validate the abstraction
3. Parallel job allocator — implement combinatorial plan generation in parjob_allocator, wire through host iterator
4. Preemption — implement XCE_DECISION_PREEMPT decision type, cross-queue victim selection policy, SIGSTP pipeline (scheduler→controller→worker), queue preemption config fields
5. Unsatisfiable resreq cache — session-level fingerprint cache, wire into scheduling loop (skip before allocation + preemption attempt)
6. Integration + testing — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, HA soak regression

Out of Boundary

• New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
• Resreq template sharing across jobs (future: resreq_registry spec)
• Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
• Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)

Upstream / Downstream

Upstream (dependencies)

• Existing plugin framework (plugin.h, plugin.c) — NO changes needed
• Existing select expression evaluator (resreq.h eval/Gc) — NO changes needed
• Existing shared resource tracking (sharedInst/sharedInUse) — NO changes needed
• Existing scheduler data structures (xce_scheduler_t, xce_sched_ctx_t) — minimal additions

Downstream (future consumers)

• New job selection policies — priority, fair-share, preemption plugins register via the configurable picker
• Affinity/GPU allocators — register as allocator plugins, implement plan iterator for topology-aware placement
• Commit barrier wiring — scheduling decisions flow through majority commit (separate hardening item)
• Resreq template registry — named resreqs shared across jobs, reference-counted

Existing Spec Touchpoints

Extends

• scheduler-controller-msg-gap — adjacent (message handling). The framework enhancement operates on the same xce_scheduler_t state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.

Adjacent

• ha-consistency-invariants — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
• worker-join-refactor — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.

Constraints

• C (gnu17) — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
• Non-blocking in plugin hooks — allocation happens on allocator thread, not iomgr. No new blocking concerns.
• Cancel-safe — job picker thread is pthread_cancel-able. Iterator next() calls must have cancellation points.
• Session isolation — iterator state and resreq cache are per-session, cleared on begin_session.
• Backward compatible — existing FCFS + seqjob behavior unchanged. New functionality is additive.
• Memory — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).