myxce/.kiro/specs/scheduler-framework/requirements.md

Requirements Document

Introduction

The XCE scheduler has a plugin architecture (plugin.h/plugin.c, xce_sched_plugin_t) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: sched_job_picker() (scheduler.c:39) sets job_pick_plugin = "xce_sched_fcfs" as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: parjob_allocator.c:73's my_alloc() takes (plugin, ctx) and immediately return 0 — any job with maxTasks > 1 (routed there by sched_make_primary_alloc_() at scheduler.c:93-103) produces no allocation at all. Third, allocation is eager and one-shot: the sibling seqjob_allocator.c:107 my_alloc() (used when maxTasks == 1) calls sched_collect_workers() then xlist_find() for the first host that satisfies the request and stops — there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.

This feature closes all three gaps with an iterator-driven allocation architecture: a lazy xce_alloc_plan_iter_t (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with a lazy xce_host_iter_t (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a parjob_allocator implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session.

Beyond the three gaps, this feature introduces a four-level scheduling architecture that separates concerns cleanly and composes upward:

Level	Picker	Question it answers	Plugin slot	Default (v1)
0	Project picker	WHO gets resources this round?	sched->project_pick_plugin	"builtin_flat" — single "default" project
1	Inter-queue picker	WHICH queue within the project?	sched->queue_pick_plugin	"builtin_strict_priority"
2	Intra-queue picker	WHICH job from this queue?	queue→project→sched precedence chain	"builtin_fcfs"
3	Preemption policy	WHAT to preempt when allocation fails?	queue->preempt_policy	"lowest_priority_first"

The scheduling loop is always project → queue → job, with Level 0 collapsing to a single implicit "default" project in v1 and expanding to hierarchical resource ownership in a future spec (project-resource-ownership). Level 3 (preemption) fires only when allocation fails for a preemptible job.

Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in .kiro/specs/scheduler-framework/brief.md.

Queue vs Project — Roles

These are fundamentally different concerns:

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered and constrained?	WHO owns resources and gets charged?
Scope	One hardware/scheduling profile	Cross-queue, cross-cluster
Key attributes	Priority, limits, time windows, picker, preemption config	Quota, usage, hierarchy, members (future)
Cardinality to job	1 job → 1 queue	1 job → 1 project
Cardinality to each other	Many-to-many	Many-to-many

• Queue = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, inter-queue priority, and the intra-queue job picker.
• Project = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. One project submits to many queues; one queue receives from many projects.

Boundary Context

• In scope: Unified picker loop (project → queue → job) with plugin slots at each level; xce_alloc_plan_iter_t and xce_host_iter_t interfaces (lazy, cancel-safe, composable); seqjob_allocator rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, maxTasks == 1 only); parjob_allocator's my_alloc() implementing real combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1) for maxTasks > 1; three builtin plugins (builtin_flat, builtin_strict_priority, builtin_fcfs) replicating today's behavior through the new architecture; configurable picker slots with precedence chain (queue → project → global); preemption decision type (XCE_DECISION_PREEMPT) flowing scheduler→controller→worker with SIGSTP mechanism; cross-queue victim selection (LSF-style per-queue preemptible_queues list); session-level unsatisfiable-resreq fingerprint cache keyed on {select, rusage, user, queue, project}; documentation of plugin interface contracts (cancel-safe, session-aware); unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection; integration test for end-to-end multi-slot parallel job allocation and preemption flow.

• Out of scope: New job selection policies beyond the three builtins (priority, fair-share, deadline, preemption, backfill) — the configurable slots enable them, but none ship in this spec beyond the builtins; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring for scheduling decisions (separate roadmap hardening item, tracked adjacent to ha-consistency-invariants); COMPOUND/ALTER resreq allocation changes beyond what composing over PRIMARY already provides; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins); hierarchical project resource quotas (Level 0 full — deferred to project-resource-ownership spec); kill-and-requeue preemption escalation (deferred — v1 is SIGSTP only); deadline field on xce_sched_job_t (deferred — needed for deadline-based pickers).

• Adjacent expectations: depends on the existing plugin framework (plugin.h/plugin.c), the existing select-expression evaluator (resreq.h, memoized via select_eval_ctx_t), and existing shared-resource tracking (xce_scheduler_t.sharedInst/sharedInUse) — none of these change. scheduler-controller-msg-gap (implemented) governs controller↔scheduler message handling this feature reads state through but does not modify. ha-consistency-invariants' commit barrier is a future consumer of scheduling decisions; this feature does not wire it. Controller hook HOOK_DETAIL_JOB_STOP (enum value 65 in cluster_def.h) already exists and is reused for preemption stop actions.

Requirements

Requirement 1: Unified picker loop (project → queue → job)

Objective: As a maintainer of the scheduler, I want a single scheduling loop that composes project, queue, and job pickers in a fixed order, so that v1 (flat project space) and v2 (hierarchical projects with resource quotas) share the same code path.

Acceptance Criteria

1. The Scheduler shall implement a scheduling loop that always follows the order: Level 0 (project picker) → Level 1 (inter-queue picker) → Level 2 (intra-queue picker) → allocation.
2. Each picker level shall conform to a common interface contract: init(sched, parent_scope), next(iter) → scope_or_NULL, fini(iter).
3. When a picker at any level returns NULL (exhausted), the loop shall advance the parent picker to its next scope and restart the child picker from that new scope.
4. In v1 (no project hierarchy), the default project picker builtin_flat shall yield exactly one scope: a "default" project containing all configured queues, so the loop behavior is identical to today's flat queue iteration.
5. Plugin resolution for Level 2 shall follow the precedence chain: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin → hardcoded default "builtin_fcfs".
6. sched_job_picker() shall be retired and replaced by the unified loop; the hardcoded "xce_sched_fcfs" literal at scheduler.c:43 shall be removed.

Requirement 2: Lazy allocation plan iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front.

Acceptance Criteria

1. The Scheduler shall define an xce_alloc_plan_iter_t interface with init(resreq, sched), next() → plan, and fini() operations.
2. Each call to next() shall generate exactly one candidate (nhosts, slots_per_host, host_filter) plan without generating or materializing any other plan in the same call.
3. The Scheduler shall order plans returned by successive next() calls from most-preferred to least-preferred, per the owning allocator's preference policy.
4. next() shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion.
5. Calling fini() shall release all resources held by the iterator, and the interface shall define no other required cleanup step.

Requirement 3: Lazy host candidate iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering.

Acceptance Criteria

1. The Scheduler shall define an xce_host_iter_t interface with init(plan, workers), next() → worker, and fini() operations.
2. Each call to next() shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.
3. next() shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call next() again after receiving NULL for that plan.
4. The host iterator shall compose with the plan iterator: a new xce_host_iter_t shall be constructible from any plan produced by Requirement 2's xce_alloc_plan_iter_t.next().
5. Calling fini() shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.

Requirement 4: Parallel job combinatorial allocation

Objective: As a cluster operator submitting a job that needs multiple task slots, I want parjob_allocator to actually allocate across hosts, so that jobs with maxTasks > 1 are no longer silently unallocatable.

Acceptance Criteria

1. When a job's xce_primary_resreq_t.maxTasks > 1 reaches parjob_allocator's job_alloc_fn (my_alloc() in parjob_allocator.c), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host × N slots, 2 hosts × ⌈N/2⌉ slots, …, N hosts × 1 slot, using Requirement 2's plan iterator.
2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan.
3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job.
4. When no plan among those generated in Criterion 1 can be satisfied, my_alloc() shall return the same not-satisfied signal (0) that the current stub returns today, so pend-reason handling in sched_make_primary_alloc_() is unaffected.
5. parjob_allocator's job_alloc_fn shall no longer unconditionally return 0 regardless of ctx contents — the current stub behavior — for any job with maxTasks > 1.

Requirement 5: Configurable picker plugin slots

Objective: As a cluster operator, I want to configure job-picking policies at the scheduler, queue, and project levels without recompiling the scheduler, so that FCFS, priority, or custom pickers can be swapped operationally and composed.

Acceptance Criteria

1. The Scheduler shall add a project_pick_plugin field to xce_scheduler_t, defaulting to "builtin_flat" when unset.
2. The Scheduler shall add a queue_pick_plugin field to xce_scheduler_t, defaulting to "builtin_strict_priority" when unset.
3. The Scheduler shall add a job_pick_plugin field to xce_scheduler_t, defaulting to "builtin_fcfs" when unset — serving as the global fallback.
4. The Scheduler shall add a job_pick_plugin field to xce_sched_queue_t, defaulting to NULL (inherit from project or global).
5. The Scheduler shall add a job_pick_plugin field to xce_sched_project_t, defaulting to NULL (inherit from global).
6. The Scheduler shall add a project_name field to xce_sched_queue_t, set from queue configuration, establishing the queue→project relationship for Level 1 scoping.
7. The Scheduler shall accept all picker plugin names from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild.
8. When a plugin slot names a plugin not registered via the existing plugin_register/key_find mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default.
9. The Scheduler shall document the picker plugin interface contract: each picker must be cancel-safe (responds to pthread_cancel at defined points) and session-aware (resets its state on end_session).

Requirement 6: Preemption policy and execution

Objective: As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention.

Acceptance Criteria

1. The Scheduler shall add a XCE_DECISION_PREEMPT opcode to xce_decision_op_t in decision.h, with a payload type xce_preempt_decision_t carrying victim_jobids[], n_victims, signal (SIGSTP in v1), and preempting_jobid.
2. When allocation fails for a job J whose queue has a non-NULL preemptible_queues list, the Scheduler shall invoke the queue's preempt_policy plugin to select victim jobs from the target queues.
3. The default preemption policy "lowest_priority_first" shall: collect running jobs from target queues, filter to those with priority < J.priority and queue->preemptible == true, rank by priority ascending, and select the minimum subset whose freed resources satisfy J's resource request.
4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via sched_commit_decision(XCE_DECISION_PREEMPT), publish it through the controller→worker pipeline, and pend job J for the next scheduling round (async preemption — the job is picked up after resources are freed).
5. When no sufficient victim set exists, the Scheduler shall pend job J with a resource-shortage reason without invoking preemption.
6. The worker receiving a preemption action shall send the specified signal (SIGSTP) to the victim job's process groups, and the controller shall transition the victim job's state to JOB_SUSPENDED.
7. The Scheduler shall add preemptible_queues (list of queue names this queue CAN preempt from), preemptible (bool — CAN this queue's jobs BE preempted by others), and preempt_policy (victim-selection plugin name, NULL = default) fields to xce_sched_queue_t.
8. The existing HOOK_DETAIL_JOB_STOP hook detail enum shall be reused for preemption stop notifications.
9. Preemption victim selection shall never select a job whose priority is greater than or equal to the preempting job's priority, nor a job whose queue has preemptible == false.

Requirement 7: Session-level unsatisfiable resreq cache

Objective: As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.

Acceptance Criteria

1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on xce_scheduler_t, keyed by a fingerprint of {select_string, rusage_string, user, queue, project}.
2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (begin_session).
3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation AND preemption (Requirement 6) also fails or is inapplicable, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job.
4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption.
5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.

Requirement 8: Builtin default plugins

Objective: As a maintainer, I want the new architecture to ship with builtin plugins that replicate today's exact scheduling behavior, so that operators who don't configure custom pickers see no behavioral change.

Acceptance Criteria

1. builtin_flat (Level 0, project picker): shall yield exactly one "default" project scope per scheduling round. All queues not explicitly assigned to a project belong to this default project. Always eligible; never exhausted during a session.
2. builtin_strict_priority (Level 1, inter-queue picker): shall iterate queues in descending priority order (queue->priority). Shall skip queues with no pending jobs (sched_queue_is_ready returns false). When scoped to a project, shall only iterate queues belonging to that project. Shall return NULL when all eligible queues are exhausted.
3. builtin_fcfs (Level 2, intra-queue picker): shall replicate today's FCFS behavior — iterate pending jobs within the queue by priority desc, queue_time asc. Shall skip jobs where sched_job_is_ready returns false. Shall communicate the picked jobid via the existing jbuf cbuf mechanism. Shall be cancel-safe and session-aware.
4. All three builtin plugins shall be registered automatically at scheduler startup without requiring explicit configuration.

Requirement 9: No regression to existing allocators, job picker, or HA soak

Objective: As a maintainer, I want the iterator-driven rework and picker architecture scoped so that existing single-slot allocation and FCFS behavior are unaffected, so that this framework change is safe to ship independently of any new policy or allocator.

Acceptance Criteria

1. seqjob_allocator's externally observable allocation behavior (which host a maxTasks == 1 job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the xce_host_iter_t interface.
2. builtin_fcfs shall produce the same job-picking order as today's fcfs_elect_job, given the same queue and job state.
3. builtin_strict_priority shall produce the same inter-queue order as today's fcfs_elect_job queue iteration loop.
4. builtin_flat shall produce the same effective behavior as today's lack of project scoping — all queues in one flat namespace.
5. The COMPOUND and ALTER resreq allocation paths (sched_make_compound_allocation, sched_make_alter_allocation in scheduler.c) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today.
6. Existing scheduler unit tests (scheduler meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string or eager-one-shot behavior this feature replaces.
7. A full HA soak (sequential and chaotic /xce-test-ha regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.

Data Structure Changes Summary

xce_scheduler_t additions

char* project_pick_plugin;   // default: "builtin_flat"
char* queue_pick_plugin;     // default: "builtin_strict_priority"
char* job_pick_plugin;       // default: "builtin_fcfs" (global fallback)
// unsatisfiable resreq cache (Req 7)

xce_sched_queue_t additions

char*    project_name;          // owning project (for Level 1 scoping)
char*    job_pick_plugin;       // intra-queue picker (NULL = inherit)
char**   preemptible_queues;    // queues this queue CAN preempt from
int      n_preemptible_queues;
bool     preemptible;           // CAN this queue's jobs BE preempted?
char*    preempt_policy;        // victim selection plugin (NULL = default)

xce_sched_project_t additions

char*    job_pick_plugin;       // per-project picker (NULL = inherit, v1 field, v2 logic)
int32_t  priority;              // inter-project ordering (future spec)
// future: resource_quota, resource_used, parent_project (project-resource-ownership spec)

decision.h additions

typedef enum {
    XCE_DECISION_NONE = 0,
    XCE_DECISION_DISPATCH,
    XCE_DECISION_PEND,
    XCE_DECISION_PREEMPT,     // NEW
    XCE_DECISION_NEW_SESSION
} xce_decision_op_t;

typedef struct {
    uint64_t* victim_jobids;    // jobs to preempt
    int       n_victims;
    int       signal;           // SIGSTP (v1), SIGKILL (future)
    uint64_t  preempting_jobid; // job that triggered preemption
} xce_preempt_decision_t;

Architecture Diagram

┌──────────────────────────────────────────────────────────┐
│ Level 0: Project Picker    [sched->project_pick_plugin]   │
│                                                            │
│ v1 (builtin_flat): yields one "default" project            │
│ v2 (future): quota-aware project selection                 │
└──────────────────────┬───────────────────────────────────┘
                       │ project scope
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin]    │
│                                                            │
│ Iterates queues scoped to the selected project             │
│ Default: strict_priority (descending queue->priority)      │
└──────────────────────┬───────────────────────────────────┘
                       │ queue
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Level 2: Intra-Queue Picker  [queue→project→sched]        │
│                                                            │
│ Picks one job from the selected queue                      │
│ Default: builtin_fcfs (priority desc, queue_time asc)      │
└──────────────────────┬───────────────────────────────────┘
                       │ job
                       ▼
┌──────────────────────────────────────────────────────────┐
│ Allocation + Preemption                                   │
│                                                            │
│ plan_iter → host_iter → try_alloc                         │
│   success → dispatch                                      │
│   fail → Level 3: preempt? → select victims → SIGSTP     │
│         → pend (no preemption possible)                   │
└──────────────────────────────────────────────────────────┘

Preemption Flow

scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next scheduling round → allocation retries on freed resources

Future: Project Resource Ownership (Level 0 full)

Deferred to a separate project-resource-ownership spec. When implemented:

• Projects form a tree with resource quota at each node
• Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
• Level 1 queue picker is scoped to the selected project's queues
• Level 2 job picker sees only jobs from the selected project within each queue
• Project quota enforcement: projects over quota are skipped by Level 0
• The unified picker loop (Requirement 1) requires zero code changes — only a new project picker plugin replaces builtin_flat

The architecture is designed so that v1 (flat) and v2 (hierarchical) share the identical scheduling loop. The only difference is which project picker plugin is configured.