myxce/.kiro/specs/scheduler-framework/brief.md

Brief: scheduler-framework

Problem

The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:

1. Job selection policy is hardcoded — sched_job_picker() in scheduler.c:39 uses the literal string "xce_sched_fcfs". The picker runs in a dedicated thread with jbuf cbuf communication. No way to swap policies without recompilation.
2. Parallel job allocation is a stub — parjob_allocator.c:my_alloc() returns 0 with no logic. Jobs requiring multiple slots (maxTasks > 1) produce no allocation.
3. Allocation is one-shot — sched_make_allocation() tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
4. No preemption — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
5. No project-level scheduling — xce_sched_project_t is just { char* name }. Projects have no resource ownership, quota, or scheduling policy.
6. Monolithic scheduler — the scheduler is tightly coupled to controller I/O (capnp, apictx). No way to replay recorded events for debugging or test scheduling policies without a live cluster.

Operators need configurable scheduling policies, parallel job support, preemption, and a way to debug/tune scheduling behavior without a live cluster.

Current State

What exists

• Plugin framework (plugin.h, plugin.c): xce_sched_plugin_t with 9 hooks + job_elect_fn / job_alloc_fn. Plugins registered in key-value map by name.
• FCFS job picker (fcfs.c): dedicated jobpicker thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via jbuf cbuf.
• Seqjob allocator (seqjob_allocator.c): handles maxTasks == 1. Checks shared resources → collects workers → tries one host → returns first match.
• Parjob allocator stub (parjob_allocator.c): registered but my_alloc() is empty.
• Resource request types (resreq.h): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
• Select expression AST (resreq.h): full parser/evaluator with per-session memoization (select_eval_ctx_t) and inter-session GC.
• Shared resources (xce_scheduler_t.sharedInst/sharedInUse): zarray of resource values tracked per scheduling session.
• Scheduling loop (scheduler.c:sched_allocator()): pick job → sched_make_allocation() → pend or dispatch → next.
• Controller integration (handler.c): receives capnp push/notify → enqueues xce_sched_update_request_t → sched_update() dequeues. Decisions published via sched->apictx.

What's missing

• Configurable job picker: the plugin name is a hardcoded string literal
• Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
• Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
• Preemption: no mechanism to preempt lower-priority running jobs
• Project-level scheduling: xce_sched_project_t is just { char* name }
• Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
• Scheduler simulation/debugging: no way to replay recorded events, no virtual clock, no deterministic mode
• Core/driver separation: scheduling logic is tightly coupled to controller I/O

Desired Outcome

Three Architectural Foundations

1. Pluggable Job Picker — the core scheduling loop calls picker->next_job() to get the next schedulable job, with zero knowledge of how jobs are selected. The default builtin_fcfs picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no jbuf cbuf.

The picker plugin interface (xce_job_picker_iface_t):

Category	Hooks	Purpose
Lifecycle	init, fini	Setup and teardown
State-change	on_queue_update, on_job_push, on_job_state_change, on_job_update	Called by core when events arrive from driver
Session	begin_session, next_job, end_session	Scheduling round lifecycle
Post-allocation	on_dispatch, on_pend	Notification after allocation attempt
Optional	on_config, on_job_rusage	Advanced pickers (NULL if unused)

2. Iterator-Driven Allocation — lazy xce_alloc_plan_iter_t (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with lazy xce_host_iter_t (candidate hosts for a given plan). parjob_allocator implements real combinatorial multi-slot allocation. Session-level unsatisfiable resreq fingerprint cache avoids re-evaluating already-proven-unsatisfiable resreqs.

3. Core/Driver Separation — the scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it to the outside world:

Mode	Driver	Input	Output
Normal	Controller-connected	capnp push/notify → core events	Core decisions → controller
Simulation	Replay file	JSONL event file → core events	Core decisions → log (with --sim-verify diff against recorded)

• Discrete event-driven virtual clock: jumps directly to next event timestamp, no fixed tick, monotonic
• Snapshot + delta replay: load state snapshot, replay only subsequent delta events
• Deterministic guarantee: same initial state + same events = same decisions
• Runtime recording control: xceadmin scheduler record start/stop commands
• JSONL replay format: one event per line, debuggable, plain text
• What-if branching: load snapshot, replay alternative delta events

Queue vs Project — Roles

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered and constrained?	WHO owns resources and gets charged?
Scope	One hardware/scheduling profile	Cross-queue, cross-cluster
Key attributes	Priority, limits, time windows, per-queue picker override, preemption config	Quota, usage, hierarchy, members (future)
Cardinality	1 job → 1 queue, many-to-many	1 job → 1 project, many-to-many

One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware). The picker plugin may use queue->project_name internally for project-scoped iteration (v2).

Alternatives Considered

Approach A: Incremental Enhancement (rejected)

Aspect	Detail
Job selection	Add job_pick_plugin config field, same as chosen approach
Parallel jobs	Implement parjob allocator with pre-computed plan list. Eager generation
Iterator	No new abstraction. Simple for-loop over plan array
Shared resreq	xlist_t unsatisfiable_resreqs in session state

Rejected because: Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.

Approach B: Iterator-Driven Allocation + Picker Plugin ★ Chosen

Aspect	Detail
Job selection	Pluggable xce_job_picker_iface_t with next_job(). Single builtin_fcfs default. Per-queue picker override. Synchronous, no dedicated thread.
Iterator abstraction	xce_alloc_plan_iter_t + xce_host_iter_t — lazy, cancel-safe, composable
Parallel jobs	Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1
Host iterator	Per-plan lazy candidate host iteration using existing select expression eval
Preemption	XCE_DECISION_PREEMPT decision type, LSF-style cross-queue victim selection, SIGSTP via controller→worker
Shared resreq	Session-level fingerprint cache: {select, rusage, user, queue, project}
Core/driver split	Passive event-driven core + dual-mode driver (normal/simulation)
Simulation	JSONL replay, snapshot + delta, discrete event-driven virtual clock, deterministic

Chosen because: Lazy generation avoids memory pressure. Single next_job() interface makes pickers truly pluggable — the core loop never changes. Core/driver separation enables simulation mode for debugging/tuning without a live cluster. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc.

Approach C: Full Scheduler Framework (deferred)

Aspect	Detail
Job selection	Policy chain — multiple plugins score/veto jobs
Iterator	Triple composition: job iterator → plan iterator → host iterator
Resreq registry	Named primary resreqs registered globally, reference-counted
Allocation plugins	Different strategies as plugins: greedy, bin-packing, round-robin
Session state machine	Formalized STAGING → ELECT → ALLOCATE → COMMIT

Deferred (not rejected): The policy chain and resreq registry are valuable but lack current demand. Approach B's picker interface is designed to compose upward — a future policy-chain picker implements the same next_job() interface.

Decision Rationale

Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The picker plugin interface with next_job() keeps the core loop simple (5 lines) and makes pickers truly pluggable. Core/driver separation enables simulation mode — a capability that would be prohibitively expensive to retrofit later.

Design evolution (2026-07-10 → 2026-07-27): The original design specified a four-level architecture with three separate builtin plugins (builtin_flat, builtin_strict_priority, builtin_fcfs) and a three-level project→queue→job loop hardcoded in the core. Analysis of the existing FCFS plugin (fcfs.c) showed it already encapsulates all picking logic internally. The architecture was simplified to a single next_job() plugin interface with one builtin picker (builtin_fcfs) — the three-level decomposition is now an internal implementation detail of the plugin, not the core loop. Core/driver separation was added (Req 10) to enable simulation mode for policy debugging and tuning.

Approach

Iterator-Driven Allocation with Pluggable Picker + Core/Driver Separation:

Core scheduling loop

void* sched_allocator(void* arg) {
    while (1) {
        picker->begin_session(picker);

        xce_sched_job_t* job;
        while ((job = picker->next_job(picker)) != NULL) {
            if (try_allocate(job) > 0) {
                picker->on_dispatch(picker, job->jobid);
                emit_decision(DISPATCH, job);
            } else if (can_preempt(job)) {
                emit_decision(PREEMPT, job, victims);
                picker->on_pend(picker, job->jobid);
            } else {
                emit_decision(PEND, job);
                picker->on_pend(picker, job->jobid);
            }
        }

        picker->end_session(picker);
        sleep(1);
    }
}

The core loop has zero knowledge of project/queue/job ordering — all selection logic lives behind next_job().

Picker plugin interface

typedef struct xce_job_picker {
    // Lifecycle
    int  (*init)(void* picker, xce_sched_core_t* core);
    void (*fini)(void* picker);

    // State-change hooks (called by core when events arrive)
    int  (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
    int  (*on_job_push)(void* picker, xce_sched_job_t* job);
    int  (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
    int  (*on_job_update)(void* picker, xce_sched_job_t* job);

    // Session lifecycle
    int  (*begin_session)(void* picker);
    int  (*end_session)(void* picker);

    // Job selection
    xce_sched_job_t* (*next_job)(void* picker);

    // Post-allocation notification
    int  (*on_dispatch)(void* picker, uint64_t jobid);
    int  (*on_pend)(void* picker, uint64_t jobid);

    // Optional hooks (NULL = not implemented)
    int  (*on_config)(void* picker, xce_conf4sched_t* config);
    int  (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages);
} xce_job_picker_iface_t;

builtin_fcfs internal design

The default picker internally implements the equivalent of today's selection:

builtin_fcfs.next_job():
  Iterate queues by priority (descending)
    Skip queues with no ready jobs
    For each queue:
      If queue has per-queue picker override:
        Delegate to that picker
      Else:
        Iterate jobs by priority desc, queue_time asc (FCFS)
        Skip jobs not ready
        Return first ready job
  Return NULL if all queues exhausted

This is an implementation detail of the plugin, not visible to the core loop.

Preemption flow

scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next round → retry alloc

Core/Driver Architecture

┌──────────────────────────────────────────────────────────────┐
│                     DRIVER LAYER                               │
│                                                                │
│  Normal Mode                  Simulation Mode                  │
│  controller capnp ──┐         replay.jsonl ──┐                │
│  handler → events   │         virtual clock   │                │
│  → publish decisions│         → record/log     │                │
│       │       ▲     │              │       ▲   │                │
└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘
        │       │ decisions          │       │ decisions
        │       │ (callback)         │       │ (callback)
        ▼       │                    ▼       │
┌──────────────────────────────────────────────────────────────┐
│                   SCHEDULER CORE                               │
│                                                                │
│  core_push_event() → picker hooks → picker->next_job()        │
│  → try_allocate() → DISPATCH / PREEMPT / PEND                 │
│                                                                │
│  ZERO dependencies on: controller, capnp, network, wall-clock │
└──────────────────────────────────────────────────────────────┘

What stays unchanged

• Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved, but the picker no longer uses the dedicated job_elect_fn thread
• seqjob_allocator — wrapped with iterator interface, internally still 1host×1slot
• Select expression system — no changes
• COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
• Controller communication (handler.c opcodes, decision cbuf in driver) — extended, not replaced
• HOOK_DETAIL_JOB_STOP — reused for preemption stop notifications

What's new

• xce_job_picker_iface_t — pluggable picker interface with full hook specification
• xce_alloc_plan_iter_t + xce_host_iter_t — lazy iterator abstractions
• XCE_DECISION_PREEMPT decision type + xce_preempt_decision_t payload
• builtin_fcfs — single builtin picker (replaces three separate builtins from original design)
• xce_sched_core_t / xce_sched_driver_t / xce_sched_sim_driver_t — core/driver struct split
• xce_core_event_t — typed event interface between driver and core
• xce_core_decision_cb_t — decision callback (core → driver)
• JSONL replay file format + snapshot/restore
• Discrete event-driven virtual clock
• project_name on queue structs
• job_pick_plugin on scheduler + queue structs (per-queue override)
• preemptible_queues, preemptible, preempt_policy on queue struct

Scope

In

• Pluggable job picker interface (xce_job_picker_iface_t with next_job()) hiding all selection logic
• Single builtin picker (builtin_fcfs) replicating today's behavior
• Configurable picker plugin with per-queue override (queue->job_pick_plugin)
• Picker hook specification: lifecycle, state-change, session, post-allocation, optional hooks
• xce_alloc_plan_iter_t + xce_host_iter_t interfaces (lazy, cancel-safe, composable)
• parjob_allocator combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
• Preemption: XCE_DECISION_PREEMPT decision type, LSF-style cross-queue victim selection, SIGSTP mechanism
• Per-queue preemption config: preemptible_queues, preemptible, preempt_policy
• Session-level unsatisfiable resreq fingerprint cache
• Core/driver separation: passive event-driven core, normal-mode driver, simulation-mode driver
• Discrete event-driven virtual clock (monotonic, jumps to next event timestamp)
• Runtime recording control via xceadmin commands (scheduler record start/stop)
• JSONL replay file format (HEADER + SNAPSHOT + events + DECISIONS)
• State snapshot/restore for replay efficiency
• Deterministic replay guarantee for debugging/tuning
• What-if branching (load snapshot, replay alternative delta events)
• Data structure changes: xce_scheduler_t, xce_sched_queue_t, xce_sched_project_t, decision.h, xce_sched_core_t, xce_sched_driver_t, xce_sched_sim_driver_t, xce_core_event_t, xce_job_picker_iface_t
• Picker plugin interface contract documentation
• Unit tests: plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, event recording/replay, snapshot/restore, deterministic replay
• Integration test: multi-slot parallel job allocation and preemption flow end-to-end

Out

• New job selection policies beyond the builtin FCFS picker — the pluggable interface enables them, but none ship here
• Hierarchical project resource quotas — deferred to project-resource-ownership spec
• Resreq template registry — future enhancement
• Select expression language changes
• Controller-side scheduling state replication (commit barrier wiring)
• Compound/ALTER resreq allocation changes
• GPU/affinity-aware allocation (pluggable later via allocator plugins)
• Kill-and-requeue preemption escalation (v1 is SIGSTP only)
• Deadline field on xce_sched_job_t
• Event payload JSON schema (open design question — deferred to design phase)
• Distributed/cluster-mode simulation (v1 is single-scheduler replay only)

Future: Project Resource Ownership

Deferred to a separate project-resource-ownership spec. When implemented:

• Projects form a tree with resource quota at each node
• A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue
• Project quota enforcement: projects over quota are skipped by the picker
• The core scheduling loop requires zero code changes — only a new picker plugin replaces builtin_fcfs
• queue->project_name (added here) provides the queue→project mapping the v2 picker consumes
• xce_sched_project_t scaffolding (added here) provides the struct the v2 picker extends

The architecture is designed so that v1 (flat, builtin_fcfs ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop.

Boundary Candidates

The work naturally splits into these implementation phases (single spec, sequential):

1. Core/driver separation — split xce_scheduler_t into xce_sched_core_t + xce_sched_driver_t; define xce_core_event_t and xce_core_decision_cb_t; implement passive event-driven core interface; wire normal-mode driver to existing handler.c
2. Picker plugin interface + builtin_fcfs — define xce_job_picker_iface_t; implement builtin_fcfs with all MUST/SHOULD hooks; retire dedicated picker thread + jbuf; make core loop call next_job() synchronously; add job_pick_plugin config with per-queue override
3. Iterator interfaces — define xce_alloc_plan_iter_t and xce_host_iter_t, implement for seqjob path first (trivial wrapper), validate the abstraction
4. Parallel job allocator — implement combinatorial plan generation in parjob_allocator, wire through host iterator
5. Preemption — implement XCE_DECISION_PREEMPT decision type, cross-queue victim selection policy, SIGSTP pipeline, queue preemption config fields
6. Unsatisfiable resreq cache — session-level fingerprint cache, wire into scheduling loop
7. Simulation driver — implement simulation-mode driver with JSONL replay, virtual clock, snapshot/restore, deterministic verification
8. Recording + xceadmin commands — implement runtime recording control, periodic snapshots during recording
9. Integration + testing — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, simulation replay, HA soak regression

Out of Boundary

• New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that implement xce_job_picker_iface_t
• Resreq template sharing across jobs (future: resreq_registry spec)
• Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
• Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)

Upstream / Downstream

Upstream (dependencies)

• Existing plugin framework (plugin.h, plugin.c) — NO changes needed
• Existing select expression evaluator (resreq.h eval/Gc) — NO changes needed
• Existing shared resource tracking (sharedInst/sharedInUse) — NO changes needed
• Existing scheduler data structures (xce_scheduler_t, xce_sched_ctx_t) — split, not replaced

Downstream (future consumers)

• New job selection policies — priority, fair-share, deadline, backfill plugins implement xce_job_picker_iface_t
• Project-resource-ownership — project-aware picker plugin consumes queue->project_name and xce_sched_project_t
• Affinity/GPU allocators — register as allocator plugins, implement plan iterator for topology-aware placement
• Commit barrier wiring — scheduling decisions flow through majority commit (separate hardening item)
• Resreq template registry — named resreqs shared across jobs, reference-counted

Existing Spec Touchpoints

Extends

• scheduler-controller-msg-gap — adjacent (message handling). The framework enhancement operates on the same scheduling state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.

Adjacent

• ha-consistency-invariants — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
• worker-join-refactor — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.

Constraints

• C (gnu17) — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
• Non-blocking in plugin hooks — allocation happens on allocator thread, not iomgr. No new blocking concerns.
• Cancel-safe — next_job() must have pthread_cancel cancellation points
• Session isolation — iterator state and resreq cache are per-session, cleared on begin_session
• Backward compatible — existing FCFS + seqjob behavior unchanged. New functionality is additive.
• Deterministic replay — same initial state + same event sequence → same decisions. No wall-clock time, no unseeded randomness, stable iteration order.
• Memory — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).