myxce/.kiro/specs/scheduler-core/brief.md

Brief: scheduler-core

Problem

The XCE scheduler has a plugin architecture but critical foundational gaps:

1. Job selection policy is hardcoded — sched_job_picker() in scheduler.c:39 uses the literal string "xce_sched_fcfs". The picker runs in a dedicated thread with jbuf cbuf communication. No way to swap policies without recompilation.

2. Monolithic scheduler tied to controller I/O — xce_scheduler_t holds both scheduling state and controller API context (apictx, cluster_cfg). No way to replay recorded events for debugging or test scheduling policies without a live cluster.

3. No picker plugin interface — the existing plugin framework has job_elect_fn and job_alloc_fn, but no defined contract for how the picker receives state-change events, manages sessions, or communicates decisions back. The FCFS picker is tightly coupled to the scheduler internals.

4. No project-level scheduling — xce_sched_project_t is just { char* name }. Projects have no resource ownership, quota, or scheduling policy.

Operators need configurable scheduling policies, and developers need a way to debug/tune scheduling behavior without a live cluster.

Current State

What exists

• Plugin framework (plugin.h, plugin.c): xce_sched_plugin_t with 9 hooks + job_elect_fn / job_alloc_fn
• FCFS job picker (fcfs.c): dedicated jobpicker thread. Iterates queues by priority, jobs by (priority desc, queue_time asc). Communicates via jbuf cbuf.
• Scheduling loop (scheduler.c): pick job → allocate → pend or dispatch → next
• Controller integration (handler.c): receives capnp push/notify → enqueues xce_sched_update_request_t → sched_update() dequeues. Decisions published via sched->apictx.
• Select expression memoization (select_eval_ctx_t): per-session AST caching

What's missing

• Pluggable job picker with defined interface contract
• Configurable picker name (no recompilation)
• Core/driver separation: scheduling logic decoupled from controller I/O
• Simulation mode: replay recorded events, virtual clock, deterministic execution
• Runtime recording for debugging/tuning
• State snapshot/restore for replay efficiency

Desired Outcome

1. Pluggable Job Picker

The core scheduling loop calls picker->next_job() to get the next schedulable job, with zero knowledge of how jobs are selected. The picker retains today's dedicated thread, communicating with the allocator thread via the existing jbuf cbuf (superseded by the full three-thread producer/consumer contract in requirements.md Req 1 AC 3-4 — this brief predates that decision).

Picker plugin interface (xce_job_picker_iface_t):

Category	Hooks	Purpose
Lifecycle	init, fini	Setup and teardown
State-change	on_queue_update, on_job_push, on_job_state_change, on_job_update	Called when events arrive
Session	begin_session, next_job, end_session	Scheduling round lifecycle
Post-allocation	on_dispatch, on_pend	After allocation attempt
Optional	on_config, on_job_rusage	Advanced pickers (NULL if unused)

2. Core/Driver Separation

Scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it:

Mode	Driver	Input	Output
Normal	Controller-connected	capnp push/notify → core events	Core decisions → controller
Simulation	Replay file	.replay capnp binary stream → core events	Core decisions → log

• Mode-agnostic core: no mode flag, no mode-specific code paths
• Virtualized clock: xce_core_now(core) with two-base + speed formula
• Discrete event-driven virtual clock: jumps to next event timestamp
• Snapshot + delta replay: load snapshot, replay delta events
• Deterministic guarantee: same initial state + same events = same decisions
• Runtime recording control: xceadmin scheduler record start/stop

3. Single Builtin Picker

builtin_fcfs internally implements queue-priority + FCFS job ordering — replicates today's behavior. Per-queue picker overrides supported.

Queue vs Project — Roles

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered?	WHO owns resources?
Key attributes	Priority, limits, time windows, per-queue picker override, preemption config	Quota, usage, hierarchy (future)
Cardinality	1 job → 1 queue	1 job → 1 project

Approach

Event-Driven Core with Pluggable Picker + Dual-Mode Driver:

Core scheduling loop

picker->begin_session(picker);
while ((job = picker->next_job(picker)) != NULL) {
    if (try_allocate(job) > 0) {
        picker->on_dispatch(picker, job->jobid);
        on_decision(driver_ctx, DISPATCH, job);
    } else if (can_preempt(job)) {
        on_decision(driver_ctx, PREEMPT, job, victims);
        picker->on_pend(picker, job->jobid);
    } else {
        on_decision(driver_ctx, PEND, job);
        picker->on_pend(picker, job->jobid);
    }
}
picker->end_session(picker);

Core knows nothing about project/queue/job ordering. next_job() hides all selection.

Clock virtualization

uint64_t xce_core_now(core) {
    wall_ns = clock_gettime(CLOCK_MONOTONIC);
    elapsed = wall_ns - core->real_clock_base;
    return core->virtual_clock_base + (elapsed * speed_num) / speed_den;
}

Normal: bases=0, speed=1/1 → returns wall clock. Simulation discrete: speed=0/1 → frozen at event ts.

What stays unchanged

• Plugin framework hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved
• Allocation logic — unchanged (in scheduler-allocation sibling spec)
• Controller communication in driver — extended, not replaced

What's new

• xce_job_picker_iface_t — pluggable picker with full hook specification
• xce_sched_core_t / xce_sched_driver_t / xce_sched_sim_driver_t
• xce_core_event_t — typed event interface
• xce_core_decision_cb_t — decision callback
• .replay capnp binary stream format + snapshot/restore
• xce_core_now() — virtualized clock accessor
• builtin_fcfs — single builtin picker
• job_pick_plugin on scheduler + per-queue override
• xceadmin recording commands

Scope

In

• Pluggable job picker interface with full hook specification
• Single builtin picker (builtin_fcfs)
• Configurable picker plugin with per-queue override
• Core/driver separation: passive core, normal driver, simulation driver
• Virtualized clock (xce_core_now() with two-base + speed formula)
• Discrete event-driven virtual clock
• .replay capnp binary stream format (HEADER + SNAPSHOT + events + DECISIONS)
• State snapshot/restore
• Deterministic replay guarantee
• Runtime recording control via xceadmin commands
• Data structures: xce_sched_core_t, xce_sched_driver_t, xce_sched_sim_driver_t, xce_core_event_t, xce_job_picker_iface_t
• Unit tests: picker hook behavior, simulation replay, snapshot/restore, deterministic verification
• HA soak regression for picker changes

Out

• Allocation iterators and parjob allocator (→ scheduler-allocation)
• Preemption policy and SIGSTP pipeline (→ scheduler-preemption)
• New picker policies beyond builtin_fcfs
• Hierarchical project resource quotas (→ project-resource-ownership)
• Dead field on xce_sched_job_t

Sibling Specs

• scheduler-allocation: Iterator-driven allocation, parjob allocator, resreq cache
• scheduler-preemption: Preemption policy, victim selection, SIGSTP pipeline

Constraints

• C (gnu17) — no C++, no external libraries beyond existing
• Cancel-safe — next_job() has pthread_cancel cancellation points
• Session isolation — picker state reset on begin_session
• Backward compatible — existing FCFS behavior unchanged
• Deterministic replay — same state + events → same decisions
• Mode-agnostic core — no mode flag, no mode-specific code paths
• No wall-clock time in core — all time through xce_core_now()