myxce/.kiro/specs/scheduler-core/brief.md

Brief: scheduler-core

Problem

The XCE scheduler has a plugin architecture but critical foundational gaps:

1. Job selection policy is hardcoded — sched_job_picker() in scheduler.c:39 uses the literal string "xce_sched_fcfs". The picker runs in a dedicated thread with jbuf cbuf communication. No way to swap policies without recompilation.

2. Monolithic scheduler tied to controller I/O — xce_scheduler_t holds both scheduling state and controller API context (apictx, cluster_cfg). No way to replay recorded events for debugging or test scheduling policies without a live cluster.

3. No picker plugin interface — the existing plugin framework has job_elect_fn and job_alloc_fn, but no defined contract for how the picker receives state-change events, manages sessions, or communicates decisions back. The FCFS picker is tightly coupled to the scheduler internals.

4. No project-level scheduling — xce_sched_project_t is just { char* name }. Projects have no resource ownership, quota, or scheduling policy.

Operators need configurable scheduling policies, and developers need a way to debug/tune scheduling behavior without a live cluster.

Current State

What exists

• Plugin framework (plugin.h, plugin.c): xce_sched_plugin_t with 9 hooks + job_elect_fn / job_alloc_fn
• FCFS job picker (fcfs.c): dedicated jobpicker thread. Iterates queues by priority, jobs by (priority desc, queue_time asc). Communicates via jbuf cbuf.
• Scheduling loop (scheduler.c): pick job → allocate → pend or dispatch → next
• Controller integration (handler.c): receives capnp push/notify → enqueues xce_sched_update_request_t → sched_update() dequeues. Decisions published via sched->apictx.
• Select expression memoization (select_eval_ctx_t): per-session AST caching

What's missing

• Pluggable job picker with defined interface contract
• Configurable picker name (no recompilation)
• Core/driver separation: scheduling logic decoupled from controller I/O
• Simulation mode: replay recorded events, virtual clock, deterministic execution
• Runtime recording for debugging/tuning
• State snapshot/restore for replay efficiency

Desired Outcome

1. Pluggable Job Picker

The core scheduling loop calls picker->next_job() to get the next schedulable job, with zero knowledge of how jobs are selected. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no jbuf cbuf.

Picker plugin interface (xce_job_picker_iface_t):

Category	Hooks	Purpose
Lifecycle	init, fini	Setup and teardown
State-change	on_queue_update, on_job_push, on_job_state_change, on_job_update	Called when events arrive
Session	begin_session, next_job, end_session	Scheduling round lifecycle
Post-allocation	on_dispatch, on_pend	After allocation attempt
Optional	on_config, on_job_rusage	Advanced pickers (NULL if unused)

2. Core/Driver Separation

Scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it:

Mode	Driver	Input	Output
Normal	Controller-connected	capnp push/notify → core events	Core decisions → controller
Simulation	Replay file	JSONL event file → core events	Core decisions → log

• Mode-agnostic core: no mode flag, no mode-specific code paths
• Virtualized clock: xce_core_now(core) with two-base + speed formula
• Discrete event-driven virtual clock: jumps to next event timestamp
• Snapshot + delta replay: load snapshot, replay delta events
• Deterministic guarantee: same initial state + same events = same decisions
• Runtime recording control: xceadmin scheduler record start/stop

3. Single Builtin Picker

builtin_fcfs internally implements queue-priority + FCFS job ordering — replicates today's behavior. Per-queue picker overrides supported.

Queue vs Project — Roles

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered?	WHO owns resources?
Key attributes	Priority, limits, time windows, per-queue picker override, preemption config	Quota, usage, hierarchy (future)
Cardinality	1 job → 1 queue	1 job → 1 project

Approach

Event-Driven Core with Pluggable Picker + Dual-Mode Driver:

Core scheduling loop

picker->begin_session(picker);
while ((job = picker->next_job(picker)) != NULL) {
    if (try_allocate(job) > 0) {
        picker->on_dispatch(picker, job->jobid);
        on_decision(driver_ctx, DISPATCH, job);
    } else if (can_preempt(job)) {
        on_decision(driver_ctx, PREEMPT, job, victims);
        picker->on_pend(picker, job->jobid);
    } else {
        on_decision(driver_ctx, PEND, job);
        picker->on_pend(picker, job->jobid);
    }
}
picker->end_session(picker);

Core knows nothing about project/queue/job ordering. next_job() hides all selection.

Clock virtualization

uint64_t xce_core_now(core) {
    wall_ns = clock_gettime(CLOCK_MONOTONIC);
    elapsed = wall_ns - core->real_clock_base;
    return core->virtual_clock_base + (elapsed * speed_num) / speed_den;
}

Normal: bases=0, speed=1/1 → returns wall clock. Simulation discrete: speed=0/1 → frozen at event ts.

What stays unchanged

• Plugin framework hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved
• Allocation logic — unchanged (in scheduler-allocation sibling spec)
• Controller communication in driver — extended, not replaced

What's new

• xce_job_picker_iface_t — pluggable picker with full hook specification
• xce_sched_core_t / xce_sched_driver_t / xce_sched_sim_driver_t
• xce_core_event_t — typed event interface
• xce_core_decision_cb_t — decision callback
• JSONL replay format + snapshot/restore
• xce_core_now() — virtualized clock accessor
• builtin_fcfs — single builtin picker
• job_pick_plugin on scheduler + per-queue override
• xceadmin recording commands

Scope

In

• Pluggable job picker interface with full hook specification
• Single builtin picker (builtin_fcfs)
• Configurable picker plugin with per-queue override
• Core/driver separation: passive core, normal driver, simulation driver
• Virtualized clock (xce_core_now() with two-base + speed formula)
• Discrete event-driven virtual clock
• JSONL replay format (HEADER + SNAPSHOT + events + DECISIONS)
• State snapshot/restore
• Deterministic replay guarantee
• Runtime recording control via xceadmin commands
• Data structures: xce_sched_core_t, xce_sched_driver_t, xce_sched_sim_driver_t, xce_core_event_t, xce_job_picker_iface_t
• Unit tests: picker hook behavior, simulation replay, snapshot/restore, deterministic verification
• HA soak regression for picker changes

Out

• Allocation iterators and parjob allocator (→ scheduler-allocation)
• Preemption policy and SIGSTP pipeline (→ scheduler-preemption)
• New picker policies beyond builtin_fcfs
• Hierarchical project resource quotas (→ project-resource-ownership)
• Dead field on xce_sched_job_t

Sibling Specs

• scheduler-allocation: Iterator-driven allocation, parjob allocator, resreq cache
• scheduler-preemption: Preemption policy, victim selection, SIGSTP pipeline

Constraints

• C (gnu17) — no C++, no external libraries beyond existing
• Cancel-safe — next_job() has pthread_cancel cancellation points
• Session isolation — picker state reset on begin_session
• Backward compatible — existing FCFS behavior unchanged
• Deterministic replay — same state + events → same decisions
• Mode-agnostic core — no mode flag, no mode-specific code paths
• No wall-clock time in core — all time through xce_core_now()