myxce/.kiro/specs/scheduler-core/brief.md

# Brief: scheduler-core

## Problem

The XCE scheduler has a plugin architecture but critical foundational gaps:

1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation.

2. **Monolithic scheduler tied to controller I/O** — `xce_scheduler_t` holds both scheduling state and controller API context (`apictx`, `cluster_cfg`). No way to replay recorded events for debugging or test scheduling policies without a live cluster.

3. **No picker plugin interface** — the existing plugin framework has `job_elect_fn` and `job_alloc_fn`, but no defined contract for how the picker receives state-change events, manages sessions, or communicates decisions back. The FCFS picker is tightly coupled to the scheduler internals.

4. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy.

Operators need configurable scheduling policies, and developers need a way to debug/tune scheduling behavior without a live cluster.

## Current State

### What exists
- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn`
- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs by (priority desc, queue_time asc). Communicates via `jbuf` cbuf.
- **Scheduling loop** (`scheduler.c`): pick job → allocate → pend or dispatch → next
- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`.
- **Select expression memoization** (`select_eval_ctx_t`): per-session AST caching

### What's missing
- Pluggable job picker with defined interface contract
- Configurable picker name (no recompilation)
- Core/driver separation: scheduling logic decoupled from controller I/O
- Simulation mode: replay recorded events, virtual clock, deterministic execution
- Runtime recording for debugging/tuning
- State snapshot/restore for replay efficiency

## Desired Outcome

### 1. Pluggable Job Picker

The core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The picker retains today's dedicated thread, communicating with the allocator thread via the existing `jbuf` cbuf (superseded by the full three-thread producer/consumer contract in requirements.md Req 1 AC 3-4 — this brief predates that decision).

Picker plugin interface (`xce_job_picker_iface_t`):

| Category | Hooks | Purpose |
|----------|-------|---------|
| Lifecycle | `init`, `fini` | Setup and teardown |
| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called when events arrive |
| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle |
| Post-allocation | `on_dispatch`, `on_pend` | After allocation attempt |
| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) |

### 2. Core/Driver Separation

Scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it:

| Mode | Driver | Input | Output |
|------|--------|-------|--------|
| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller |
| Simulation | Replay file | `.replay` capnp binary stream → core events | Core decisions → log |

- **Mode-agnostic core**: no mode flag, no mode-specific code paths
- **Virtualized clock**: `xce_core_now(core)` with two-base + speed formula
- **Discrete event-driven virtual clock**: jumps to next event timestamp
- **Snapshot + delta replay**: load snapshot, replay delta events
- **Deterministic guarantee**: same initial state + same events = same decisions
- **Runtime recording control**: xceadmin `scheduler record start/stop`

### 3. Single Builtin Picker

`builtin_fcfs` internally implements queue-priority + FCFS job ordering — replicates today's behavior.
Per-queue picker overrides supported.

### Queue vs Project — Roles

| | Queue | Project |
|---|---|---|
| **Role** | Scheduling mechanism | Resource ownership |
| **Question** | HOW are jobs ordered? | WHO owns resources? |
| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy (future) |
| **Cardinality** | 1 job → 1 queue | 1 job → 1 project |

## Approach

**Event-Driven Core with Pluggable Picker + Dual-Mode Driver**:

### Core scheduling loop

```c
picker->begin_session(picker);
while ((job = picker->next_job(picker)) != NULL) {
    if (try_allocate(job) > 0) {
        picker->on_dispatch(picker, job->jobid);
        on_decision(driver_ctx, DISPATCH, job);
    } else if (can_preempt(job)) {
        on_decision(driver_ctx, PREEMPT, job, victims);
        picker->on_pend(picker, job->jobid);
    } else {
        on_decision(driver_ctx, PEND, job);
        picker->on_pend(picker, job->jobid);
    }
}
picker->end_session(picker);
```

Core knows nothing about project/queue/job ordering. `next_job()` hides all selection.

### Clock virtualization

```c
uint64_t xce_core_now(core) {
    wall_ns = clock_gettime(CLOCK_MONOTONIC);
    elapsed = wall_ns - core->real_clock_base;
    return core->virtual_clock_base + (elapsed * speed_num) / speed_den;
}
```

Normal: bases=0, speed=1/1 → returns wall clock. Simulation discrete: speed=0/1 → frozen at event ts.

### What stays unchanged
- Plugin framework hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved
- Allocation logic — unchanged (in scheduler-allocation sibling spec)
- Controller communication in driver — extended, not replaced

### What's new
- `xce_job_picker_iface_t` — pluggable picker with full hook specification
- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t`
- `xce_core_event_t` — typed event interface
- `xce_core_decision_cb_t` — decision callback
- `.replay` capnp binary stream format + snapshot/restore
- `xce_core_now()` — virtualized clock accessor
- `builtin_fcfs` — single builtin picker
- `job_pick_plugin` on scheduler + per-queue override
- xceadmin recording commands

## Scope

### In
- Pluggable job picker interface with full hook specification
- Single builtin picker (`builtin_fcfs`)
- Configurable picker plugin with per-queue override
- Core/driver separation: passive core, normal driver, simulation driver
- Virtualized clock (`xce_core_now()` with two-base + speed formula)
- Discrete event-driven virtual clock
- `.replay` capnp binary stream format (HEADER + SNAPSHOT + events + DECISIONS)
- State snapshot/restore
- Deterministic replay guarantee
- Runtime recording control via xceadmin commands
- Data structures: `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t`
- Unit tests: picker hook behavior, simulation replay, snapshot/restore, deterministic verification
- HA soak regression for picker changes

### Out
- Allocation iterators and parjob allocator (→ scheduler-allocation)
- Preemption policy and SIGSTP pipeline (→ scheduler-preemption)
- New picker policies beyond builtin_fcfs
- Hierarchical project resource quotas (→ project-resource-ownership)
- Dead field on `xce_sched_job_t`

## Sibling Specs
- **scheduler-allocation**: Iterator-driven allocation, parjob allocator, resreq cache
- **scheduler-preemption**: Preemption policy, victim selection, SIGSTP pipeline

## Constraints
- **C (gnu17)** — no C++, no external libraries beyond existing
- **Cancel-safe** — `next_job()` has pthread_cancel cancellation points
- **Session isolation** — picker state reset on begin_session
- **Backward compatible** — existing FCFS behavior unchanged
- **Deterministic replay** — same state + events → same decisions
- **Mode-agnostic core** — no mode flag, no mode-specific code paths
- **No wall-clock time in core** — all time through xce_core_now()