myxce/.kiro/specs/scheduler-framework/brief.md

# Brief: scheduler-framework

## Problem
The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation.
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy.
6. **Monolithic scheduler** — the scheduler is tightly coupled to controller I/O (capnp, `apictx`). No way to replay recorded events for debugging or test scheduling policies without a live cluster.

Operators need configurable scheduling policies, parallel job support, preemption, and a way to debug/tune scheduling behavior without a live cluster.

## Current State

### What exists
- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn`. Plugins registered in key-value map by name.
- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs within queue by (priority desc, queue_time asc). Communicates picked jobid via `jbuf` cbuf.
- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources → collects workers → tries one host → returns first match.
- **Parjob allocator stub** (`parjob_allocator.c`): registered but `my_alloc()` is empty.
- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds).
- **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC.
- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session.
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`.

### What's missing
- Configurable job picker: the plugin name is a hardcoded string literal
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
- Preemption: no mechanism to preempt lower-priority running jobs
- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }`
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
- Scheduler simulation/debugging: no way to replay recorded events, no virtual clock, no deterministic mode
- Core/driver separation: scheduling logic is tightly coupled to controller I/O

## Desired Outcome

### Three Architectural Foundations

**1. Pluggable Job Picker** — the core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no `jbuf` cbuf.

The picker plugin interface (`xce_job_picker_iface_t`):

| Category | Hooks | Purpose |
|----------|-------|---------|
| Lifecycle | `init`, `fini` | Setup and teardown |
| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called by core when events arrive from driver |
| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle |
| Post-allocation | `on_dispatch`, `on_pend` | Notification after allocation attempt |
| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) |

**2. Iterator-Driven Allocation** — lazy `xce_alloc_plan_iter_t` (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with lazy `xce_host_iter_t` (candidate hosts for a given plan). `parjob_allocator` implements real combinatorial multi-slot allocation. Session-level unsatisfiable resreq fingerprint cache avoids re-evaluating already-proven-unsatisfiable resreqs.

**3. Core/Driver Separation** — the scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it to the outside world:

| Mode | Driver | Input | Output |
|------|--------|-------|--------|
| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller |
| Simulation | Replay file | JSONL event file → core events | Core decisions → log (with `--sim-verify` diff against recorded) |

- **Discrete event-driven virtual clock**: jumps directly to next event timestamp, no fixed tick, monotonic
- **Snapshot + delta replay**: load state snapshot, replay only subsequent delta events
- **Deterministic guarantee**: same initial state + same events = same decisions
- **Runtime recording control**: xceadmin `scheduler record start/stop` commands
- **JSONL replay format**: one event per line, debuggable, plain text
- **What-if branching**: load snapshot, replay alternative delta events

### Queue vs Project — Roles

| | Queue | Project |
|---|---|---|
| **Role** | Scheduling mechanism | Resource ownership |
| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? |
| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster |
| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy, members (future) |
| **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many |

One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware). The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2).

## Alternatives Considered

### Approach A: Incremental Enhancement (rejected)

| Aspect | Detail |
|--------|--------|
| **Job selection** | Add `job_pick_plugin` config field, same as chosen approach |
| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation |
| **Iterator** | No new abstraction. Simple for-loop over plan array |
| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state |

**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost.

### Approach B: Iterator-Driven Allocation + Picker Plugin ★ Chosen

| Aspect | Detail |
|--------|--------|
| **Job selection** | Pluggable `xce_job_picker_iface_t` with `next_job()`. Single `builtin_fcfs` default. Per-queue picker override. Synchronous, no dedicated thread. |
| **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable |
| **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 |
| **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval |
| **Preemption** | `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP via controller→worker |
| **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` |
| **Core/driver split** | Passive event-driven core + dual-mode driver (normal/simulation) |
| **Simulation** | JSONL replay, snapshot + delta, discrete event-driven virtual clock, deterministic |

**Chosen because:** Lazy generation avoids memory pressure. Single `next_job()` interface makes pickers truly pluggable — the core loop never changes. Core/driver separation enables simulation mode for debugging/tuning without a live cluster. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc.

### Approach C: Full Scheduler Framework (deferred)

| Aspect | Detail |
|--------|--------|
| **Job selection** | Policy chain — multiple plugins score/veto jobs |
| **Iterator** | Triple composition: job iterator → plan iterator → host iterator |
| **Resreq registry** | Named primary resreqs registered globally, reference-counted |
| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin |
| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT |

**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's picker interface is designed to compose upward — a future policy-chain picker implements the same `next_job()` interface.

### Decision Rationale

Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The picker plugin interface with `next_job()` keeps the core loop simple (5 lines) and makes pickers truly pluggable. Core/driver separation enables simulation mode — a capability that would be prohibitively expensive to retrofit later.

**Design evolution (2026-07-10 → 2026-07-27):** The original design specified a four-level architecture with three separate builtin plugins (`builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`) and a three-level `project→queue→job` loop hardcoded in the core. Analysis of the existing FCFS plugin (`fcfs.c`) showed it already encapsulates all picking logic internally. The architecture was simplified to a single `next_job()` plugin interface with one builtin picker (`builtin_fcfs`) — the three-level decomposition is now an internal implementation detail of the plugin, not the core loop. Core/driver separation was added (Req 10) to enable simulation mode for policy debugging and tuning.

## Approach

**Iterator-Driven Allocation with Pluggable Picker + Core/Driver Separation**:

### Core scheduling loop

```c
void* sched_allocator(void* arg) {
    while (1) {
        picker->begin_session(picker);

        xce_sched_job_t* job;
        while ((job = picker->next_job(picker)) != NULL) {
            if (try_allocate(job) > 0) {
                picker->on_dispatch(picker, job->jobid);
                emit_decision(DISPATCH, job);
            } else if (can_preempt(job)) {
                emit_decision(PREEMPT, job, victims);
                picker->on_pend(picker, job->jobid);
            } else {
                emit_decision(PEND, job);
                picker->on_pend(picker, job->jobid);
            }
        }

        picker->end_session(picker);
        sleep(1);
    }
}
```

The core loop has zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`.

### Picker plugin interface

```c
typedef struct xce_job_picker {
    // Lifecycle
    int  (*init)(void* picker, xce_sched_core_t* core);
    void (*fini)(void* picker);

    // State-change hooks (called by core when events arrive)
    int  (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
    int  (*on_job_push)(void* picker, xce_sched_job_t* job);
    int  (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
    int  (*on_job_update)(void* picker, xce_sched_job_t* job);

    // Session lifecycle
    int  (*begin_session)(void* picker);
    int  (*end_session)(void* picker);

    // Job selection
    xce_sched_job_t* (*next_job)(void* picker);

    // Post-allocation notification
    int  (*on_dispatch)(void* picker, uint64_t jobid);
    int  (*on_pend)(void* picker, uint64_t jobid);

    // Optional hooks (NULL = not implemented)
    int  (*on_config)(void* picker, xce_conf4sched_t* config);
    int  (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages);
} xce_job_picker_iface_t;
```

### builtin_fcfs internal design

The default picker internally implements the equivalent of today's selection:

```
builtin_fcfs.next_job():
  Iterate queues by priority (descending)
    Skip queues with no ready jobs
    For each queue:
      If queue has per-queue picker override:
        Delegate to that picker
      Else:
        Iterate jobs by priority desc, queue_time asc (FCFS)
        Skip jobs not ready
        Return first ready job
  Return NULL if all queues exhausted
```

This is an implementation detail of the plugin, not visible to the core loop.

### Preemption flow

```
scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next round → retry alloc
```

### Core/Driver Architecture

```
┌──────────────────────────────────────────────────────────────┐
│                     DRIVER LAYER                               │
│                                                                │
│  Normal Mode                  Simulation Mode                  │
│  controller capnp ──┐         replay.jsonl ──┐                │
│  handler → events   │         virtual clock   │                │
│  → publish decisions│         → record/log     │                │
│       │       ▲     │              │       ▲   │                │
└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘
        │       │ decisions          │       │ decisions
        │       │ (callback)         │       │ (callback)
        ▼       │                    ▼       │
┌──────────────────────────────────────────────────────────────┐
│                   SCHEDULER CORE                               │
│                                                                │
│  core_push_event() → picker hooks → picker->next_job()        │
│  → try_allocate() → DISPATCH / PREEMPT / PEND                 │
│                                                                │
│  ZERO dependencies on: controller, capnp, network, wall-clock │
└──────────────────────────────────────────────────────────────┘
```

### What stays unchanged
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved, but the picker no longer uses the dedicated `job_elect_fn` thread
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
- Select expression system — no changes
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
- Controller communication (handler.c opcodes, decision cbuf in driver) — extended, not replaced
- `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications

### What's new
- `xce_job_picker_iface_t` — pluggable picker interface with full hook specification
- `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy iterator abstractions
- `XCE_DECISION_PREEMPT` decision type + `xce_preempt_decision_t` payload
- `builtin_fcfs` — single builtin picker (replaces three separate builtins from original design)
- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t` — core/driver struct split
- `xce_core_event_t` — typed event interface between driver and core
- `xce_core_decision_cb_t` — decision callback (core → driver)
- JSONL replay file format + snapshot/restore
- Discrete event-driven virtual clock
- `project_name` on queue structs
- `job_pick_plugin` on scheduler + queue structs (per-queue override)
- `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct

## Scope

### In
- Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic
- Single builtin picker (`builtin_fcfs`) replicating today's behavior
- Configurable picker plugin with per-queue override (`queue->job_pick_plugin`)
- Picker hook specification: lifecycle, state-change, session, post-allocation, optional hooks
- `xce_alloc_plan_iter_t` + `xce_host_iter_t` interfaces (lazy, cancel-safe, composable)
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
- Preemption: `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP mechanism
- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy`
- Session-level unsatisfiable resreq fingerprint cache
- Core/driver separation: passive event-driven core, normal-mode driver, simulation-mode driver
- Discrete event-driven virtual clock (monotonic, jumps to next event timestamp)
- Runtime recording control via xceadmin commands (`scheduler record start/stop`)
- JSONL replay file format (HEADER + SNAPSHOT + events + DECISIONS)
- State snapshot/restore for replay efficiency
- Deterministic replay guarantee for debugging/tuning
- What-if branching (load snapshot, replay alternative delta events)
- Data structure changes: `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`, `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t`
- Picker plugin interface contract documentation
- Unit tests: plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, event recording/replay, snapshot/restore, deterministic replay
- Integration test: multi-slot parallel job allocation and preemption flow end-to-end

### Out
- New job selection policies beyond the builtin FCFS picker — the pluggable interface enables them, but none ship here
- Hierarchical project resource quotas — deferred to `project-resource-ownership` spec
- Resreq template registry — future enhancement
- Select expression language changes
- Controller-side scheduling state replication (commit barrier wiring)
- Compound/ALTER resreq allocation changes
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
- Kill-and-requeue preemption escalation (v1 is SIGSTP only)
- Deadline field on `xce_sched_job_t`
- Event payload JSON schema (open design question — deferred to design phase)
- Distributed/cluster-mode simulation (v1 is single-scheduler replay only)

## Future: Project Resource Ownership

Deferred to a separate `project-resource-ownership` spec. When implemented:

- Projects form a tree with resource quota at each node
- A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue
- Project quota enforcement: projects over quota are skipped by the picker
- The core scheduling loop requires zero code changes — only a new picker plugin replaces `builtin_fcfs`
- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes
- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends

The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop.

## Boundary Candidates

The work naturally splits into these implementation phases (single spec, sequential):

1. **Core/driver separation** — split `xce_scheduler_t` into `xce_sched_core_t` + `xce_sched_driver_t`; define `xce_core_event_t` and `xce_core_decision_cb_t`; implement passive event-driven core interface; wire normal-mode driver to existing handler.c
2. **Picker plugin interface + builtin_fcfs** — define `xce_job_picker_iface_t`; implement `builtin_fcfs` with all MUST/SHOULD hooks; retire dedicated picker thread + `jbuf`; make core loop call `next_job()` synchronously; add `job_pick_plugin` config with per-queue override
3. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
4. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
5. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline, queue preemption config fields
6. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop
7. **Simulation driver** — implement simulation-mode driver with JSONL replay, virtual clock, snapshot/restore, deterministic verification
8. **Recording + xceadmin commands** — implement runtime recording control, periodic snapshots during recording
9. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, simulation replay, HA soak regression

## Out of Boundary
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that implement `xce_job_picker_iface_t`
- Resreq template sharing across jobs (future: `resreq_registry` spec)
- Commit barrier wiring for scheduler decisions (separate hardening item in roadmap)
- Controller-side changes (the scheduler runs independently, receives updates via existing opcodes)

## Upstream / Downstream

### Upstream (dependencies)
- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed
- Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed
- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed
- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — split, not replaced

### Downstream (future consumers)
- **New job selection policies** — priority, fair-share, deadline, backfill plugins implement `xce_job_picker_iface_t`
- **Project-resource-ownership** — project-aware picker plugin consumes `queue->project_name` and `xce_sched_project_t`
- **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement
- **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item)
- **Resreq template registry** — named resreqs shared across jobs, reference-counted

## Existing Spec Touchpoints

### Extends
- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same scheduling state. No conflicts; the msg-gap spec fills handler gaps this spec reads from.

### Adjacent
- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume.
- **worker-join-refactor** — worker state management. Allocation consumes worker resource availability pushed via the existing join/update path.

## Constraints
- **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads)
- **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns.
- **Cancel-safe** — `next_job()` must have `pthread_cancel` cancellation points
- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`
- **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive.
- **Deterministic replay** — same initial state + same event sequence → same decisions. No wall-clock time, no unseeded randomness, stable iteration order.
- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded).