# XCE Scheduler

## Overview
C-based job scheduling engine with a plugin architecture. Determines job-to-node assignments based on resource requirements and scheduling policies. Since `scheduler-core` (2026-08), the engine is split into a mode-agnostic **core** state machine driven by two interchangeable **drivers** — a controller-connected normal-mode driver and a standalone `.replay`-file simulation driver — so scheduling decisions can be recorded live and deterministically replayed offline for debugging/tuning.

## Build
```bash
/xce-build scheduler    # debug + ASAN + tests; add `all` instead of `scheduler` when a downstream
                         # task also needs xceadmin/controller binaries coexisting in the same build dir
```

## Architecture
- **Core** (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c`) — passive, event-driven `xce_sched_core_t` state machine. Zero controller I/O (no `netio`/`xceapi`/`client`/`delay`/`idgen`); `core_clock.c` is the one sanctioned exception (`clock_gettime`, virtualized two-base+speed clock formula, `xce_core_now()`). `test_core_isolation.c` mechanically enforces this for `core.c`/`core.h`/`core_events.c`/`core_session.c` (source-text scan) plus `core_clock.c` too on the `nm(1)` undefined-symbol layer (all four compiled together into the isolated `sched_core_isolation_lib` archive, meson.build) — `core_clock.c` is deliberately excluded only from the wall-clock text scan, since it owns the one sanctioned `clock_gettime()` call. Picker→allocator handoff is a strict `jbuf` cbuf capacity-1 rendezvous — this is the project's determinism invariant, not a tuning knob; never raise it without re-deriving Req 5 AC 7 in `.kiro/specs/scheduler-core/requirements.md`.
- **Picker plugin interface** (`plugin.h`'s `xce_job_picker_iface_t`, `plugins/builtin_fcfs.c`) — job-selection strategies are pluggable; `scheduler.c`'s `sched_job_picker()` branches on whether a plugin still uses the legacy `job_elect_fn`-only shape (`plugins/fcfs.c`, kept as-is for a compat-wrapper exercise, not the live default) vs the native hook interface (`builtin_fcfs`, the registered default). New scheduling policies → add as a picker plugin under `src/plugins/`, do not modify the core loop.
- **Drivers** — `driver.c` (normal mode: capnp↔core event translation, decision publishing, deferred admin-request handling) and `sim.c`/`sim_loop.c`/`sim_record.c`/`sim_verify.c` (simulation mode: `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`). Both drive the identical core session/decision-callback protocol — one core, two consumers. **This equivalence is a DESIGN INVARIANT, not just an architectural description — see `.kiro/steering/scheduler-driver-equivalence.md` for the precise (state-level, not just decision-level) contract, known violations (bug-1611, bug-1613), and the testing discipline required of any change in this area.**
- **Replay/recording** — `recording.c`/`recording_events.c`/`recording_session.c` (live capture) and `sim.c`/`sim_loop.c` (simulation-side decode) do the actual capnp encode/decode of the `.replay` binary stream (`src/schema/controller.capnp`); `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` only populate/consume the plain in-memory `sched_snapshot_data_t` struct that `recording.c`'s `write_snapshot_entry()` then serializes — no capnp or file I/O lives in the snapshot files themselves. Controlled via `OPCode_schedAdmin` (`src/controller/cluster.c`) and `xceadmin scheduler record start|stop|status` / `scheduler dump <FILE>`.
- **Interactive REPL** (`repl.c`/`repl_break.c`/`repl_info.c`/`repl_params.c`, `--interactive`) — single-step/breakpoint/state-inspection/snapshot-capture control over the simulation driver; drives the exact same `sched_sim_driver_run()` as batch `--sim`, gated only by `sim->interactive`, so REPL and batch runs are decision-equivalent by construction (`test_repl_equivalence.c`).

- **Decision trace** (`scheduler-sim-decision-trace`, simulator REPL only) — `include/sched/trace.h` is the public interface and the `SCHED_TRACE(core, call)` hook macro, which evaluates `call` only while `core->trace_rec` is set; live never sets it, so hooks cost one pointer test. The sources are split to stay under the line cap:
  - `sched_trace.c`: the record, its arena, the entry sequence and line budget, and requirement/plan/check entries.
  - `sched_trace_scan.c`: positions, host verdicts and rows, summary tallies, rollback, commit and search end.
  - `sched_trace_capture.c`: copies selection facts, cluster-wide limits, check rows and attempt reasons out of scheduler state.
  - `sched_trace_outcome.c`: dispatch and pend outcomes, and reconstructed-dispatch records.
  - `sched_trace_internal.h`: helpers shared between those files (`sti_` prefix).

  None of them may use stdio, file I/O or logging, and none belongs in `sched_core_isolation_lib`; `test_core_isolation.c` scans all five. Every white-box test that `#include`s a hooked source (resource, host_iter, alloc_runner, scheduler, sched_election, driver) must also `#include` the four trace `.c` files, exactly once.

  The REPL side (sim thread, stdio allowed, declared in `repl_internal.h`; `repl.h` holds the interactive-only gates `sim_loop.c` calls):
  - `repl_trace.c`: the `trace` command grammar, `trace status`, and the refcounted `next`/`cur` session settings with their trace file.
  - `repl_trace_emit.c`: session begin/end hooks and per-decision emission to the console and trace file, then into the store.
  - `repl_trace_store.c`: latest record per job, oldest-first eviction at 64 MiB (never the newest), discarded marks, `info trace`.
  - `repl_trace_render.c`, `repl_trace_render_outcome.c`, `repl_trace_render.h` (private): the one renderer; every output path reuses it.

  Every test that `#include`s `repl.c` or `sim_loop.c` (rather than linking them from `scheduler_srcs`) must also `#include` the five REPL trace `.c` files and the four core trace `.c` files, exactly once; `sched_sim_isolation_lib` carries all nine, and `test_sim_isolation.c` scans the REPL trace files and `repl_trace_render.h`. A new REPL trace file goes into all three. Bounds under large traces are covered by `test_trace_bounds.c`.

Links against `libxce` from `src/lib/`. Full design/requirements: `.kiro/specs/scheduler-core/`.

## Conventions
- C standard: gnu17
- Format with `clang-format`
- New scheduling policies → add as a plugin under `src/plugins/`, do not modify the core engine
- Shared types defined in `src/include/xce/` (libxce)
- Files stay under the repo's 800-line cap — several core/sim/repl/snapshot/recording modules are deliberately split across multiple `*_events.c`/`*_session.c`/`*_wire.c`-style files for this reason; keep new logic in the file matching its concern rather than growing one file past the cap. Not yet true project-wide: `scheduler.c` (2060 lines), `sim_loop.c` (1844), `repl_info.c` (1779), `core_events.c` (1488), `driver.c` (1117), `resource.c` (1106), `plugins/builtin_fcfs.c` (1104), `repl_break.c` (980) and `include/core.h` (861) (measured 2026-10-08) are pre-existing exceptions over the cap with no split or exemption note — do not use them as precedent for new files, and prefer splitting them opportunistically rather than growing them further.

## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)
- Architecture/requirements: `.kiro/specs/scheduler-core/design.md`, `.kiro/specs/scheduler-core/requirements.md`