myxce/src/scheduler/CLAUDE.md

XCE Scheduler

Overview

C-based job scheduling engine with a plugin architecture. Determines job-to-node assignments based on resource requirements and scheduling policies. Since scheduler-core (2026-08), the engine is split into a mode-agnostic core state machine driven by two interchangeable drivers — a controller-connected normal-mode driver and a standalone .replay-file simulation driver — so scheduling decisions can be recorded live and deterministically replayed offline for debugging/tuning.

Build

/xce-build scheduler    # debug + ASAN + tests; add `all` instead of `scheduler` when a downstream
                         # task also needs xceadmin/controller binaries coexisting in the same build dir

Architecture

• Core (core.c/core_events.c/core_session.c/core_clock.c) — passive, event-driven xce_sched_core_t state machine. Zero controller I/O (no netio/xceapi/client/delay/idgen); core_clock.c is the one sanctioned exception (clock_gettime, virtualized two-base+speed clock formula, xce_core_now()). test_core_isolation.c mechanically enforces this for core.c/core.h/core_events.c/core_session.c (source-text scan) plus core_clock.c too on the nm(1) undefined-symbol layer (all four compiled together into the isolated sched_core_isolation_lib archive, meson.build) — core_clock.c is deliberately excluded only from the wall-clock text scan, since it owns the one sanctioned clock_gettime() call. Picker→allocator handoff is a strict jbuf cbuf capacity-1 rendezvous — this is the project's determinism invariant, not a tuning knob; never raise it without re-deriving Req 5 AC 7 in .kiro/specs/scheduler-core/requirements.md.

• Picker plugin interface (plugin.h's xce_job_picker_iface_t, plugins/builtin_fcfs.c) — job-selection strategies are pluggable; scheduler.c's sched_job_picker() branches on whether a plugin still uses the legacy job_elect_fn-only shape (plugins/fcfs.c, kept as-is for a compat-wrapper exercise, not the live default) vs the native hook interface (builtin_fcfs, the registered default). New scheduling policies → add as a picker plugin under src/plugins/, do not modify the core loop.

• Drivers — driver.c (normal mode: capnp↔core event translation, decision publishing, deferred admin-request handling) and sim.c/sim_loop.c/sim_record.c/sim_verify.c (simulation mode: .replay file loading, discrete virtual clock, --sim-verify/--sim-alt-events/--sim-output). Both drive the identical core session/decision-callback protocol — one core, two consumers. This equivalence is a DESIGN INVARIANT, not just an architectural description — see .kiro/steering/scheduler-driver-equivalence.md for the precise (state-level, not just decision-level) contract, known violations (bug-1611, bug-1613), and the testing discipline required of any change in this area.

• Replay/recording — recording.c/recording_events.c/recording_session.c (live capture) and sim.c/sim_loop.c (simulation-side decode) do the actual capnp encode/decode of the .replay binary stream (src/schema/controller.capnp); snapshot.c/snapshot_wire.c/snapshot_load.c only populate/consume the plain in-memory sched_snapshot_data_t struct that recording.c's write_snapshot_entry() then serializes — no capnp or file I/O lives in the snapshot files themselves. Controlled via OPCode_schedAdmin (src/controller/cluster.c) and xceadmin scheduler record start|stop|status / scheduler dump <FILE>.

• Interactive REPL (repl.c/repl_break.c/repl_info.c/repl_params.c, --interactive) — single-step/breakpoint/state-inspection/snapshot-capture control over the simulation driver; drives the exact same sched_sim_driver_run() as batch --sim, gated only by sim->interactive, so REPL and batch runs are decision-equivalent by construction (test_repl_equivalence.c).

• Decision trace (scheduler-sim-decision-trace, simulator REPL only) — include/sched/trace.h is the public interface and the SCHED_TRACE(core, call) hook macro, which evaluates call only while core->trace_rec is set; live never sets it, so hooks cost one pointer test. The sources are split to stay under the line cap:

  • sched_trace.c: the record, its arena, the entry sequence and line budget, and requirement/plan/check entries.
  • sched_trace_scan.c: positions, host verdicts and rows, summary tallies, rollback, commit and search end.
  • sched_trace_capture.c: copies selection facts, cluster-wide limits, check rows and attempt reasons out of scheduler state.
  • sched_trace_outcome.c: dispatch and pend outcomes, and reconstructed-dispatch records.
  • sched_trace_internal.h: helpers shared between those files (sti_ prefix).

  None of them may use stdio, file I/O or logging, and none belongs in sched_core_isolation_lib; test_core_isolation.c scans all five. Every white-box test that #includes a hooked source (resource, host_iter, alloc_runner, scheduler, sched_election, driver) must also #include the four trace .c files, exactly once.

  The REPL side (sim thread, stdio allowed, declared in repl_internal.h; repl.h holds the interactive-only gates sim_loop.c calls):

  • repl_trace.c: the trace command grammar, trace status, and the refcounted next/cur session settings with their trace file.
  • repl_trace_emit.c: session begin/end hooks and per-decision emission to the console and trace file, then into the store.
  • repl_trace_store.c: latest record per job, oldest-first eviction at 64 MiB (never the newest), discarded marks, info trace.
  • repl_trace_render.c, repl_trace_render_outcome.c, repl_trace_render.h (private): the one renderer; every output path reuses it.

  Every test that #includes repl.c or sim_loop.c (rather than linking them from scheduler_srcs) must also #include the five REPL trace .c files and the four core trace .c files, exactly once; sched_sim_isolation_lib carries all nine, and test_sim_isolation.c scans the REPL trace files and repl_trace_render.h. A new REPL trace file goes into all three. Bounds under large traces are covered by test_trace_bounds.c.

Links against libxce from src/lib/. Full design/requirements: .kiro/specs/scheduler-core/.

Conventions

• C standard: gnu17
• Format with clang-format
• New scheduling policies → add as a plugin under src/plugins/, do not modify the core engine
• Shared types defined in src/include/xce/ (libxce)
• Files stay under the repo's 800-line cap — several core/sim/repl/snapshot/recording modules are deliberately split across multiple *_events.c/*_session.c/*_wire.c-style files for this reason; keep new logic in the file matching its concern rather than growing one file past the cap. Not yet true project-wide: scheduler.c (2060 lines), sim_loop.c (1844), repl_info.c (1779), core_events.c (1488), driver.c (1117), resource.c (1106), plugins/builtin_fcfs.c (1104), repl_break.c (980) and include/core.h (861) (measured 2026-10-08) are pre-existing exceptions over the cap with no split or exemption note — do not use them as precedent for new files, and prefer splitting them opportunistically rather than growing them further.

See Also

• Shared library: src/lib/CLAUDE.md
• Root guidelines: CLAUDE.md (repo root)
• Architecture/requirements: .kiro/specs/scheduler-core/design.md, .kiro/specs/scheduler-core/requirements.md