myxce/.kiro/specs/scheduler-core/requirements.md

Requirements Document

Introduction

The XCE scheduler has a plugin architecture but critical foundational gaps. Job selection policy is hardcoded (sched_job_picker() sets job_pick_plugin = "xce_sched_fcfs" as a literal). The scheduler is tightly coupled to controller I/O (apictx, cluster_cfg embedded in xce_scheduler_t). There is no way to swap picking policies without recompilation, and no way to debug or tune scheduling behavior without a live cluster.

This spec establishes one central design invariant and three architectural foundations to achieve it.

Central invariant — Normal↔Sim equivalence: Given identical initial configuration and identical event sequences, the scheduler shall produce identical scheduling decisions regardless of whether it runs in production (normal mode, controller-connected, wall-clock time) or in simulation (replay file, virtual clock). Record a live run → replay it in simulation → the decisions match byte-for-byte. This is NOT a debugging convenience — it is the validation contract for the entire architecture. If normal and simulation diverge, the core/driver separation is broken. This invariant enables every simulation-mode use case: debugging is faithful, tuning is predictive, regression testing is trustworthy, and what-if analysis reflects real behavior.

Three foundations deliver this invariant:

Pluggable job picker: A xce_job_picker_iface_t — the picker thread calls picker->next_job() to get the next schedulable job and hands it to the allocator thread via cbuf, with zero knowledge of how jobs are selected. The default builtin_fcfs picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior. Per-queue picker overrides are supported. The picker runs in a dedicated thread communicating with the allocator via cbuf (existing producer/consumer pattern preserved).

Core/driver separation: The scheduler core is a passive, event-driven engine with zero controller I/O dependencies. A normal-mode driver connects it to a live controller; a simulation-mode driver replays recorded cluster events through a discrete event-driven virtual clock, enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster.

Core is mode-agnostic: The core has no concept of which mode it runs in. It processes events identically whether the driver is in normal mode or simulation mode. There is no mode flag, no mode-specific code path, and the same init function is used for both modes.

Full discovery context is in .kiro/specs/scheduler-core/brief.md. Sibling specs: scheduler-allocation, scheduler-preemption.

Queue vs Project — Roles

These are fundamentally different concerns:

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered and constrained?	WHO owns resources and gets charged?
Scope	One hardware/scheduling profile	Cross-queue, cross-cluster
Key attributes	Priority, limits, time windows, per-queue picker override, preemption config	Quota, usage, hierarchy, members (future)
Cardinality to job	1 job → 1 queue	1 job → 1 project
Cardinality to each other	Many-to-many	Many-to-many

• Queue = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, priority (for inter-queue ordering within the picker), per-queue picker override, and preemption config.
• Project = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. The picker plugin may use queue->project_name internally for project-scoped iteration (v2).

Boundary Context

- **In scope**: Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic behind a single plugin; single builtin picker (`builtin_fcfs`) replicating today's behavior through the new interface; configurable picker plugin with per-queue override (`queue->job_pick_plugin`); **core/driver separation** with passive event-driven core interface, mode-agnostic core (no mode flag, no mode-specific code paths), normal-mode driver (controller-connected), simulation-mode driver as a **standalone investigation tool** (replay recorded events with discrete event-driven virtual clock, zero controller dependencies, self-contained `.replay` file — single capnp binary stream with header + snapshot + events + decisions + periodic checkpoints in one artifact, what-if editing via event file substitution, diff-based comparison of simulated vs recorded decisions via `--sim-output`), virtualized clock access (`xce_core_now()` with two-base + speed formula), runtime recording control via xceadmin commands, `.replay` capnp stream format, state snapshot/restore for replay efficiency, **interactive REPL** with single-step, breakpoints, state inspection, and snapshot capture via `--interactive`, and deterministic replay guarantee for debugging/tuning; **normal↔sim equivalence guarantee** (same config + same events → identical decisions regardless of mode, AC 7.1); `xceadmin scheduler dump <FILE>` for human-readable replay file inspection; documentation of picker plugin interface contract (hooks, cancel-safe, session-aware); unit tests for picker hook behavior, **event recording/replay, snapshot/restore, and deterministic replay**; HA soak regression for picker changes.

• Out of scope: Allocation iterators and parjob allocator (→ scheduler-allocation spec); preemption policy and SIGSTP pipeline (→ scheduler-preemption spec); new job selection policies beyond the builtin FCFS picker — the pluggable picker interface enables them, but none ship in this spec beyond builtin_fcfs; hierarchical project resource quotas (deferred to project-resource-ownership spec — only project_name on queue and xce_sched_project_t scaffolding ship here); decision payload capnp schema for DispatchDecision/PendDecision/PreemptDecision (deferred to design phase — replay file format schema is in AC 20); distributed/cluster-mode simulation (v1 is single-scheduler replay only); visual inspection / UI tool for replay files (deferred — xceadmin scheduler dump provides human-readable output, diff of dumps is the primary comparison workflow for v1).

• Adjacent expectations: Depends on the existing plugin framework (plugin.h/plugin.c), the existing select-expression evaluator (resreq.h, memoized via select_eval_ctx_t), and existing shared-resource tracking — none of these change. Sibling spec scheduler-allocation provides the try_allocate() function this spec's core loop calls. Sibling spec scheduler-preemption provides the can_preempt() check and preemption decision type.

Implementation Phases

The spec is implemented in three sequential phases. Each phase ships independently, passes HA soak regression, and preserves existing scheduling behavior. Phases are ordered by dependency — earlier phases produce primitives that later phases consume.

Phase 1: Pluggable Picker (Requirements 1–4)

Scope: Replace the hardcoded "xce_sched_fcfs" literal with a configurable xce_job_picker_iface_t. Implement builtin_fcfs as the default plugin, replicating today's FCFS behavior exactly through the new interface hooks.

Changes to existing code:

• Add xce_job_picker_iface_t with lifecycle, state-change, session, and post-allocation hooks (Req 1)
• Implement builtin_fcfs plugin with private ordq_t state, all MUST hooks (Req 3)
• Replace sched_job_picker() hardcoded string with plugin_register/key_find lookup (Req 1 AC 7)
• Add job_pick_plugin to xce_scheduler_t and xce_sched_queue_t (Req 2)
• Add xce_sched_project_t scaffolding + project_name on queue (Req 2 AC 3)

What does NOT change:

• Today's main loop (sched_update → sched_begin_session → sched_go → decision drain → sched_end_session) is untouched
• Today's threads (picker, allocator) and cbuf communication are untouched
• Today's controller integration (scheduler_req_handler, capnp) is untouched
• Today's build system, test suite structure is untouched

Verification: HA soak — same job-picking order as today, same decisions, zero divergence.

Why first: Phase 2's core_push_events() must call picker hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update). Implementing the picker first means Phase 2 only moves existing hook call sites — same calls, different location. No dead wiring, no incomplete interfaces.

Phase 2: Core/Driver Separation (Requirement 5, ACs 1–8)

Scope: Extract the scheduling core into a passive, mode-agnostic engine (xce_sched_core_t). Introduce the three-call driver↔core protocol (core_push_events, core_go, on_decision). Build the normal-mode driver that recreates today's controller-connected behavior.

Changes to existing code:

• Extract xce_sched_core_t from xce_scheduler_t — all scheduling state, zero controller deps (AC 1)
• Implement core_push_events() — batch-push events, call picker hooks synchronously (AC 2, AC 4)
• Implement core_go() — non-blocking signal to picker thread via cbuf_put(&wbuf) (AC 6)
• Implement on_decision() callback — per-decision emission to driver (AC 6)
• Build normal-mode driver: capnp → core events → decisions → xcecli (AC 8)
• Replace today's main.c decision switch with driver callback

What does NOT change:

• Picker interface and builtin_fcfs plugin (Phase 1 already done)
• Threading: picker → jbuf → allocator → decisions cbuf pattern preserved
• Controller integration: same capnp messages, same xcecli API, same behavior
• Simulation: not yet wired — core is mode-agnostic but no sim driver exists

Verification: HA soak — same decisions as Phase 1, same controller interaction, zero regression.

Why second: Core extraction needs the picker hooks to exist (Phase 1 done). Core extraction is a prerequisite for Phase 3 — simulation needs a mode-agnostic core that is already tested and passing HA soak in production.

Phase 3: Simulation Mode + Recording + Interactive REPL (Requirement 5, ACs 9–22)

Scope: Add virtual clock (xce_core_now()), snapshot/restore (save_snapshot/load_snapshot), simulation driver (replay file → events → decisions → record file), interactive REPL (step, breakpoints, state inspection, snapshot capture), and runtime recording hooks in the normal-mode driver.

Changes to existing code:

• Add clock virtualization atoms to core: virtual_clock_base, real_clock_base, speed_num, speed_den (AC 3)
• Implement save_snapshot() / load_snapshot() with picker rebuild from core state (AC 21)
• Build simulation driver: load HEADER+SNAPSHOT, replay events, emit decisions to --sim-output (AC 13, AC 19)
• Build interactive REPL: stdin/stdout command loop, breakpoint matching, state inspection queries (AC 22)
• Add recording state to normal driver: scheduler record start/stop, event writing, periodic snapshots (AC 10–11, AC 12)
• Add simulation CLI flags: --sim, --sim-speed, --sim-verify, --sim-output, --interactive / -i

What does NOT change:

• Core is already mode-agnostic (Phase 2 done) — no if (simulation) branches added
• Picker is already clean (Phase 1 done) — no changes to picker interface
• Normal driver is already split (Phase 2 done) — recording hooks are boolean-gated, zero overhead when off
• Production scheduling behavior is identical — simulation is an opt-in mode

Verification: HA soak (normal mode — no regression). Normal↔sim equivalence (AC 7.1): record a live run, replay in simulation, diff <(xceadmin scheduler dump recorded.replay) <(xceadmin scheduler dump simulated.replay) → decision content identical. Deterministic replay (AC 7): same .replay file twice → same output. Recording: file written correctly, simulation replays recorded file correctly. Interactive REPL (AC 22): --interactive mode steps, breaks, and completes to identical decisions as batch --sim mode. State inspection commands report accurate live state (verified against known-good snapshot dumps).

Why last: Simulation needs Phase 1 (picker hooks exist) and Phase 2 (core is mode-agnostic, driver boundary is clean). It is the only phase that adds net-new capability rather than restructuring existing code — all risk is in Phase 1 and 2. By Phase 3, the new architecture has already been validated in production via HA soak.

Phase Dependency Graph

Phase 1: Pluggable Picker ──► Phase 2: Core/Driver Split ──► Phase 3: Simulation
       (Req 1-4)                    (Req 5 AC 1-8)               (Req 5 AC 9-22)
            │                              │                              │
            │ No deps                      │ Needs picker hooks           │ Needs mode-agnostic core
            │                              │   from Phase 1               │   + clean driver boundary
            │                              │                              │   from Phase 2
            ▼                              ▼                              ▼
     HA soak pass                   HA soak pass                   HA soak pass

Each phase: TDD (red → green → refactor) → HA soak regression → merge. No phase gates on unfinished work from a later phase. Each phase ships value independently.

Requirements

Requirement 1: Pluggable job picker interface

Objective: As a maintainer of the scheduler, I want the core scheduling loop to call a single next_job() plugin interface with no knowledge of how jobs are selected, so that picking strategies (FCFS, priority, fair-share, deadline, etc.) are truly pluggable and the core loop never changes regardless of the picking algorithm.

Acceptance Criteria

1. The Scheduler core loop shall call picker->next_job(picker) to get the next schedulable job (or NULL if none ready), and shall not contain any logic about project iteration, queue iteration, or job ordering — those concerns are entirely encapsulated inside the picker plugin.

2. The job picker plugin interface (xce_job_picker_iface_t) shall define these operations, organized by category:

   Lifecycle hooks (MUST):

   Operation	Signature	Semantics
   init	int (*init)(void* picker, xce_sched_core_t* core)	One-time setup, receives core reference for state queries
   fini	void (*fini)(void* picker)	Teardown and cleanup of all picker-owned resources

   State-change hooks (MUST — called by core when events arrive from driver):

   Operation	Signature	Semantics	Triggered by
   on_queue_update	int (*on_queue_update)(void* picker, xce_sched_queue_t* queue)	Add/update queue in picker's internal ordering	QUEUE_UPDATE event
   on_job_push	int (*on_job_push)(void* picker, xce_sched_job_t* job)	Add new pending job to picker's candidate set	JOBS_PUSH event
   on_job_state_change	int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t new_state)	Remove job from candidates if no longer pending; re-add if returning to pending	JOB_STATE_CHANGE event
   on_job_update	int (*on_job_update)(void* picker, xce_sched_job_t* job)	Reorder job in candidates if priority/queue changed	JOB_UPDATE event

   Session hooks (MUST — called at round boundaries):

   Operation	Signature	Semantics
   begin_session	int (*begin_session)(void* picker)	Reset iteration cursor; called at start of each scheduling round
   next_job	xce_sched_job_t* (*next_job)(void* picker)	Return next schedulable job per picker's ordering, or NULL when exhausted
   end_session	int (*end_session)(void* picker)	End-of-round notification; called before core sleeps between rounds

   Post-allocation hooks (MUST — called after allocation attempt):

   Operation	Signature	Semantics
   on_dispatch	int (*on_dispatch)(void* picker, uint64_t jobid)	Job was allocated; remove from candidates permanently
   on_pend	int (*on_pend)(void* picker, uint64_t jobid)	Job was pended; keep for next round but may deprioritize

   Optional hooks (MAY — NULL if not needed):

   Operation	Signature	Semantics	When needed
   on_config	int (*on_config)(void* picker, xce_conf4sched_t* config)	Rebuild picker strategy from config	Config-driven pickers (deadline, fair-share weights)
   on_job_rusage	int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages)	Track per-job resource usage	Resource-aware pickers (fair-share, backfill)

   Hooks not defined on the picker interface (worker updates, resource updates, resval updates) are not forwarded to the picker — picking is about job ordering, not resource availability.

3. Threading model: The core uses three threads communicating via cbuf (bounded FIFO with mutex + condition variables, src/include/mylock.h):

   Thread	Role	cbuf In	cbuf Out	Blocked On
   Picker	Iterates queues/jobs via next_job(), pushes jobids	wbuf (from core main)	jbuf (to allocator)	cbuf_get(&wbuf) between sessions
   Allocator	Calls try_allocate()/can_preempt(), builds decisions	jbuf (from picker)	decisions (to driver)	cbuf_get(&jbuf) between jobs
   Core Main (driver context)	Calls core_push_events(), signals core_go(), drains decisions	decisions (from allocator)	wbuf (to picker)	cbuf_get(&decisions) during session

   Driver → Core event delivery: State-change hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update) are called synchronously during core_push_events() in the driver thread, before the call returns. This is safe because core_push_events() is only called between sessions when the picker thread is idle (blocked on cbuf_get(&wbuf)) and the allocator thread is idle (blocked on cbuf_get(&jbuf)). No mid-session event injection.

   Session hooks (begin_session, next_job, end_session) run in the picker thread. Post-allocation hooks (on_dispatch, on_pend) run in the allocator thread.

4. The core scheduling loop shall use a producer/consumer pattern across three threads communicating via cbuf:

   Picker thread (fcfs_elect_job equivalent):

   while (1) {
       int go = 0;
       cbuf_get(&core->wbuf, &go);           // block until core_go() signals
       picker->begin_session(picker);
       xce_sched_job_t* job;
       while ((job = picker->next_job(picker)) != NULL) {
           cbuf_put(&core->jbuf, &job->jobid);  // hand off to allocator
       }
       uint64_t end_marker = 0;
       cbuf_put(&core->jbuf, &end_marker);      // end-of-session signal
       picker->end_session(picker);
   }
   

   Allocator thread:

   while (1) {
       uint64_t jobid = 0;
       cbuf_get(&core->jbuf, &jobid);         // block until picker sends job
       if (jobid == 0) {
           // end-of-session marker
           cbuf_put(&core->decisions, NULL);
           continue;
       }
       xce_sched_job_t* job = sched_get_job(core, jobid);
       int rc = try_allocate(job);
       if (rc > 0) {
           picker->on_dispatch(picker, job->jobid);
           decision = make_dispatch_decision(job->allocs);
       } else if (can_preempt(job)) {
           decision = make_preempt_decision(job, victims);
           picker->on_pend(picker, job->jobid);
       } else {
           decision = make_pend_decision(job->reason);
           picker->on_pend(picker, job->jobid);
       }
       cbuf_put(&core->decisions, &decision);
   }
   

   Main loop (driver thread context — core.go()):

   atomic_store(&core->session_active, 1, memory_order_release);
   core_begin_session_internal(core);     // select eval ctx, reset workers
   cbuf_put(&core->wbuf, &go);           // NON-BLOCKING: signal picker, return
   // Driver then drains decisions cbuf:
   while (1) {
       cbuf_get(&core->decisions, &decision);
       if (decision == NULL) break;       // session exhausted
       core->on_decision(core->driver_ctx, decision);  // publish immediately
   }
   core_end_session_internal(core);      // GC, reset workers
   atomic_store(&core->session_active, 0, memory_order_release);
   

   The picker and allocator contain zero knowledge of project/queue/job ordering — all selection logic lives behind next_job(). The allocator contains zero knowledge of how the driver publishes decisions — it only calls cbuf_put(&decisions, &decision).

5. The picker plugin shall be cancel-safe: next_job() shall have defined pthread_cancel cancellation points. The picker shall be session-aware: begin_session() resets iteration state, end_session() is called before sleep-between-rounds.

6. The hardcoded "xce_sched_fcfs" literal at scheduler.c:43 (in sched_job_picker()) shall be replaced by loading the configured sched->job_pick_plugin via the existing plugin_register/key_find mechanism. If the configured plugin name is not registered, the Scheduler shall fail with a clear configuration error.

7. The existing multi-threaded producer/consumer architecture (picker thread → jbuf cbuf → allocator thread → decisions cbuf → main loop) shall be preserved. The driver thread signals the picker thread via wbuf cbuf (non-blocking cbuf_put). The only thread eliminated is the conceptual one: today's FCFS picker is a dedicated thread, and it stays a dedicated thread — but the core scheduling loop in the driver is simplified because decision publishing moves from a hardcoded main.c switch to a driver-registered on_decision callback.

Requirement 2: Configurable picker plugin

Objective: As a cluster operator, I want to configure which job-picking policy the scheduler uses without recompiling, and override it per-queue, so that FCFS, priority, or custom pickers can be swapped operationally.

Acceptance Criteria

1. The Scheduler shall add a job_pick_plugin field to xce_scheduler_t, defaulting to "builtin_fcfs" when unset — this is the global picker plugin name.

2. The Scheduler shall add a job_pick_plugin field to xce_sched_queue_t, defaulting to NULL (inherit from global). When non-NULL, the configured global picker plugin shall use this per-queue override for jobs within that queue. How the picker plugin uses per-queue overrides is an implementation detail of the plugin, not the core loop.

3. The Scheduler shall add a project_name field to xce_sched_queue_t, set from queue configuration, establishing the queue→project relationship. The picker plugin may use this field internally for project-scoped iteration.

4. The Scheduler shall accept the picker plugin name from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild.

5. When the configured plugin name is not registered via the existing plugin_register/key_find mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default.

6. The picker plugin interface contract (xce_job_picker_iface_t) shall be documented: each picker must be cancel-safe (responds to pthread_cancel at defined points in next_job()) and session-aware (resets its iteration state on begin_session()).

Requirement 3: Builtin default picker

Objective: As a maintainer, I want the new architecture to ship with a builtin picker plugin that replicates today's exact scheduling behavior, so that operators who don't configure a custom picker see no behavioral change.

Acceptance Criteria

1. Registration: builtin_fcfs shall be registered automatically at scheduler startup as "builtin_fcfs" and set as the default sched->job_pick_plugin.

2. Hook implementation — builtin_fcfs shall implement every MUST and SHOULD hook from xce_job_picker_iface_t (Req 1 AC 2) as follows:

   Hook	FCFS Implementation
   init	Allocate fcfs_plugin_data_t. Initialize ordq_t queues (ordered by queue->priority desc). Set iteration cursor to NULL.
   fini	Free all per-queue ordq_t pendJobs, the ordq_t queues, and plugin data.
   on_queue_update	Upsert queue into ordq_t queues. If queue exists, update its priority and reorder. If new, allocate fcfs_queue_t with empty ordq_t pendJobs.
   on_job_push	If job->state & JOB_STAT_PEND: find the job's queue in ordq_t queues, add fcfs_job_t{job} to that queue's ordq_t pendJobs (ordered by priority desc, queue_time asc).
   on_job_state_change	If new_state is not pending (EXIT/FAIL/DONE): remove job from its queue's pendJobs by jobid. If new_state returns to PEND: re-add to pendJobs.
   on_job_update	If job's priority or queue field changed: remove from old position in pendJobs, re-insert at correct position per ordering.
   begin_session	Reset iteration cursor to head of ordq_t queues.
   next_job	Walk ordq_t queues from cursor. For each queue, skip if sched_queue_is_ready() returns false. Within queue, walk ordq_t pendJobs. Return first job where sched_job_is_ready() returns true. Advance cursor. Return NULL when all queues and jobs exhausted.
   end_session	No-op for FCFS (iteration state reset in begin_session).
   on_dispatch	Remove job from its queue's ordq_t pendJobs by jobid.
   on_pend	No-op for FCFS — job remains in pendJobs, will be retried next round in same position.
   on_config	NULL — FCFS is config-agnostic.
   on_job_rusage	NULL — FCFS is not resource-aware.

3. Internal data structures — builtin_fcfs shall maintain its own private state, not query the core's sched->queues/sched->jobs during next_job(). All candidate management happens in the state-change hooks. This ensures:

   • next_job() is fast — no core lock acquisition, no hash table scans
   • The picker is independently testable — feed hooks, verify next_job() order
   • Simulation snapshots capture picker state naturally via the hooks already recorded in the event stream

4. Ordering equivalence — builtin_fcfs shall produce the same job-picking order as today's fcfs_elect_job: inter-queue by descending queue->priority, intra-queue by priority desc, queue_time asc. Behavior with per-queue picker overrides is as specified in Req 2 AC 2.

5. Dedicated picker thread retained — builtin_fcfs shall implement next_job() as a synchronous call within the picker's dedicated thread. The picker thread communicates with the allocator thread via jbuf cbuf (existing pattern). pthread_cancel cancellation points shall exist in the next_job() iteration loop and in cbuf_get(&wbuf) where the picker blocks between sessions.

6. Session state — end_session() is a no-op. Iteration state is reset in begin_session(). Undispatched jobs remain in pendJobs across sessions (they carry over to the next round).

Requirement 4: No regression to existing picker behavior or HA soak

Objective: As a maintainer, I want the picker rework and core/driver separation scoped so that existing FCFS behavior is unaffected, making the framework change safe to ship independently of any new policy.

Acceptance Criteria

1. builtin_fcfs shall produce the same job-picking order as today's fcfs_elect_job, given the same queue and job state — same inter-queue priority ordering and same intra-queue (priority desc, queue_time asc) ordering.
2. Existing scheduler unit tests (scheduler meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string and the refactored sched_go() / decision publishing path this feature replaces. The picker thread, allocator thread, and cbuf communication pattern are preserved.
3. A full HA soak (sequential and chaotic /xce-test-ha regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.

Requirement 5: Core/Driver Separation with Dual-Mode Operation

Objective: As a scheduler developer, I want the scheduling core separated from all external I/O so that the same core logic runs deterministically in production (driven by a live controller) and in simulation (driven by recorded cluster events with a discrete event-driven virtual clock), enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster.

Acceptance Criteria

Driver ↔ Core Protocol (overview)

The driver and core communicate through a three-call synchronous interface with cbuf-based internal threading. The driver buffers controller events externally between sessions, pushes them as a batch before each scheduling round, and receives decisions one-by-one as they are produced.

Direction	Call	Semantics	Thread
Driver → Core	core_push_events(core, events, n)	Batch-push events before session. State-change hooks called synchronously. Only valid when picker is idle (between sessions).	Driver thread
Driver → Core	core_go(core)	Start scheduling session. NON-BLOCKING — signals picker thread via cbuf, returns immediately. Returns -EINVAL if session already active.	Driver thread
Core → Driver	on_decision(ctx, decision)	Each scheduling decision emitted immediately as produced. Called from allocator thread context.	Allocator thread
(implicit)	picker sends jobid=0 through jbuf	End-of-session marker. Allocator forwards as NULL through decisions cbuf.	Picker → Allocator → Driver

Event buffering discipline: The driver accumulates controller events (capnp push/notify messages from the handler thread) into pend_updates (existing dataq swap mechanism). Between sessions, the driver drains pend_updates into xce_core_event_t array and calls core_push_events(). No events are injected mid-session — the picker thread is blocked on cbuf_get(&wbuf) and the allocator thread on cbuf_get(&jbuf), so core_push_events() has exclusive access to picker state.

Decision publishing: Each decision is published immediately to the controller (not batched at session end). The driver's on_decision callback calls xcecli_jobdispatch() / xcecli_set_pend_reason() / xcecli_preempt() synchronously. This matches today's main.c:181-196 behavior.

Mode selection: Mode is chosen at startup via CLI flags. The core has zero knowledge of which mode is active:

// main.c — mode selection (single branch)
if (arguments.sim_file) {
    sched_sim_driver_run(&sim_driver, arguments);
} else {
    sched_driver_run(&driver, &config, cluster_cfg, handler);
}

Flag	Mode	Purpose
(none)	Normal	Controller-connected (current behavior)
--sim FILE	Simulation	Replay .replay file
--sim-speed N	Simulation	0=fast, 1=real-time, N=N×
--sim-start-ts NS	Simulation	Start replay from logical time
--sim-verify	Simulation	Compare decisions against recorded
--interactive / -i	Simulation	Open REPL for step-through debugging
--sim-alt-events FILE	Simulation	What-if branching
--sim-output FILE	Simulation	Write simulated decisions to .replay file

Migration from old job_elect_fn: The existing xce_sched_plugin_t->job_elect_fn (which fcfs_elect_job implements) is superseded by xce_job_picker_iface_t->next_job(). The old job_elect_fn shall be deprecated with a warning at plugin register time. The new builtin_fcfs picker implements the new interface exclusively. Existing plugins that only implement job_elect_fn continue to work via a compatibility wrapper during the transition period, but new pickers shall implement xce_job_picker_iface_t.

---

Core Interface Contract

1. Mode-agnostic core: The Scheduler core shall have no concept of which mode it runs in. It processes events identically whether the driver is in normal mode (controller-connected, wall-clock time) or simulation mode (replay file, virtual clock). Specifically:

   • There shall be no mode flag (bool simulation_mode or equivalent) anywhere in the core
   • There shall be no if (simulation) { ... } else { ... } code path in the core
   • xce_sched_core_init() shall take identical parameters regardless of mode
   • The core's decision callback shall be the same function pointer type in both modes
   • The core's save_snapshot() and load_snapshot() shall be identical in both modes

2. Passive event-driven interface: The core shall expose a pure event-driven interface — it receives typed events from the driver, updates internal scheduling state, and emits decisions through a driver-registered callback. The core shall have zero direct dependency on: controller communication, capnp serialization, network I/O, filesystem access, or wall-clock time sources.

3. Virtualized clock access: The core shall access time exclusively through xce_core_now(). The core shall never call time(NULL), clock_gettime(), gettimeofday(), or any other wall-clock time function directly.

   Clock virtualization mechanism — two-base + speed formula:

   The core stores four fields (atomically updated by the driver as a group):

   Field	Type	Description
   virtual_clock_base	_Atomic uint64_t	Virtual-time anchor (nanoseconds)
   real_clock_base	_Atomic uint64_t	Wall-clock anchor captured when virtual_clock_base was set (nanoseconds, CLOCK_MONOTONIC)
   speed_num / speed_den	_Atomic uint64_t	Speed multiplier as rational fraction

   xce_core_now(core) computes:

   uint64_t wall_ns = clock_gettime(CLOCK_MONOTONIC);  // only wall-clock call in the core
   uint64_t elapsed = wall_ns - core->real_clock_base;
   return core->virtual_clock_base + (elapsed * core->speed_num) / core->speed_den;
   

   Mode	virtual_clock_base	real_clock_base	speed	Behavior
   Normal	0	0	1/1	xce_core_now() = wall clock (degraded: 0 + (wall - 0) × 1)
   Simulation discrete	event ts	wall at load	0/1	xce_core_now() = frozen at event ts (time does not advance between events)
   Simulation real-time	snapshot ts	wall at load	1/1	xce_core_now() tracks wall clock from virtual anchor
   Simulation fast/slow	snapshot ts	wall at load	N/1 or 1/N	xce_core_now() runs N× faster or slower than wall clock

   The driver atomically updates all four fields before each event dispatch. Multi-thread safety: xce_core_now() reads atomic fields lock-free. Both picker and allocator threads independently compute the same virtual time from the shared bases — no mutex needed.

   All internal time uses go through xce_core_now(): scheduling logic, logging (via tlog format function using userptr), error messages, debug traces. Job sub_time and queue_time are set by the driver before pushing JOBS_PUSH.

   Third-party library constraint: any library linked into the core that needs time must obtain it through xce_core_now() or a core-provided wrapper. Specifically:

   • Logging (tlog): The core shall register its own format function via tlog_reg_format_func(fn, core_ptr) where core_ptr is the xce_sched_core_t*. This format function shall ignore info->time and instead read xce_core_now(core_ptr), converting nanoseconds to struct tm fields via gmtime_r() (deterministic — pure computation).
   • libxce subset: The core shall link only against the minimal data-structure subset of libxce (ordq, dataq, cbuf, xlist, zarray, kvmgr, rtti) — not against netio, delay, xceapi, or idgen which call wall-clock time internally.
   • pthread timed-blocking: The core is multi-threaded. In simulation mode, thread interleaving is made deterministic — pthread_cond_timedwait() and pthread_mutex_timedlock() are not needed in the deterministic execution path.
   • Libraries that cannot avoid calling wall-clock time directly must not be linked into the core; such concerns stay in the driver layer.

4. Core event types — detailed payload specification: The core interface shall accept at minimum these input events. Each event carries a timestamp (ts, nanoseconds since epoch) and a type-specific payload. The field set below is normative for implementation; the exact capnp message schema is deferred to the design phase.

   CONFIG_PUSH — Full configuration push (scheduler startup or config reload):

   Field	Type	Description
   config	xce_conf4sched_t*	All queues, resource thresholds, preemption config, time windows, global policy settings
   → Picker hook: on_config(picker, config)

   JOBS_PUSH — Newly submitted jobs:

   Field	Type	Description
   njobs	int	Number of jobs in this batch
   jobs	xce_job_t*[njobs]	Job records: jobid, user, project, queue, priority, state, queue_time, resreq
   → Picker hook: on_job_push(picker, job) for each job

   WORKERS_PUSH — Worker list update (nodes added/removed):

   Field	Type	Description
   nworkers	int	Number of workers in this update
   workers	xce_worker_t*[nworkers]	Worker records: workerid, host, resources, state, assigned_jobs

   RESOURCES_PUSH — Resource definitions (new resource types registered):

   Field	Type	Description
   n	int	Number of resource definitions
   resources	xce_resource_t*[n]	Resource records: name, type, total, flags

   RESVALS_FULL — Full resource value snapshot (periodic or after large change):

   Field	Type	Description
   n	int	Number of resource value entries
   resvals	xce_resval_t*[n]	Each: workerid, resource_name, available, used, reserved

   RESVALS_DELTA — Incremental resource value update (fine-grained changes):

   Field	Type	Description
   deltas	xce_resval_list_t*	Linked list of individual resval changes (add/subtract)

   JOB_STATE_CHANGE — Job started, exited, or failed:

   Field	Type	Description
   jobid	uint64_t	Job identifier
   old_state	uint32_t	Previous job state (for context)
   new_state	uint32_t	New job state: RUNNING, EXIT, FAIL, DONE, PEND
   exit_code	int32_t	Exit code (meaningful only for EXIT/FAIL)
   → Picker hook: on_job_state_change(picker, job, new_state)

   JOB_UPDATE — In-place job field modification (priority change, queue reassignment):

   Field	Type	Description
   jobid	uint64_t	Job identifier
   changed_fields	uint32_t	Bitmask of fields that changed (JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ)
   job	xce_job_t*	Full job record with updated fields
   → Picker hook: on_job_update(picker, job)

   QUEUE_UPDATE — Queue definition added or modified:

   Field	Type	Description
   queue	xce_queue_t*	Queue record: name, priority, RUNLIMIT, MAX_RUNNING, state, time_window, job_pick_plugin, project_name
   → Picker hook: on_queue_update(picker, queue)

   JOB_RUSAGE — Job resource usage update (for resource-aware pickers):

   Field	Type	Description
   jobid	uint64_t	Job identifier
   rusages	xce_job_rusages_t*	Resource usage metrics: cpu_time, mem_bytes, io_bytes
   → Picker hook: on_job_rusage(picker, job, rusages) (optional — NULL if not registered)

   BEGIN_ROUND — Trigger a scheduling round (session boundary, see AC 18):

   Field	Type	Description
   timestamp_ns	uint64_t	Logical time at which the round should execute

   END_SESSION — Explicit end-of-recording marker (written to replay file, never sent to core):

   Field	Type	Description
   timestamp_ns	uint64_t	Logical time when recording stopped

5. Decision output types: The core shall emit decisions through the on_decision callback. Three decision types exist:

   Decision Type	Fields	Driver Action
   DISPATCH	jobid, workerid, allocations[] (resource, amount per allocation)	xcecli_jobdispatch()
   PEND	jobid, reason (code + message)	xcecli_set_pend_reason()
   PREEMPT	jobid, victims[] (jobid, workerid per victim), reason	xcecli_preempt()

   These are recorded as ReplayDecision messages within ReplaySession.decisions. The exact decision payload schema is deferred to the design phase.

6. Decision output callback: The core shall emit all scheduling decisions (DISPATCH, PEND, PREEMPT) through a xce_core_decision_cb_t callback registered by the driver at init time. Signature: int (*on_decision)(void* driver_ctx, xce_decision_t* decision). This replaces the current internal decisions cbuf poll loop in main.c.

7. Deterministic guarantee: Given identical initial state and identical event sequences with identical timestamps, the core shall produce identical decision sequences. The core is multi-threaded (picker + allocator threads communicating via cbuf). Deterministic replay is achieved through:

   • cbuf capacity=1 serialization: The jbuf cbuf between picker and allocator has capacity 1 (cbuf_init(&jbuf, 1, sizeof(uint64_t))). This acts as a rendezvous — the picker blocks on cbuf_put until the allocator takes the current jobid. Jobs flow through the pipeline one at a time in fully determined order.
   • Stabilized hash table iteration: All hash table / key-value store iteration that affects scheduling output order shall be stabilized (sorted by key before iterating).
   • PRNG for tie-breaking: Any required randomness (e.g., tie-breaking) shall use a PRNG seeded by the driver at init time (core->prng_seed).
   • Same events → same picker state: Since core_push_events() is called between sessions (picker idle), the picker's internal data structures (ordq_t queues, ordq_t pendJobs) are populated identically given identical event sequences. The picker iteration then produces jobids in identical order.

   7.1 Normal-mode ↔ simulation equivalence: Given identical initial configuration (xce_conf4sched_t) and identical event sequences (same job submissions, state changes, resource updates at the same logical positions), the core shall produce identical decision sequences in both normal mode and simulation mode. Specifically:

   • Record a normal-mode scheduling session (AC 10) → produce .replay file R
   • Run simulation on R (AC 13) → produce .replay file S
   • diff <(xceadmin scheduler dump R) <(xceadmin scheduler dump S) → only non-decision metadata may differ (timestamps in ReplayHeader, wall-clock annotations in SessionSummary). Every ReplayDecision shall match identically: same jobid, same workerid, same allocation amounts, same pend reasons.

   This constraint is the validation contract for the entire core/driver architecture. If normal mode and simulation mode diverge, one of three things is broken:

   • The core has a hidden dependency on wall-clock time or controller I/O (violates AC 1)
   • The event recording is lossy — dropping events that matter to scheduling decisions (violates AC 10)
   • The simulation driver's input translation from capnp or .replay format is not faithfully reproducing the normal-mode event stream

   This constraint turns recording + replay from a debugging convenience into a regression test: after any scheduler change, record a live run, replay it in simulation, and verify the decisions are byte-for-byte identical. Any divergence is a bug.

8. Driver event queue: The driver shall retain an internal pending-event queue to buffer events received from the controller between scheduling rounds. Events accumulate in the driver between BEGIN_ROUND invocations.

Normal-Mode Driver

9. Controller integration: The normal-mode driver shall receive capnp push/notify messages from the controller, translate each to the corresponding core event type, and invoke the core's event push interface. The driver shall publish core-emitted decisions back to the controller via the existing xcecli_ctx_t API.

10. Runtime recording control — detailed mechanism: The scheduler shall support runtime recording of all scheduling events and decisions to a replay file for later simulation and analysis.

    Admin command flow: xceadmin CLI connects directly to the controller via TLS + capnp (same as xceadmin config push — no apiserver, no REST). The controller receives OPCode_schedAdmin and forwards it to the scheduler via the existing scheduler_t.ep endpoint. The scheduler processes the request and replies; the controller relays the reply back to xceadmin. This is a request-reply opcode, NOT a one-way notify — recording status returns live scheduler state that the controller does not own.

    Why request-reply (not notify): The existing OPCode_notify is fire-and-forget — no reply path. Recording start/stop could theoretically use notify, but status must return live state (recording, file, event_count, started_at). Using two different mechanisms for three commands on the same subsystem is worse than one consistent request-reply path. Additionally, start/stop benefit from a synchronous reply: start returns the auto-generated filename, and stop returns the final event count.

    Why a dedicated opcode (not reusing OP_upgradeConfig): OP_upgradeConfig carries a config blob through the controller's own config-replication pipeline. Recording control is a scheduler-scoped admin action — it touches no controller state, needs no WAL persistence, and should be forwarded directly to the scheduler without controller-side interpretation. A dedicated OPCode_schedAdmin keeps these concerns separate.

    Cap'n Proto schema additions (in controller.capnp):

    New OPCode:

    enum OPCode {
      ...
      schedAdmin   @42;
    }
    

    New types:

    enum SchedAdminAction {
      recordStart  @0;
      recordStop   @1;
      recordStatus @2;
    }
    
    struct SchedAdminRequest $C.mapname("sched_admin_req_t") {
      action     @0: SchedAdminAction;
      recordFile @1: Text;   # path for recordStart; empty = auto-generate
    }
    
    struct SchedAdminReply $C.mapname("sched_admin_reply_t") {
      ok         @0: Bool;
      error      @1: Text;    # human-readable error if !ok
    
      # Fields populated for recordStatus (also returned by recordStart/recordStop
      # for confirmation — caller can verify state after each action):
      recording  @2: Bool;    # true between start and stop
      file       @3: Text;    # record file path (auto-generated name returned by start)
      eventCount @4: UInt64;  # events written since recording started
      startedAt  @5: UInt64;  # logical time recording started (nanoseconds since epoch)
    }
    

    New Request.body union entry:

    schedadminmsg @31: SchedAdminRequest;
    

    New RequestReply.data union entry:

    schedadminreplymsg @19: SchedAdminReply;
    

    Controller forwarding (in myreq_handler):

    case OPCode_schedAdmin:
        → validate xceadmin auth (admin RBAC, same as config push)
        → lock(&c->sched.lock)
        → if sched.ep == NULL:
              unlock; reply {ok=false, error="scheduler not connected"}
        → forward to sched.ep via existing capnp push/request
        → await scheduler reply (timeout: 5s)
        → unlock
        → relay SchedAdminReply back to xceadmin via RequestReply
    

    The controller is a pass-through — it owns no recording state, interprets no fields beyond the opcode dispatch. All recording logic lives in the scheduler's normal-mode driver. In the new architecture, recording is a driver-layer concern — the core has zero knowledge of whether recording is active.

    Edge cases:

    • Scheduler not connected (sched.ep == NULL): Controller replies {ok=false, error="scheduler not connected"} immediately. No capnp round-trip needed.
    • Scheduler disconnects mid-request: The 5s timeout triggers; controller replies {ok=false, error="request timeout"}.
    • Controller is not primary: xceadmin's existing redirect-following logic handles this — if the contacted controller is not the primary, it redirects to the primary (same as config push). The scheduler is a singleton attached to the primary controller, so recording commands always reach the primary's scheduler.

    xceadmin CLI surface:

    xceadmin scheduler record start [FILE]
        → SchedAdminRequest{action=recordStart, recordFile=FILE or ""}
        → Success: {ok=true, file="<auto or specified>", recording=true,
                    eventCount=0, startedAt=<logical_time>}
        → Error:   {ok=false, error="already recording", recording=true, file="<current>"}
    
    xceadmin scheduler record stop
        → SchedAdminRequest{action=recordStop}
        → Success: {ok=true, recording=false, eventCount=14273, file="<was>",
                    startedAt=<original_start>}
        → Error:   {ok=false, error="not recording", recording=false}
    
    xceadmin scheduler record status
        → SchedAdminRequest{action=recordStatus}
        → Success: {ok=true, recording=true|false, file="<current or empty>",
                    eventCount=N, startedAt=<original_start or 0>}
    

    Reply field semantics by action:

    Field	recordStart (ok)	recordStart (err)	recordStop (ok)	recordStop (err)	recordStatus
    ok	true	false	true	false	true
    error	""	reason (e.g. "already recording")	""	reason (e.g. "not recording")	""
    recording	true	true (unchanged)	false	false (unchanged)	true/false
    file	current path	current path	last path (informational)	""	current or ""
    eventCount	0	current count	final count	0	current
    startedAt	start time	original start	original start (historical)	0	original or 0

    Error codes reference:

    Condition	ok	error
    record start while already recording	false	"already recording"
    record stop while not recording	false	"not recording"
    Scheduler not connected to this controller	false	"scheduler not connected"
    Request timeout (5s)	false	"request timeout"
    Disk full / write error (start)	false	"cannot write to <file>: <errno>"

    Status output (xceadmin formats the capnp reply):

    recording:   yes
    file:        /var/log/xce/scheduler-replay-prod-cluster-1-20260729-020900.replay
    events:      14273
    started at:  2026-07-29T02:09:00
    

    Permissions: All three commands require admin-level authentication (same RBAC role as config push). The controller validates the xceadmin auth before forwarding to the scheduler — the scheduler trusts the controller's forwarding (same trust model as existing notify events from the controller).

    Recording state (in xce_sched_driver_t):

    bool     recording;                    // true between start and stop
    char*    record_file_path;             // strdup'd path for status queries
    FILE*    record_file;                  // replay file handle (NULL when not recording)
    uint64_t record_event_count;           // counter for periodic snapshot
    uint64_t record_snapshot_interval;     // N events between snapshots (0 = disabled)
    uint64_t record_start_ts;             // logical time when recording started (0 = not started)
    

    record_file_path is a strdup'd copy of the auto-generated or user-specified path, owned by the record file lifecycle: strdup on start, free+NULL on stop or error. This field exists so SchedAdminReply.file can be populated even after record_file is closed — it persists for informational queries (e.g., status after a disk-full auto-stop).

    Recording lifecycle — what gets written at each step:

    1. On record start:

       • Take full core snapshot via save_snapshot(core, &snapshot_data) — fills SnapshotData struct in memory
       • Write ReplayHeader as first message of the file (recorder version, config hash, cluster identity, timestamps, sched policy)
       • Serialize SnapshotEntry{ts, data=snapshot_data} and write as second message
       • Set recording = true, record_event_count = 0

    2. On each core_push_events() call (driver thread, between sessions):

       • For each event in the batch, serialize and write the corresponding ReplayEvent message to the .replay file
       • Increment record_event_count for each event written

    3. On core_go() (driver thread, start of session):

       • Begin building a ReplaySession message with beginRound = xce_core_now(core)

    4. In the decision drain loop (driver thread, during session):

       • For each decision from cbuf_get(&decisions): append a ReplayDecision to the current ReplaySession.decisions list
       • Decisions are in allocator output order, before the NULL end-of-session marker

    5. On core_end_session() (driver thread):

       • Finalize the ReplaySession with session summary, write the complete message to the .replay file

    6. On periodic checkpoint (driver thread):

       • If record_event_count > 0 && record_event_count % record_snapshot_interval == 0:
       • Serialize and write SnapshotEntry message from save_snapshot(core, &snapshot_data)

    7. On record stop:

       • Write ReplayEvent{endSession} message
       • fflush() + fclose() the record file
       • Set recording = false, record_file = NULL

    Recording insert points in the normal-mode driver main loop (annotated):

    while (!sched.stop) {
        sched_update(&sched);                          // process capnp push/notify
        // [RECORDING] write each buffered event as ReplayEvent message
        // [RECORDING] periodic checkpoint check
    
        sched_begin_session(&sched);
        // [RECORDING] build ReplaySession{beginRound = now}
        sched_go(&sched);
    
        while (cbuf_get(&sched.decisions, &d), d) {
            // [RECORDING] append ReplayDecision to session
            switch (d->op) {
            case DISPATCH: sched_publish_dispatch(...); break;
            case PEND:     sched_publish_pend_reason(...); break;
            case PREEMPT:  sched_publish_preempt(...); break;
            }
        }
    
        sched_end_session(&sched);
        // [RECORDING] finalize + write ReplaySession message
        // [RECORDING] periodic checkpoint after session
        sleep(1);
    }
    

    Performance: When recording is inactive (recording == false), the scheduler shall incur zero overhead — no file I/O, no extra allocations, no event buffering beyond what the non-recording path already does. The recording check is a single boolean branch per session.

    Error handling: If writing to the record file fails (disk full, permission error), the driver shall log the error, set recording = false, fflush() + fclose() the record file, free(record_file_path) + set record_file_path = NULL, and continue normal scheduling operation. Recording failure shall not cause the scheduler to stop or crash.

11. Periodic snapshot during recording: While recording is active, the driver shall periodically append fresh snapshots to the replay file (default: every N events, configurable via --record-snapshot-interval N, N=0 disables periodic snapshots).

12. Session response time metrics: The core shall track session duration for diagnostics. At core_go(), the core stores session_start = xce_core_now(). At core_end_session(), the core computes session_duration_ns = xce_core_now() - session_start. The duration is written to the ReplaySession.summary field as a SessionSummary struct. The SessionSummary is optional for replay correctness (the simulation driver ignores it) but provides valuable diagnostics:

    • Normal mode: reveals real scheduling latency — is the picker slow? is allocation the bottleneck?
    • Simulation mode: in real-time mode, compare virtual vs wall-clock session duration; in fast mode, the virtual clock freezes during core_go() so duration_ns is typically 0 (all decisions at same ts)
    • Tuning: diff session durations before and after a policy change to quantify performance impact

    The core stores session_start internally — no new API functions, no change to the on_decision callback, no overhead beyond a single xce_core_now() call per session.

Simulation-Mode Driver

13. Simulation mode startup: The scheduler shall support simulation mode activated by --sim FILE and --sim-start-ts TIMESTAMP. The startup sequence is:

    Step 1 — Open and validate replay file:

    sim->replay_file = fopen(arguments.sim_file, "r");
    read first capnp message → validate it is ReplayHeader
    

    If the file is missing, unreadable, or the first message is not a valid ReplayHeader, the simulation driver shall fail with a clear error message.

    Step 2 — Locate starting snapshot:

    scan forward from ReplayHeader, decode each SnapshotEntry
    find the most recent SnapshotEntry with ts <= --sim-start-ts
    

    If no SnapshotEntry exists after ReplayHeader, the simulation driver shall reject the file. Without initial state, simulation has no starting point.

    Step 3 — Load snapshot and rebuild picker:

    xce_sched_core_load_snapshot(&sim->core, snapshot_entry.data)
    // Walk core state to rebuild picker private state:
    for each queue in core.queues:
        picker->on_queue_update(picker, queue)
    for each job in core.jobs where job.state & JOB_STAT_PEND:
        picker->on_job_push(picker, job)
    

    After this step, core state and picker state are fully restored. The core is ready to run.

    Step 4 — Set virtual clock:

    sim->core.virtual_clock_base = snapshot_ts
    sim->core.real_clock_base   = clock_gettime(CLOCK_MONOTONIC)  // wall anchor
    sim->core.speed_num         = arguments.sim_speed  // 0=fast, 1=real-time, N>1=N×
    sim->core.speed_den         = 1
    

    From this point, xce_core_now() returns virtual time computed from the snapshot's logical timestamp.

    Step 5 — Enter event loop (AC 18): accumulate events → BEGIN_ROUND → core_push_events → core_go → drain decisions → core_end_session → repeat until EOF.

    The ReplayHeader + SnapshotEntry together form the complete self-contained starting point. Nothing outside the replay file is needed to begin simulation.

14. Discrete event-driven virtual clock: The simulation driver shall implement a virtual clock that advances by jumping directly to the timestamp of the next queued event — no fixed tick interval, no wall-clock waiting. Strictly monotonically increasing. Same-timestamp events delivered in file order. Supports:

    • --sim-speed N — N=0 as-fast-as-possible, N=1 real-time, N>1 N× real-time
    • --interactive / -i — open REPL for step-through debugging, breakpoints, state inspection, and snapshot capture (AC 22)

15. Deterministic replay verification: In simulation mode with --sim-verify, the driver shall compare each core-emitted decision against the recorded decisions in the replay file. Any mismatch shall be reported as a regression with exact logical-time and decision-detail context.

16. What-if branching: The simulation driver shall accept --sim-alt-events FILE. When both a base .replay file (for snapshot loading) and an alt-events file are provided, the driver loads the snapshot from the base file, then replays events from the alt-events file (same .replay capnp stream format) — enabling "what-if" analysis.

17. Simulation isolation: In simulation mode, the scheduler process shall not establish any connection to the controller. The xce_sched_sim_driver_t shall contain no xcecli_ctx_t, no xce_cluster_config_t, no capnp socket, and no RPC to any controller. All scheduling input comes exclusively from the replay file. All scheduling decisions go to the simulation record file (see AC 19) and/or stdout. Attempting to use a controller-facing API from simulation mode is a programming error that shall be caught at compile time (via struct field absence) or fail with a clear error at startup.

18. BEGIN_ROUND as session boundary: The simulation driver shall treat each ReplaySession message as the trigger for one complete scheduling session. The driver accumulates ReplayEvent messages into a batch, and upon encountering ReplaySession, executes the full session cycle:

    // Simulation driver event loop
    xce_core_event_t buf_events[MAX_EVENTS_PER_ROUND];
    int buf_n = 0;
    
    while (decode_next_message(replay_file, &msg) == 0) {
        switch (msg.which) {
        case ReplayEvent:
            buf_events[buf_n++] = replay_event_to_core_event(&msg.event);
            break;
    
        case ReplaySession:                                     // ← SESSION BOUNDARY
            // 1. Push accumulated events (picker hooks called synchronously)
            core_push_events(&sim->core, buf_events, buf_n);
    
            // 2. Kick scheduling round (non-blocking signal to picker thread)
            core_go(&sim->core);
    
            // 3. Drain decisions until end-of-session marker (NULL)
            xce_decision_t* d;
            int di = 0;
            while (cbuf_get(&sim->core.decisions, &d) == 0 && d != NULL) {
                // 4. Compare against recorded decisions (if verify enabled)
                if (sim->verify && di < msg.session.decisions.len) {
                    xce_decision_t expected = from_replay_decision(msg.session.decisions[di]);
                    if (!decision_eq(&expected, d))
                        log_mismatch(sim, &expected, d);       // --sim-verify
                    di++;
                }
                if (sim->record_file)
                    write_replay_decision(sim->record_file, d); // record output
                free_decision(d);
            }
    
            // 5. End session (GC, reset workers)
            core_end_session(&sim->core);
            buf_n = 0;                                        // reset for next round
            break;
    
        case SnapshotEntry:
            // Mid-replay snapshots (periodic checkpoints): skipped during replay.
            // Only the initial SnapshotEntry (loaded at startup) is consumed.
            break;
        }
    }
    

    The driver thread blocks during the decision drain loop (step 3-4) because cbuf_get(&decisions) blocks until the allocator produces a decision or the end-of-session marker. The core_go() call itself is non-blocking (it only signals the picker thread via cbuf_put(&wbuf) and returns immediately). After all decisions are drained and the NULL marker arrives, core_end_session() runs cleanup, and the driver resumes reading the next message from the .replay file.

    Decision ordering constraint: ReplayDecision messages appear within the ReplaySession.decisions list. Their order matches the allocator thread's output order through the decisions cbuf. A ReplayDecision appearing outside a ReplaySession is a format error.

19. Simulation record file: In simulation mode with --sim-output FILE, the driver shall write a .replay file in the identical capnp stream format as the input replay file (ReplayHeader → SnapshotEntry → ReplayEvent → ReplaySession cycles). This record file captures the simulation's actual decisions, enabling:

    • Diff-based debugging: diff <(xceadmin scheduler dump original.replay) <(xceadmin scheduler dump simulated.replay) reveals exactly which decisions changed
    • Tuning feedback loop: Modify picker parameters, re-run simulation, diff the record files to quantify the impact
    • What-if comparison: Compare --sim-alt-events run output against the base run output

    The record file starts with a ReplayHeader (carrying scheduler version + config hash from the sim run, NOT copied from the replay file), followed by the same SnapshotEntry that was loaded, then events + ReplaySession cycles exactly as produced by the core. If --sim-output is not specified, decisions are printed to stdout only and no record file is written.

    Symmetry with normal-mode recording: Both the normal-mode driver (when recording is active, AC 10) and the simulation-mode driver (with --sim-output) produce .replay files in the identical capnp stream format. The same file layout, same message types, same decision encoding. This symmetry is intentional:

    Aspect	Normal-mode recording	Simulation --sim-output
    Trigger	scheduler record start (runtime)	--sim-output FILE (CLI at startup)
    Input source	Controller capnp events	Replay file events
    Writes events?	Yes — records all input events as ReplayEvent messages	Yes — passes through events from replay file as ReplayEvent messages
    Writes decisions?	Yes — records scheduler decisions in ReplaySession messages	Yes — records simulated decisions in ReplaySession messages
    Output file	.replay file (for later simulation)	.replay file (for diff analysis)
    ReplayHeader origin	Live cluster metadata	Sim run metadata
    SnapshotEntry origin	save_snapshot(core) at record start	SnapshotEntry from replay file
    Purpose	Capture reality	Capture simulation result

    This means diff <(xceadmin scheduler dump original.replay) <(xceadmin scheduler dump simulated.replay) works out of the box — both files have the same structure, same timestamps, same event types. Only the decisions may differ, which is exactly what you want to compare.

Replay File Format

20. Single-file capnp stream format (.replay): The replay file shall be a single file containing consecutive Cap'n Proto messages in the standard framing (segment table + data per message). Each message is one of the types below. The file is appendable — recording writes each message as it occurs; no need to know final size upfront.

    File suffix: .replay — self-documenting, no collision with schema files (.capnp), configuration blobs (.bin), or text formats (.json, .jsonl).

    Schema (in controller.capnp):

    struct ReplayHeader {
      recorderVersion @0: Text;     # scheduler binary version that wrote this file
      configHash      @1: Data;     # sha256 of xce_conf4sched_t
      cluster         @2: ReplayCluster;
      recordingStartTs @3: UInt64;  # logical time recording started (ns)
      recordingStartWall @4: Text;  # wall-clock time recording started (ISO 8601)
      schedPolicy     @5: Text;     # active picker plugin name (e.g. "builtin_fcfs")
    }
    
    struct ReplayCluster {
      name        @0: Text;
      controllers @1: List(Text);   # ["host1:9991", "host2:9992", "host3:9993"]
      recordingHost @2: Text;       # controller hostname that made the recording
    }
    
    struct SnapshotEntry {
      ts   @0: UInt64;              # logical time of snapshot
      data @1: SnapshotData;        # full core state
    }
    
    struct ReplayEvent {
      ts    @0: UInt64;             # logical time
      body  @1: union {
        configPush     @2: Conf4Sched;
        jobsPush       @3: SchedJobData;
        workersPush    @4: WorkerData;
        resourcesPush  @5: ResourceData;
        resvalsFull    @6: ResValList;
        resvalsDelta   @7: ResValList;
        jobStateChange @8: JobStateChangeEvent;
        jobUpdate      @9: JobUpdateEvent;
        queueUpdate    @10: QueueConf;
        jobRusage      @11: SchedJobRusages;
        endSession     @12: Void;     # recording stop marker
      }
    }
    
    struct ReplaySession {
      beginRound  @0: UInt64;          # logical time (ts value)
      decisions   @1: List(ReplayDecision);
      summary     @2: SessionSummary;  # optional diagnostic metadata
    }
    
    struct ReplayDecision {
      ts   @0: UInt64;
      body @1: union {
        dispatch @2: DispatchDecision;
        pend     @3: PendDecision;
        preempt  @4: PreemptDecision;
      }
    }
    
    struct SessionSummary {
      durationNs  @0: UInt64;
      nDecisions  @1: UInt32;
      nDispatched @2: UInt32;
      nPended     @3: UInt32;
    }
    

    File structure: The .replay file is a single stream of consecutive capnp messages:

    [msg 1]  ReplayHeader
    [msg 2]  SnapshotEntry          ← initial state (MUST be present, MUST be first after header)
    [msg 3]  ReplayEvent            ← input event
    [msg 4]  ReplayEvent            ← input event
    ...
    [msg N]  ReplaySession           ← BEGIN_ROUND → decisions → SESSION_SUMMARY
    [msg N+1] ReplayEvent
    ...
    [msg M]  SnapshotEntry           ← periodic checkpoint (every N events, AC 11)
    ...
    [msg K]  ReplayEvent{endSession} ← recording stop marker
    

    • ReplayHeader: first message. Identifies the file type (.replay files start with a known capnp struct, not an ambiguous byte sequence). Carries metadata about the recording origin — recorder version, config hash, cluster identity, timestamps, active scheduling policy.
    • The .replay file is fully self-contained: the ReplayHeader identifies which config was active (configHash), and the first SnapshotEntry carries the full config payload plus all scheduling state. No external config file, controller, or database is needed to replay — one file has everything.
    • SnapshotEntry: the first data message after ReplayHeader MUST be a SnapshotEntry. The simulation driver shall reject any .replay file whose second message is not a SnapshotEntry — without initial state, simulation has no starting point. Mid-stream SnapshotEntry messages (from periodic recording, AC 11) provide seek points for --sim-start-ts.
    • ReplayEvent: input event carrying payload in a discriminated union. The body union tag selects the event type (configPush, jobsPush, etc.). No separate "type" field needed — the capnp union tag is the type discriminator.
    • ReplaySession: groups a BEGIN_ROUND trigger with its resulting decisions and session summary. A ReplaySession between ReplayEvent messages marks a session boundary; a ReplayDecision outside a ReplaySession is a format error. Decision order within decisions matches allocator output order.
    • ReplayEvent{endSession}: final message in a completed recording. Signals clean shutdown. The simulation driver ignores it (replay runs until EOF).
    • No text parsing, no JSON schema, no two-format code paths — the same capnp decode path reads every message. The simulation driver decodes each message sequentially from the stream.

    Human inspection: xceadmin scheduler dump <FILE> decodes a .replay file to human-readable text. xceadmin already bundles controller.capnp schemas for config push/config build — scheduler dump adds one new subcommand reusing the same schema infrastructure.

    Recording (append): The .replay format is append-friendly — save_snapshot() and event writes serialize one capnp message at a time. On record start: write ReplayHeader + initial SnapshotEntry. Each event: write one ReplayEvent. Each session: write one ReplaySession. Periodic checkpoint: write one SnapshotEntry. On record stop: write ReplayEvent{endSession} + fflush + fclose. Recording failure mid-write: the file is truncated to the last complete message — no partial-message corruption.

    Simulation (read): --sim-start-ts NS seeks forward through SnapshotEntry messages (deserialize each, check ts <= NS, keep latest match), loads the matching snapshot, then replays all messages with ts >= NS.

State Snapshot

21. Snapshot serialization: The core shall support save_snapshot() producing a SnapshotData capnp struct containing all scheduler configuration (xce_conf4sched_t* — queues, priorities, resource thresholds, preemption config, time windows), jobs, workers, resources, resvals, users, projects, shared-resource tracking state, unsatisfiable-resreq cache, PRNG state, and session state. The core shall support load_snapshot() restoring from SnapshotData to that exact state.

    The snapshot is fully self-contained for simulation. After load_snapshot() and picker rebuild, the core is ready to process core_push_events() + core_go() and produce decisions — no other state, external config, or preceding events are needed. Given a .replay file, the simulation driver can start from any SnapshotEntry (via --sim-start-ts): load the snapshot, rebuild picker, set virtual clock to snapshot ts, then replay all messages with ts >= snapshot.ts. The result is identical to replaying from the beginning — the snapshot is a complete checkpoint.

    Picker state is NOT stored in snapshot. The builtin picker's private state (ordq_t queues, per-queue ordq_t pendJobs) is derived from core state: every queue → on_queue_update, every pending job → on_job_push. After load_snapshot() restores core state, the driver shall rebuild the picker's private state by walking the core's queues and jobs and calling the picker's state-change hooks. This ensures:

    • Snapshots capture only canonical core state (no duplicate of picker internals)
    • Picker rebuild is deterministic: same core state + same hook sequence = same picker state
    • Custom pickers with additional state can rebuild from the same core-state walk

    Snapshot contents checklist — everything required to run a scheduling session:

    Category	Contents
    Config	xce_conf4sched_t* — all queues, priorities, resource thresholds, preemption config, time windows
    Jobs	All job records with current state, priority, queue assignment, queue_time
    Queues	All queue definitions with RUNLIMIT, MAX_RUNNING, state, per-queue picker override
    Workers	All worker records with resources, state, assigned jobs
    Resources	Resource definitions (resreq templates)
    Resvals	Current resource value snapshots for all workers
    Users	User records with resource limits, group membership
    Projects	Project scaffolding with priority (v1: flat, v2: hierarchical quota)
    Shared-resource	Shared resource tracking state across workers
    Unsatisfiable cache	Resreqs that could not be satisfied in previous rounds (for early skip)
    Session state	Current scheduling round, iterator positions, GC state
    PRNG state	Current PRNG value (for deterministic tie-breaking resume mid-file)

22. Interactive simulation REPL: In simulation mode with --interactive (or -i), the scheduler shall open a read-eval-print loop on stdin/stdout after loading the replay file but before the first event is processed. The REPL gives the operator full control over simulation execution — single-step, breakpoints, state inspection, and snapshot capture — turning the simulation driver into a real debugger for scheduling policy.

    Interaction model: The simulation process blocks on stdin at startup (pre-replay pause) and at each breakpoint hit. No controller, no network, no authentication — this is a local investigation tool. The REPL prompt is (sim) .

    Minimum command set:

    Execution control:

    Command	Description
    step [N]	Advance N sessions (default 1), pause after. Processes all events up to and including the Nth BEGIN_ROUND from the current position, then returns to REPL.
    continue	Run until next breakpoint hit or EOF. If no breakpoints are set, runs to completion.
    quit	Exit the scheduler immediately. No cleanup guarantees beyond fclose on open files.

    Breakpoints:

    Command	Description
    break <event_type> [key=value ...]	Set breakpoint on event type (e.g., JOB_STATE_CHANGE, JOB_PUSH, BEGIN_ROUND, DECISION). Optional key=value filters narrow the match: break JOB_STATE_CHANGE jobid=12345, break JOB_STATE_CHANGE state=RUN, break DECISION type=DISPATCH queue=normal. Multiple filters are AND'd. Key names match the field names in the capnp schema for each message type.
    info break	List all breakpoints with their IDs, event types, and filters.
    delete <id>	Delete breakpoint by ID. delete all removes all breakpoints.

    State inspection:

    Command	Description
    info session	Show current simulation position: logical time (xce_core_now()), event index (Nth message processed), session count, last SessionSummary if available.
    info job <jobid>	Dump one job: state, queue, priority, queue_time, resreq, allocations. Jobid uses the same format as xcecli (e.g., 12345).
    info queue <name>	Dump one queue: priority, state (open/closed), RUNLIMIT, MAX_RUNNING, job_pick_plugin, list of queued job IDs with priorities. info queue all lists all queues.
    info worker <name>	Dump one worker: state, resources (total + available), assigned jobs. info worker all lists all workers.
    info resource <name>	Dump one resource: type, total capacity, available, allocation count. info resource all lists all resources.
    info summary	Quick overview: N_jobs (M pending, K running), N_workers (M online), N_queues, N_resources.

    Snapshots:

    Command	Description
    snapshot [FILE]	Write a full save_snapshot() to FILE as a standalone .replay file (contains ReplayHeader + SnapshotEntry). If FILE is omitted, auto-generate: <replay_basename>-snap-<event_index>.replay. The snapshot is a complete state dump (same format as periodic recording checkpoints). The operator can restart simulation from this snapshot via --sim <snapshot_file> --sim-start-ts <time_at_snapshot>.

    Help:

    Command	Description
    help	Print available commands with one-line descriptions. Covers all REPL commands: execution control (step, continue, quit), breakpoints (break, info break, delete), state inspection (info session/job/queue/worker/resource/summary), snapshots (snapshot), sim parameters (set, info config).

    Simulation parameters (interactive set / info config):

    Command	Description
    set speed <N>	Set simulation speed from this point forward. 0 = as-fast-as-possible (no wall-clock delay), 1 = real-time, N = N× real-time. Overrides the --sim-speed CLI value.
    set output <FILE>	Start writing simulation decisions to FILE in the same .replay capnp stream format as --sim-output. Opens ReplayHeader+SnapshotEntry at the current position if the file is new. File is written incrementally — each subsequent session's decisions land in the file.
    set output off	Close the current output file. Subsequent decisions are not recorded. info config confirms the file path and event count at close time.
    set verify on|off	Enable or disable decision verification against the recorded replay file. When enabled, each decision is compared against the corresponding recorded decision at the same logical time. Mismatches are printed immediately. Default: matches --sim-verify CLI flag.
    info config	Show current sim parameters: `speed=N, output=<path or "off">, verify=on

    Why interactive parameters matter: Without these, tuning simulation requires restart: change --sim-speed, re-run from snapshot, re-check results. With set speed and set output, the operator can fast-forward through known-good regions (set speed 0), slow down for suspicious sections (set speed 0.1), and only start recording output when the interesting behavior starts — all in one session. This turns the REPL from a passive inspector into an active investigation tool where parameters co-evolve with the operator's understanding of the trace.

    REPL integration with the simulation loop:

    load replay file + snapshot
    if --interactive:
        print "[sim] loaded: N events, snapshot @ ts=..."
        print "[sim] ready. type 'help' for commands."
        repl_pause()          // pre-replay pause — operator can set breakpoints
    
    for each event in replay_file:
        // ── sim parameter: speed ──
        if sim->sim_speed_num > 0:
            wall_clock_delay(delta_ns * sim->sim_speed_den / sim->sim_speed_num)
    
        if check_breakpoints(event):
            print "[sim] hit breakpoint @ event #I: <event_type> <key>..."
            repl_pause()
    
        core_push_events(core, event)
    
        if msg.which == ReplaySession:
            core_go(core)
            for each decision:
                // ── sim parameter: verify ──
                if sim->verify:
                    compare_and_report(decision, expected)  // AC 15; REPL 'set verify on/off'
    
                // ── sim parameter: output ──
                if sim->output_file:
                    write_replay_decision(sim->output_file, decision)  // AC 19; REPL 'set output'
    
                if check_breakpoints(decision):
                    print "[sim] hit breakpoint @ decision: <type> job=<jobid>..."
                    repl_pause()
    
    if --interactive:
        print "[sim] replay complete. N_sessions processed."
        if sim->output_file:
            print "[sim] output written to: <path>"
    

    What is NOT in v1:

    • Write-side mutations to scheduler state: No set queue priority, set job state, add job, remove job. The set command in v1 only tunes simulation parameters (speed, output, verify) — it never mutates core scheduling state. Operator inspects state but cannot alter it. What-if analysis uses --sim-alt-events (AC 16) instead. State mutation from REPL is a follow-up.
    • Reverse execution: No step back or time-travel undo. Operator can restart from snapshots captured at earlier points.
    • Expression evaluation: No print core->sessions_count or arbitrary C evaluation. Commands are fixed.
    • Scripting/batch mode: No source FILE or -c "step 5; info summary". The REPL is interactive-only. For automated simulation runs, use the batch flags (--sim, --sim-output, --sim-verify) without --interactive.
    • No semantic snapshot diff in xceadmin: xceadmin scheduler dump produces human-readable output. Comparing two dumps (diff <(xceadmin scheduler dump snap1) <(xceadmin scheduler dump snap2)) works but is generic text diff — no scheduler-aware semantic diff ("queue priority changed from 10 to 20", "3 new jobs added"). Semantic diff is a v2 feature.

    Deterministic replay is preserved: The REPL never alters core state or event stream. It only controls when events are processed (by pausing), not what events are processed. The same replay file with the same breakpoints ignored (treated as skip) produces identical decisions as batch mode. Interactive use with step and continue that runs to completion produces the same output as --sim without --interactive.

    Implementation note: The REPL is a single fgets loop in the simulation driver. It is not a separate thread — when the REPL blocks on stdin, the simulation loop is parked. This is correct and intentional: the operator is in control, and the simulation has no wall-clock deadlines.

Full object sharing (resreq template registry, pend reason dedup pool) is deferred to a follow-up spec. However, the architecture must not block sharing from being added later:

1. Resreq immutability: Resreq objects shall be treated as immutable once created. No code path shall mutate a resreq in place. This ensures sharing via const pointer is safe to introduce later.

2. Pend reason factory: All pend reason allocation shall go through make_pendreason() (the single factory in lib/preason.c). Callers never calloc(sizeof(xce_pendreason_t)) directly.

3. Pointer-identity fast path: Code comparing two resreqs or pend reasons shall use pointer equality (a == b) as a fast path before deep comparison.

4. Select expression memoization: The existing select_eval_ctx_t per-session memoization shall be preserved and extended. Parsed/evaluated form attached to resreq on first evaluation, reused for subsequent evaluations.

5. Snapshot forward-compatibility: The snapshot format shall not preclude object sharing. Shared objects shall be serializable by identity (object table with IDs).

These invariants impose zero overhead today but rule out design choices that would make sharing expensive or impossible to retrofit.

Data Structure Changes Summary

xce_scheduler_t additions

char* job_pick_plugin;         // default: "builtin_fcfs" (Req 2)

xce_sched_queue_t additions

char*    project_name;          // owning project (picker uses internally)
char*    job_pick_plugin;       // per-queue picker override (NULL = use global)

xce_sched_project_t additions

int32_t  priority;              // inter-project ordering (future spec)
// future: resource_quota, resource_used, parent_project

xce_job_picker_iface_t (new)

typedef struct {
    int  (*init)(void* picker, xce_sched_core_t* core);
    void (*fini)(void* picker);
    int  (*begin_session)(void* picker);
    xce_sched_job_t* (*next_job)(void* picker);
    int  (*end_session)(void* picker);
    int  (*on_dispatch)(void* picker, uint64_t jobid);
    int  (*on_pend)(void* picker, uint64_t jobid);
    int  (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
    int  (*on_job_push)(void* picker, xce_sched_job_t* job);
    int  (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
    int  (*on_job_update)(void* picker, xce_sched_job_t* job);
    int  (*on_config)(void* picker, xce_conf4sched_t* config);
    int  (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages);
} xce_job_picker_iface_t;

Core/Driver split

// Core event (driver → core)
typedef enum { XCE_CORE_EVENT_CONFIG_PUSH = 0, XCE_CORE_EVENT_JOBS_PUSH,
    XCE_CORE_EVENT_WORKERS_PUSH, XCE_CORE_EVENT_RESOURCES_PUSH,
    XCE_CORE_EVENT_RESVALS_FULL, XCE_CORE_EVENT_RESVALS_DELTA,
    XCE_CORE_EVENT_JOB_STATE_CHANGE, XCE_CORE_EVENT_JOB_UPDATE,
    XCE_CORE_EVENT_QUEUE_UPDATE, XCE_CORE_EVENT_JOB_RUSAGE,
    XCE_CORE_EVENT_BEGIN_ROUND } xce_core_event_type_t;

typedef struct {
    xce_core_event_type_t type;
    uint64_t              ts;
    union { /* event-type-specific payload */ } payload;
} xce_core_event_t;

typedef int (*xce_core_decision_cb_t)(void* driver_ctx, xce_decision_t* decision);

typedef struct {
    // Scheduling state (moved from xce_scheduler_t)
    xce_conf4sched_t* config; var plugins; var jobs, queues, workers, users, projects;
    // ... (full state)

    // Core/driver boundary
    void*                  driver_ctx;
    xce_core_decision_cb_t on_decision;

    // Internal threading — cbuf producer/consumer
    pthread_t picker_thread, allocator_thread;
    cbuf_t    wbuf;       // driver→picker trigger (int, cap=1)
    cbuf_t    jbuf;       // picker→allocator job handoff (uint64_t, cap=1)
    cbuf_t    decisions;  // allocator→driver decision output (xce_decision_t*, cap=5)
    _Atomic int session_active;  // 1 during session, 0 between

    // Clock virtualization — atomically updated by driver
    _Atomic uint64_t virtual_clock_base, real_clock_base, speed_num, speed_den;
    uint64_t         prng_seed;

    // Picker plugin slot
    char* job_pick_plugin;
} xce_sched_core_t;

typedef struct {
    xce_sched_core_t     core;
    xcecli_ctx_t         apictx;
    xce_cluster_config_t cluster_cfg;
    dataq_t*             pend_updates[2];
    // Recording state (driver-layer only — core is unaware)
    char*    record_file_path;       // strdup'd path for status queries (AC 10)
    FILE*    record_file;            // replay file handle (NULL when not recording)
    bool     recording;
    uint64_t record_event_count;
    uint64_t record_snapshot_interval;
    uint64_t record_start_ts;
} xce_sched_driver_t;

typedef struct {
    xce_sched_core_t  core;
    FILE *replay_file, *alt_events_file, *record_file;
    // sim parameters (mutable via REPL 'set' command — AC 22)
    uint64_t sim_speed_num, sim_speed_den;  // 0/1=fast, 1/1=real-time, N/1=Nx
    bool     verify;                         // decision comparison on/off
    bool     interactive;                    // REPL active
} xce_sched_sim_driver_t;

// Core interface
int  xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed);
int  xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents);  // batch push before session
int  xce_core_go(xce_sched_core_t*);              // NON-BLOCKING: signal picker thread
int  xce_core_end_session(xce_sched_core_t*);     // GC, reset workers
uint64_t xce_core_now(xce_sched_core_t*);         // virtualized clock accessor
int  xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out);  // fills out with core state
int  xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in);  // restores from snapshot
void xce_sched_core_fini(xce_sched_core_t*);

Architecture Diagrams

Internal Core Thread Architecture (Multi-Threaded, cbuf-Based)

┌──────────────────────────────────────────────────────────────────┐
│                          CORE (process)                            │
│                                                                    │
│  [PICKER THREAD]           [ALLOCATOR THREAD]                      │
│                                                                    │
│  cbuf_get(&wbuf) ──┐       cbuf_get(&jbuf) ──┐                    │
│  begin_session()   │       try_allocate()     │                    │
│  while next_job(): │       can_preempt()      │                    │
│    cbuf_put(&jbuf,─┼──►    cbuf_put(          │                    │
│          &jobid)    │       &decisions,──► on_decision callback    │
│  cbuf_put(&jbuf, 0) │       &decision)        │                    │
│  end_session()     │       (NULL on EOS)     │                    │
│  ┌─────────────────┘       ┌─────────────────┘                    │
│  │                         │                                       │
│  │  wbuf        jbuf       │  decisions                           │
│  │ (int)      (uint64_t)   │  (xce_decision_t*)                   │
│  │ cap=1       cap=1       │  cap=5                               │
│  └─────────────────────────┴──────────────────────────────────────│
│                                                                    │
│  [Clock atoms — lock-free]        [Core state]                    │
│  virtual_clock_base               queues, jobs, workers           │
│  real_clock_base                  plugins, resources, resvals     │
│  speed_num, speed_den             picker plugin instance          │
│  prng_seed                        users, projects                 │
└──────────────────────────────────────────────────────────────────┘

Core Scheduling Loop (per-round)

                    ┌─────────────────────┐
                    │   DRIVER THREAD     │
                    │                     │
                    │ core_push_events()  │  ← batch: CONFIG_PUSH, JOBS_PUSH,
                    │   picker hooks      │    JOB_STATE_CHANGE, QUEUE_UPDATE,
                    │   called sync       │    WORKERS_PUSH, RESVALS_*, etc.
                    │                     │
                    │ core_go() ──────────┼── cbuf_put(&wbuf, &go)
                    │   NON-BLOCKING      │
                    │                     │
                    │ while (1):          │
                    │   cbuf_get(&decisions)── block per decision
                    │   if NULL: break    │    ← end-of-session marker
                    │   on_decision(d) ───┼── publish to controller NOW
                    │                     │    (xcecli_jobdispatch /
                    │ core_end_session()  │     xcecli_set_pend_reason /
                    │                     │     xcecli_preempt)
                    │ sleep(1) or next    │
                    │ event trigger       │
                    └─────────────────────┘

Core/Driver Architecture

┌──────────────────────────────────────────────────────────────┐
│                     DRIVER LAYER                               │
│  Normal Mode                  Simulation Mode                  │
│  controller capnp ──┐         replay.replay ──┐               │
│  handler → events   │         virtual clock   │                │
│  → publish decisions│         → record/log     │                │
│       │       ▲     │              │       ▲   │                │
└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘
        │       │ decisions          │       │ decisions
        ▼       │                    ▼       │
┌──────────────────────────────────────────────────────────────┐
│                   SCHEDULER CORE                               │
│  core_push_event() → picker hooks → picker->next_job()        │
│  → try_allocate() → on_decision() → DISPATCH / PREEMPT / PEND │
│  ZERO dependencies on: controller, capnp, network, wall-clock, filesystem │
└──────────────────────────────────────────────────────────────┘

Future: Project Resource Ownership

Deferred to a separate project-resource-ownership spec. When implemented:

• Projects form a tree with resource quota at each node
• A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue
• The core scheduling loop requires zero code changes — only a new picker plugin replaces builtin_fcfs
• queue->project_name (added here) provides the queue→project mapping the v2 picker consumes
• xce_sched_project_t scaffolding (added here) provides the struct the v2 picker extends

The architecture is designed so that v1 (flat, builtin_fcfs ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. The only difference is which picker plugin is configured.