myxce/.kiro/specs/scheduler-framework/requirements.md

Requirements Document

Introduction

The XCE scheduler has a plugin architecture (plugin.h/plugin.c, xce_sched_plugin_t) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: sched_job_picker() (scheduler.c:39) sets job_pick_plugin = "xce_sched_fcfs" as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: parjob_allocator.c:73's my_alloc() takes (plugin, ctx) and immediately return 0 — any job with maxTasks > 1 (routed there by sched_make_primary_alloc_() at scheduler.c:93-103) produces no allocation at all. Third, allocation is eager and one-shot: the sibling seqjob_allocator.c:107 my_alloc() (used when maxTasks == 1) calls sched_collect_workers() then xlist_find() for the first host that satisfies the request and stops — there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.

This feature closes all three gaps and adds two architectural foundations:

Allocation: A lazy xce_alloc_plan_iter_t (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with a lazy xce_host_iter_t (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a parjob_allocator implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session.

Job picking: A pluggable xce_job_picker_iface_t — the core scheduling loop calls picker->next_job() to get the next schedulable job, with zero knowledge of how jobs are selected (FCFS, priority, fair-share, deadline, etc.). The default builtin_fcfs picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. Preemption fires at Level 3 when allocation fails for a preemptible job.

Core/driver separation: The scheduler core is a passive, event-driven engine with zero controller I/O dependencies. A normal-mode driver connects it to a live controller; a simulation-mode driver replays recorded cluster events through a discrete event-driven virtual clock, enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster.

Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in .kiro/specs/scheduler-framework/brief.md.

Queue vs Project — Roles

These are fundamentally different concerns:

	Queue	Project
Role	Scheduling mechanism	Resource ownership
Question	HOW are jobs ordered and constrained?	WHO owns resources and gets charged?
Scope	One hardware/scheduling profile	Cross-queue, cross-cluster
Key attributes	Priority, limits, time windows, picker, preemption config	Quota, usage, hierarchy, members (future)
Cardinality to job	1 job → 1 queue	1 job → 1 project
Cardinality to each other	Many-to-many	Many-to-many

• Queue = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, priority (for inter-queue ordering within the picker), per-queue picker override, and preemption config.
• Project = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. One project submits to many queues; one queue receives from many projects. The picker plugin may use queue->project_name internally for project-scoped iteration (v2).

Boundary Context

• In scope: Pluggable job picker interface (xce_job_picker_iface_t with next_job()) hiding all selection logic behind a single plugin; xce_alloc_plan_iter_t and xce_host_iter_t interfaces (lazy, cancel-safe, composable); seqjob_allocator rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, maxTasks == 1 only); parjob_allocator's my_alloc() implementing real combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1) for maxTasks > 1; single builtin picker (builtin_fcfs) replicating today's behavior through the new interface; configurable picker plugin with per-queue override (queue->job_pick_plugin); preemption decision type (XCE_DECISION_PREEMPT) flowing scheduler→controller→worker with SIGSTP mechanism; cross-queue victim selection (LSF-style per-queue preemptible_queues list); session-level unsatisfiable-resreq fingerprint cache keyed on {select, rusage, user, queue, project}; core/driver separation with passive event-driven core interface, normal-mode driver (controller-connected), simulation-mode driver (replay recorded events with discrete event-driven virtual clock), runtime recording control via xceadmin commands, JSONL replay file format, state snapshot/restore for replay efficiency, and deterministic replay guarantee for debugging/tuning; documentation of picker plugin interface contract (hooks, cancel-safe, session-aware); unit tests for plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, event recording/replay, snapshot/restore, and deterministic replay; integration test for end-to-end multi-slot parallel job allocation and preemption flow.

• Out of scope: New job selection policies beyond the builtin FCFS picker (priority, fair-share, deadline, backfill) — the pluggable picker interface enables them, but none ship in this spec beyond builtin_fcfs; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring for scheduling decisions (separate roadmap hardening item, tracked adjacent to ha-consistency-invariants); COMPOUND/ALTER resreq allocation changes beyond what composing over PRIMARY already provides; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins); hierarchical project resource quotas (deferred to project-resource-ownership spec — only project_name on queue and xce_sched_project_t scaffolding ship here); kill-and-requeue preemption escalation (deferred — v1 is SIGSTP only); deadline field on xce_sched_job_t (deferred — needed for deadline-based pickers); event payload JSON schema (open design question — deferred to design phase); distributed/cluster-mode simulation (v1 is single-scheduler replay only).

• Adjacent expectations: depends on the existing plugin framework (plugin.h/plugin.c), the existing select-expression evaluator (resreq.h, memoized via select_eval_ctx_t), and existing shared-resource tracking (xce_scheduler_t.sharedInst/sharedInUse) — none of these change. scheduler-controller-msg-gap (implemented) governs controller↔scheduler message handling this feature reads state through but does not modify. ha-consistency-invariants' commit barrier is a future consumer of scheduling decisions; this feature does not wire it. Controller hook HOOK_DETAIL_JOB_STOP (enum value 65 in cluster_def.h) already exists and is reused for preemption stop actions.

Allocation Architecture (Reqs 2–4)

Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER):

Resreq Type Hierarchy

ALTER     →  OR-of-compounds: first compound that fully allocates wins
  └─ COMPOUND →  AND-of-primaries: ALL primaries must allocate for success
       └─ PRIMARY  →  plan_iter → host_iter → try_alloc  (leaf level)

The iterator composition (xce_alloc_plan_iter_t + xce_host_iter_t) operates at the PRIMARY leaf. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today's sched_make_compound_allocation() and sched_make_alter_allocation() dispatch.

Iterator Composition (PRIMARY leaf)

                    ┌──────────────────────────┐
                    │  xce_alloc_plan_iter_t    │  Req 2
                    │                            │
                    │  init(primary_resreq)      │
                    │  next() → plan             │  generates (nhosts, slots_per_host)
                    │  fini()                    │  candidates, most-preferred first
                    └────────────┬───────────────┘
                                 │ plan
                                 ▼
                    ┌──────────────────────────┐
                    │  xce_host_iter_t          │  Req 3
                    │                            │
                    │  init(plan, workers)       │  for a given plan, iterate
                    │  next() → worker           │  candidate hosts matching
                    │  fini()                    │  select expression + availability
                    └────────────┬───────────────┘
                                 │ worker
                                 ▼
                    ┌──────────────────────────┐
                    │  try_alloc(worker, plan)  │  Req 4 (parjob) or
                    │                            │  existing seqjob logic
                    │  check slots, shared res,  │
                    │  commit or advance         │
                    └────────────────────────────┘

Composition Algorithm (All Three Types)

// === PRIMARY leaf: iterator composition ===
// Returns >0 on success, 0 on failure
int allocate_primary(sched, primary_resreq, ctx) {
    plan_iter = plan_iter_init(primary_resreq, sched);
    while ((plan = plan_iter->next(plan_iter)) != NULL) {
        host_iter = host_iter_init(plan, sched->workers);
        while ((worker = host_iter->next(host_iter)) != NULL) {
            if (try_alloc_on_host(worker, plan, ctx) > 0) {
                host_iter->fini(host_iter);
                plan_iter->fini(plan_iter);
                return ALLOC_SUCCESS;
            }
        }
        host_iter->fini(host_iter);
    }
    plan_iter->fini(plan_iter);
    return ALLOC_FAIL;
}

// === COMPOUND: AND of primaries ===
// ALL primaries must allocate. On any failure, roll back prior allocations
// and return failure. Same logic as today's sched_make_compound_allocation().
int allocate_compound(sched, compound_resreq, ctx) {
    for (int i = 0; i < compound_resreq->primaries.count; i++) {
        primary = compound_resreq->primaries[i];
        if (allocate_primary(sched, primary, ctx) <= 0) {
            // Roll back all primaries allocated so far in this compound
            rollback_compound_allocations(ctx, i);
            return ALLOC_FAIL;
        }
    }
    return ALLOC_SUCCESS;
}

// === ALTER: OR of compounds ===
// First compound that fully allocates wins. Same logic as today's
// sched_make_alter_allocation(). On failure, no rollback needed
// (prior compounds were never committed — they failed their own checks).
int allocate_alter(sched, alter_resreq, ctx) {
    for (int i = 0; i < alter_resreq->compounds.count; i++) {
        compound = alter_resreq->compounds[i];
        if (allocate_compound(sched, compound, ctx) > 0) {
            return ALLOC_SUCCESS;
        }
        // This compound failed — advance to next alternative
    }
    return ALLOC_FAIL;
}

// === Top-level dispatch ===
// Unchanged from today's sched_make_allocation() dispatch on obj->tag.
int allocate_job(sched, job, ctx) {
    switch (job->resreq_type) {
    case PRIMARY:
        return allocate_primary(sched, job->primary_resreq, ctx);
    case COMPOUND:
        return allocate_compound(sched, job->compound_resreq, ctx);
    case ALTER:
        return allocate_alter(sched, job->alter_resreq, ctx);
    }
}

COMPOUND rollback note: If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back — the compound is atomic. Today this is handled by sched_make_compound_allocation() tracking allocations in ctx and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract.

Who Uses This Composition

Allocator	maxTasks	Plan iterator	Host iterator	Behavior
seqjob_allocator	== 1	Trivial: single plan (1 host × 1 slot)	Full: filters hosts by select expression	Unchanged from today — first matching host wins
parjob_allocator	> 1	Full combinatorial: 1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1	Full: filters hosts with enough available slots	New — walks plans until one succeeds
Future allocators	any	Custom strategy (e.g., topology-aware, bin-packing)	Reuses same xce_host_iter_t	Pluggable via job_alloc_fn

Key Design Properties

• Lazy: plan_iter->next() generates one plan at a time. host_iter->next() returns one host at a time. No full materialization.
• Cancel-safe: both next() calls have pthread_cancel cancellation points.
• Independent lifecycles: a host_iter is created from a plan, used, and destroyed; the plan_iter is advanced to the next plan only after the host_iter is done. fini() on one does not affect the other.
• COMPOUND/ALTER unchanged: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside allocate_primary().
• seqjob rewrap: seqjob_allocator is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host × 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case.

Requirements

Requirement 1: Pluggable job picker interface

Objective: As a maintainer of the scheduler, I want the core scheduling loop to call a single next_job() plugin interface with no knowledge of how jobs are selected, so that picking strategies (FCFS, priority, fair-share, deadline, etc.) are truly pluggable and the core loop never changes regardless of the picking algorithm.

Acceptance Criteria

1. The Scheduler core loop shall call picker->next_job(picker) to get the next schedulable job (or NULL if none ready), and shall not contain any logic about project iteration, queue iteration, or job ordering — those concerns are entirely encapsulated inside the picker plugin.

2. The job picker plugin interface (xce_job_picker_iface_t) shall define these operations, organized by category:

   Lifecycle hooks (MUST):

   Operation	Signature	Semantics
   init	int (*init)(void* picker, xce_sched_core_t* core)	One-time setup, receives core reference for state queries
   fini	void (*fini)(void* picker)	Teardown and cleanup of all picker-owned resources

   State-change hooks (MUST — called by core when events arrive from driver):

   Operation	Signature	Semantics	Triggered by
   on_queue_update	int (*on_queue_update)(void* picker, xce_sched_queue_t* queue)	Add/update queue in picker's internal ordering	QUEUE_UPDATE event
   on_job_push	int (*on_job_push)(void* picker, xce_sched_job_t* job)	Add new pending job to picker's candidate set	JOBS_PUSH event
   on_job_state_change	int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t new_state)	Remove job from candidates if no longer pending; re-add if returning to pending	JOB_STATE_CHANGE event
   on_job_update	int (*on_job_update)(void* picker, xce_sched_job_t* job)	Reorder job in candidates if priority/queue changed	JOB_UPDATE event

   Session hooks (MUST — called at round boundaries):

   Operation	Signature	Semantics
   begin_session	int (*begin_session)(void* picker)	Reset iteration cursor; called at start of each scheduling round
   next_job	xce_sched_job_t* (*next_job)(void* picker)	Return next schedulable job per picker's ordering, or NULL when exhausted
   end_session	int (*end_session)(void* picker)	End-of-round notification; called before core sleeps between rounds

   Post-allocation hooks (MUST — called after allocation attempt):

   Operation	Signature	Semantics
   on_dispatch	int (*on_dispatch)(void* picker, uint64_t jobid)	Job was allocated; remove from candidates permanently
   on_pend	int (*on_pend)(void* picker, uint64_t jobid)	Job was pended; keep for next round but may deprioritize

   Optional hooks (MAY — NULL if not needed):

   Operation	Signature	Semantics	When needed
   on_config	int (*on_config)(void* picker, xce_conf4sched_t* config)	Rebuild picker strategy from config	Config-driven pickers (deadline, fair-share weights)
   on_job_rusage	int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages)	Track per-job resource usage	Resource-aware pickers (fair-share, backfill)

   Hooks not defined on the picker interface (worker updates, resource updates, resval updates) are not forwarded to the picker — picking is about job ordering, not resource availability.

3. The core shall call state-change hooks synchronously during core_push_event() processing, before the event returns. Session hooks and post-allocation hooks are called synchronously during the scheduling session (between BEGIN_ROUND and session exhaustion). The picker runs in the core's scheduling thread — no dedicated picker thread.

4. The core scheduling loop shall:

   begin_session();
   while ((job = picker->next_job(picker)) != NULL) {
       if (try_allocate(job) > 0) {
           picker->on_dispatch(picker, job->jobid);
           emit_decision(DISPATCH, job);
       } else if (can_preempt(job)) {
           emit_decision(PREEMPT, job, victims);
           picker->on_pend(picker, job->jobid);  // retry next round
       } else {
           emit_decision(PEND, job);
           picker->on_pend(picker, job->jobid);
       }
   }
   end_session();
   

   The core loop shall contain zero knowledge of project/queue/job ordering — all selection logic lives behind next_job().

5. The picker plugin shall be cancel-safe: next_job() shall have defined pthread_cancel cancellation points. The picker shall be session-aware: begin_session() resets iteration state, end_session() is called before sleep-between-rounds.

6. The hardcoded "xce_sched_fcfs" literal at scheduler.c:43 (in sched_job_picker()) shall be replaced by loading the configured sched->job_pick_plugin via the existing plugin_register/key_find mechanism. If the configured plugin name is not registered, the Scheduler shall fail with a clear configuration error (same pattern as Req 5 AC 5).

7. sched_job_picker()'s dedicated thread and jbuf cbuf communication shall be retired. The picker runs synchronously within the core's scheduling thread — next_job() is called directly, not via a separate thread + cbuf. This eliminates one thread and simplifies the session lifecycle for simulation mode.

Requirement 2: Lazy allocation plan iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front.

Acceptance Criteria

1. The Scheduler shall define an xce_alloc_plan_iter_t interface with init(resreq, sched), next() → plan, and fini() operations.
2. Each call to next() shall generate exactly one candidate (nhosts, slots_per_host, host_filter) plan without generating or materializing any other plan in the same call.
3. The Scheduler shall order plans returned by successive next() calls from most-preferred to least-preferred, per the owning allocator's preference policy.
4. next() shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion.
5. Calling fini() shall release all resources held by the iterator, and the interface shall define no other required cleanup step.

Requirement 3: Lazy host candidate iterator

Objective: As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering.

Acceptance Criteria

1. The Scheduler shall define an xce_host_iter_t interface with init(plan, workers), next() → worker, and fini() operations.
2. Each call to next() shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.
3. next() shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call next() again after receiving NULL for that plan.
4. The host iterator shall compose with the plan iterator: a new xce_host_iter_t shall be constructible from any plan produced by Requirement 2's xce_alloc_plan_iter_t.next().
5. Calling fini() shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.

Requirement 4: Parallel job combinatorial allocation

Objective: As a cluster operator submitting a job that needs multiple task slots, I want parjob_allocator to actually allocate across hosts, so that jobs with maxTasks > 1 are no longer silently unallocatable.

Acceptance Criteria

1. When a job's xce_primary_resreq_t.maxTasks > 1 reaches parjob_allocator's job_alloc_fn (my_alloc() in parjob_allocator.c), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host × N slots, 2 hosts × ⌈N/2⌉ slots, …, N hosts × 1 slot, using Requirement 2's plan iterator.
2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan.
3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job.
4. When no plan among those generated in Criterion 1 can be satisfied, my_alloc() shall return the same not-satisfied signal (0) that the current stub returns today, so pend-reason handling in sched_make_primary_alloc_() is unaffected.
5. parjob_allocator's job_alloc_fn shall no longer unconditionally return 0 regardless of ctx contents — the current stub behavior — for any job with maxTasks > 1.

Requirement 5: Configurable picker plugin

Objective: As a cluster operator, I want to configure which job-picking policy the scheduler uses without recompiling, and override it per-queue, so that FCFS, priority, or custom pickers can be swapped operationally.

Acceptance Criteria

1. The Scheduler shall add a job_pick_plugin field to xce_scheduler_t, defaulting to "builtin_fcfs" when unset — this is the global picker plugin name.

2. The Scheduler shall add a job_pick_plugin field to xce_sched_queue_t, defaulting to NULL (inherit from global). When non-NULL, the configured global picker plugin shall use this per-queue override for jobs within that queue. How the picker plugin uses per-queue overrides is an implementation detail of the plugin, not the core loop.

3. The Scheduler shall add a project_name field to xce_sched_queue_t, set from queue configuration, establishing the queue→project relationship. The picker plugin may use this field internally for project-scoped iteration.

4. The Scheduler shall accept the picker plugin name from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild.

5. When the configured plugin name is not registered via the existing plugin_register/key_find mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default.

6. The picker plugin interface contract (xce_job_picker_iface_t) shall be documented: each picker must be cancel-safe (responds to pthread_cancel at defined points in next_job()) and session-aware (resets its iteration state on begin_session()).

Requirement 6: Preemption policy and execution

Objective: As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention.

Acceptance Criteria

1. The Scheduler shall add a XCE_DECISION_PREEMPT opcode to xce_decision_op_t in decision.h, with a payload type xce_preempt_decision_t carrying victim_jobids[], n_victims, signal (SIGSTP in v1), and preempting_jobid.
2. When allocation fails for a job J whose queue has a non-NULL preemptible_queues list, the Scheduler shall invoke the queue's preempt_policy plugin to select victim jobs from the target queues.
3. The default preemption policy "lowest_priority_first" shall: collect running jobs from target queues, filter to those with priority < J.priority and queue->preemptible == true, rank by priority ascending, and select the minimum subset whose freed resources satisfy J's resource request.
4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via sched_commit_decision(XCE_DECISION_PREEMPT), publish it through the controller→worker pipeline, and pend job J for the next scheduling round (async preemption — the job is picked up after resources are freed).
5. When no sufficient victim set exists, the Scheduler shall pend job J with a resource-shortage reason without invoking preemption.
6. The worker receiving a preemption action shall send the specified signal (SIGSTP) to the victim job's process groups, and the controller shall transition the victim job's state to JOB_SUSPENDED.
7. The Scheduler shall add preemptible_queues (list of queue names this queue CAN preempt from), preemptible (bool — CAN this queue's jobs BE preempted by others), and preempt_policy (victim-selection plugin name, NULL = default) fields to xce_sched_queue_t.
8. The existing HOOK_DETAIL_JOB_STOP hook detail enum shall be reused for preemption stop notifications.
9. Preemption victim selection shall never select a job whose priority is greater than or equal to the preempting job's priority, nor a job whose queue has preemptible == false.

Requirement 7: Session-level unsatisfiable resreq cache

Objective: As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.

Acceptance Criteria

1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on xce_scheduler_t, keyed by a fingerprint of {select_string, rusage_string, user, queue, project}.
2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (begin_session).
3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation AND preemption (Requirement 6) also fails or is inapplicable, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job.
4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption.
5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.

Architectural Support for Future Object Sharing (Deferred)

Full object sharing (resreq template registry, pend reason dedup pool) is deferred to a follow-up spec. However, the architecture designed in this spec must not block sharing from being added later. The following design invariants apply to all resreq, pend-reason, and related objects touched by this spec:

1. Resreq immutability: xce_primary_resreq_t, xce_compound_resreq_t, and xce_alter_resreq_t objects shall be treated as immutable once created and validated. No code path shall mutate a resreq in place. Any modification (e.g., expanding a job-array template into per-element resreqs) shall create a new copy. This ensures that sharing via const pointer is safe to introduce later without auditing every consumer for mutation.

2. Pend reason factory: All pend reason allocation shall go through make_pendreason() (already the single factory in lib/preason.c). Callers never calloc(sizeof(xce_pendreason_t)) directly. This ensures a session-level dedup pool can be added inside the factory later without changing any caller.

3. Pointer-identity fast path: Code that compares two resreqs or two pend reasons shall use pointer equality (a == b) as a fast path before falling back to deep comparison. When a resreq registry is introduced, identical resreqs will have the same pointer — the fast path avoids repeated deep comparison for no runtime cost today.

4. Select expression memoization: The existing select_eval_ctx_t per-session memoization of parsed select-expression ASTs shall be preserved and extended to the core's internal resreq objects. The parsed/evaluated form of a select expression shall be attached to the resreq on first evaluation and reused for subsequent evaluations (same session, same resreq pointer).

5. Snapshot forward-compatibility: The snapshot format shall not preclude object sharing. Shared objects in the snapshot shall be serializable by identity (an object table with IDs) rather than requiring inline embedding at every reference site. This is a design-phase concern but constrains the JSON schema choice.

These invariants impose zero overhead in the current implementation — immutability is already the pattern, pointer-identity fast paths are already valid C, and the factory already exists. They simply rule out design choices that would make sharing expensive or impossible to retrofit.

Requirement 8: Builtin default picker

Objective: As a maintainer, I want the new architecture to ship with a builtin picker plugin that replicates today's exact scheduling behavior, so that operators who don't configure a custom picker see no behavioral change.

Acceptance Criteria

1. Registration: builtin_fcfs shall be registered automatically at scheduler startup as "builtin_fcfs" and set as the default sched->job_pick_plugin.

2. Hook implementation — builtin_fcfs shall implement every MUST and SHOULD hook from xce_job_picker_iface_t (Req 1 AC 2) as follows:

   Hook	FCFS Implementation
   init	Allocate fcfs_plugin_data_t. Initialize ordq_t queues (ordered by queue->priority desc). Set iteration cursor to NULL.
   fini	Free all per-queue ordq_t pendJobs, the ordq_t queues, and plugin data.
   on_queue_update	Upsert queue into ordq_t queues. If queue exists, update its priority and reorder. If new, allocate fcfs_queue_t with empty ordq_t pendJobs.
   on_job_push	If job->state & JOB_STAT_PEND: find the job's queue in ordq_t queues, add fcfs_job_t{job} to that queue's ordq_t pendJobs (ordered by priority desc, queue_time asc).
   on_job_state_change	If new_state is not pending (EXIT/FAIL/DONE): remove job from its queue's pendJobs by jobid. If new_state returns to PEND: re-add to pendJobs.
   on_job_update	If job's priority or queue field changed: remove from old position in pendJobs, re-insert at correct position per ordering.
   begin_session	Reset iteration cursor to head of ordq_t queues.
   next_job	Walk ordq_t queues from cursor. For each queue, skip if sched_queue_is_ready() returns false. Within queue, walk ordq_t pendJobs. Return first job where sched_job_is_ready() returns true. Advance cursor. Return NULL when all queues and jobs exhausted.
   end_session	No-op for FCFS (iteration state reset in begin_session).
   on_dispatch	Remove job from its queue's ordq_t pendJobs by jobid.
   on_pend	No-op for FCFS — job remains in pendJobs, will be retried next round in same position.
   on_config	NULL — FCFS is config-agnostic.
   on_job_rusage	NULL — FCFS is not resource-aware.

3. Internal data structures — builtin_fcfs shall maintain its own private state, not query the core's sched->queues/sched->jobs during next_job(). All candidate management happens in the state-change hooks. This ensures:

   • next_job() is fast — no core lock acquisition, no hash table scans
   • The picker is independently testable — feed hooks, verify next_job() order
   • Simulation snapshots capture picker state naturally via the hooks already recorded in the event stream

4. Ordering equivalence — builtin_fcfs shall produce the same job-picking order as today's fcfs_elect_job: inter-queue by descending queue->priority, intra-queue by priority desc, queue_time asc. Behavior with per-queue picker overrides is as specified in Req 5 AC 2.

5. No dedicated thread — builtin_fcfs shall implement next_job() as a synchronous call, not a dedicated thread + jbuf cbuf. The picker runs in the core's scheduling thread. pthread_cancel cancellation points shall exist in the next_job() iteration loop.

6. Session state — end_session() is a no-op. Iteration state is reset in begin_session(). Undispatched jobs remain in pendJobs across sessions (they carry over to the next round).

Requirement 9: No regression to existing allocators, job picker, or HA soak

Objective: As a maintainer, I want the iterator-driven rework and picker architecture scoped so that existing single-slot allocation and FCFS behavior are unaffected, so that this framework change is safe to ship independently of any new policy or allocator.

Acceptance Criteria

1. seqjob_allocator's externally observable allocation behavior (which host a maxTasks == 1 job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the xce_host_iter_t interface.
2. builtin_fcfs shall produce the same job-picking order as today's fcfs_elect_job, given the same queue and job state — same inter-queue priority ordering and same intra-queue (priority desc, queue_time asc) ordering.
3. The COMPOUND and ALTER resreq allocation paths (sched_make_compound_allocation, sched_make_alter_allocation in scheduler.c) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today.
4. Existing scheduler unit tests (scheduler meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string, dedicated picker thread, jbuf cbuf, or eager-one-shot behavior this feature replaces.
5. A full HA soak (sequential and chaotic /xce-test-ha regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.

Requirement 10: Core/Driver Separation with Dual-Mode Operation

Objective: As a scheduler developer, I want the scheduling core separated from all external I/O so that the same core logic runs deterministically in production (driven by a live controller) and in simulation (driven by recorded cluster events with a discrete event-driven virtual clock), enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster.

Acceptance Criteria

Core Interface Contract

1. Mode-agnostic core: The Scheduler core shall have no concept of which mode it runs in. It processes events identically whether the driver is in normal mode (controller-connected, wall-clock time) or simulation mode (replay file, virtual clock). Specifically:

   • There shall be no mode flag (bool simulation_mode or equivalent) anywhere in the core
   • There shall be no if (simulation) { ... } else { ... } code path in the core
   • xce_sched_core_init() shall take identical parameters regardless of mode (no mode-specific init arguments)
   • The core's decision callback shall be the same function pointer type in both modes — the driver decides what to do with decisions (publish to controller vs. log to file)
   • The core's save_snapshot() and load_snapshot() shall be identical in both modes — they serialize/deserialize core state, not driver state

2. Passive event-driven interface: The core shall expose a pure event-driven interface — it receives typed events from the driver, updates internal scheduling state, and emits decisions through a driver-registered callback. The core shall have zero direct dependency on: controller communication, capnp serialization, network I/O, filesystem access, or wall-clock time sources.

3. Virtualized clock access: The core shall access time exclusively through xce_core_now(). The core shall never call time(NULL), clock_gettime(), gettimeofday(), or any other wall-clock time function directly.

   Clock virtualization mechanism — two-base + speed formula:

   The core stores three fields (atomically updated by the driver as a group):

   Field	Type	Description
   virtual_clock_base	_Atomic uint64_t	Virtual-time anchor (nanoseconds)
   real_clock_base	_Atomic uint64_t	Wall-clock anchor captured when virtual_clock_base was set (nanoseconds, CLOCK_MONOTONIC)
   speed_num / speed_den	_Atomic uint64_t	Speed multiplier as rational fraction

   xce_core_now(core) computes:

   uint64_t wall_ns = clock_gettime(CLOCK_MONOTONIC);  // only wall-clock call in the core
   uint64_t elapsed = wall_ns - core->real_clock_base;
   return core->virtual_clock_base + (elapsed * core->speed_num) / core->speed_den;
   

   Mode	virtual_clock_base	real_clock_base	speed	Behavior
   Normal	0	0	1/1	xce_core_now() = wall clock (degraded: 0 + (wall - 0) × 1)
   Simulation discrete	event ts	wall at load	0/1	xce_core_now() = frozen at event ts (time does not advance between events)
   Simulation real-time	snapshot ts	wall at load	1/1	xce_core_now() tracks wall clock from virtual anchor
   Simulation fast/slow	snapshot ts	wall at load	N/1 or 1/N	xce_core_now() runs N× faster or slower than wall clock

   The driver atomically updates all three fields before each event dispatch. In simulation discrete mode (speed=0/1), the clock jumps directly from one event ts to the next — no continuous advancement. In normal mode (bases=0, speed=1/1), the formula degrades to returning wall-clock time directly, with zero overhead beyond the single clock_gettime() call.

   Multi-thread safety: xce_core_now() reads atomic fields lock-free. Both picker and allocator threads independently compute the same virtual time from the shared bases — no mutex needed. The driver updates all three fields as an atomic group (via a single mutex or sequenced atomic stores with a version counter).

   • All internal time uses go through xce_core_now(): scheduling logic (job queue-time ordering, session boundaries, pend-reason timestamps, job aging), logging (log message timestamps via tlog format function), error messages, debug traces — any code path that needs the current time calls xce_core_now()
   • Job sub_time and queue_time are set by the driver (from xce_core_now() in normal mode, from event ts in simulation mode) before pushing the JOBS_PUSH event — the core never generates timestamps for externally-visible fields
   • This virtualization is what enables deterministic replay: the same event file produces the same xce_core_now() return values (and therefore same log timestamps, same scheduling decisions) regardless of wall-clock speed
   • Third-party library constraint: any library linked into the scheduler core that needs the current time must obtain it through xce_core_now() or a core-provided wrapper — never by calling time(), clock_gettime(), gettimeofday(), or similar directly. Specifically:
     • Logging (tlog): The tlog library internally captures struct tlog_time from wall-clock time before calling the registered format callback (my_tlog_format). The core shall register its own format function via tlog_reg_format_func(fn, core_ptr) where core_ptr is the xce_sched_core_t*. This format function shall ignore info->time entirely and instead read xce_core_now(core_ptr), converting nanoseconds to struct tm fields via gmtime_r() (deterministic — pure computation from an integer, no system clock access). The userptr parameter of tlog_reg_format_func() carries the core pointer into the format callback.
     • libxce subset: The core shall link only against the minimal data-structure subset of libxce (ordq, dataq, cbuf, xlist, zarray, kvmgr, rtti) — not against netio, delay, xceapi, or idgen which call wall-clock time internally
     • pthread timed-blocking: The core is multi-threaded (picker + allocator threads). In simulation mode, thread interleaving is made deterministic per AC 6 — pthread_cond_timedwait() and pthread_mutex_timedlock() are not needed in the deterministic execution path
     • If a library cannot avoid calling wall-clock time directly (e.g., a third-party compression or serialization library that internally timestamps), it must not be linked into the core; such concerns stay in the driver layer

4. Core event types: The core interface shall accept at minimum these input events:

   Event Type	Payload	Description
   CONFIG_PUSH	xce_conf4sched_t*	Full configuration push (startup or config change)
   JOBS_PUSH	int njobs, xce_job_t**	Newly submitted jobs
   WORKERS_PUSH	int nworkers, xce_worker_t**	Worker list update
   RESOURCES_PUSH	int n, xce_resource_t**	Resource definitions
   RESVALS_FULL	int n, xce_resval_t**	Full resource value snapshot
   RESVALS_DELTA	xce_resval_list_t*	Incremental resource value update
   JOB_STATE_CHANGE	uint64_t jobid, uint32_t state	Job started / exited / failed
   JOB_UPDATE	xce_job_t*	In-place job field update (notify path)
   QUEUE_UPDATE	xce_queue_t*	Queue definition update
   JOB_RUSAGE	xce_job_rusages_t*	Job resource usage update
   BEGIN_ROUND	uint64_t timestamp_ns	Trigger scheduling round at given logical time

5. Decision output callback: The core shall emit all scheduling decisions (DISPATCH, PEND, PREEMPT) through a xce_core_decision_cb_t callback function pointer registered by the driver at core init time, replacing the current internal decisions cbuf + main.c poll loop pattern. The callback signature shall be int (*on_decision)(void* driver_ctx, xce_decision_t* decision).

6. Deterministic guarantee: Given identical initial state and identical event sequences with identical timestamps, the core shall produce identical decision sequences. The core is multi-threaded (picker thread + allocator thread, matching today's architecture). To achieve deterministic replay:

   • All hash table / key-value store iteration that affects scheduling output order shall be stabilized (sorted by key before iterating)
   • Any required randomness (e.g., tie-breaking) shall use a PRNG seeded by the driver at init time
   • In simulation mode, thread interleaving shall either be made deterministic (e.g., the simulation driver serializes event dispatch and decision collection through a single-threaded core path), or the core's threads shall synchronize on a simulation barrier that advances in lock-step with the virtual clock

7. Driver event queue: The driver shall retain an internal pending-event queue (the existing xce_sched_update_request_t / pend_updates mechanism, or equivalent) to buffer events received from the controller between scheduling rounds. Events accumulate in the driver between BEGIN_ROUND invocations. When the core receives BEGIN_ROUND, it drains all buffered events, processes a scheduling session, and emits decisions.

Normal-Mode Driver

8. Controller integration: The normal-mode driver shall receive capnp push/notify messages from the controller (via the existing scheduler_req_handler callback), translate each to the corresponding core event type, and invoke the core's event push interface. The driver shall publish core-emitted decisions back to the controller via the existing xcecli_ctx_t API (sched_publish_dispatch / sched_publish_pend_reason semantics, generalized to include PREEMPT).

9. Runtime recording control via xceadmin: The scheduler shall support two new xceadmin commands:

   • scheduler record start [FILE] — take a full state snapshot, write it to the replay file, then begin recording all subsequent core input events and output decisions to the file
   • scheduler record stop — stop recording, close the replay file, resume normal operation

   Recording shall have zero impact on scheduling correctness — the core processes identically whether recording is active or not.

10. Periodic snapshot during recording: While recording is active, the driver shall periodically append fresh snapshots to the replay file (default: every N events, configurable via --record-snapshot-interval N, where N=0 disables periodic snapshots and records only the initial snapshot). This bounds replay time for any desired start point.

Simulation-Mode Driver

11. Simulation mode startup: The scheduler shall support a simulation mode activated by --sim FILE (replay file path) and --sim-start-ts TIMESTAMP (logical time to begin replay, default: start of file). In simulation mode, the driver loads the most recent snapshot with ts <= --sim-start-ts, then replays all delta events from that point forward.

12. Discrete event-driven virtual clock: The simulation driver shall implement a virtual clock that advances by jumping directly to the timestamp of the next queued event — no fixed tick interval, no wall-clock waiting between events. The clock shall be strictly monotonically increasing. When multiple events share the same timestamp, they shall be delivered in file order. The driver shall support:

    • --sim-speed N — rate limit: N=0 means "as fast as possible" (no delay), N=1 means real-time, N>1 means N× real-time
    • --sim-pause-at TIMESTAMP — pause simulation at a specific logical time for interactive debugging

13. Deterministic replay verification: In simulation mode with --sim-verify, the driver shall compare each core-emitted decision against the recorded decisions in the replay file. Any mismatch (missing, extra, or different decision) shall be reported as a regression with exact logical-time and decision-detail context.

14. What-if branching: The simulation driver shall accept --sim-alt-events FILE (alternative delta event file). When both a base replay file (for snapshot loading) and an alt-events file are provided, the driver loads the snapshot from the base file, then replays delta events from the alt-events file — enabling "what-if" analysis (e.g., "what if job X had been submitted with higher priority?").

Replay File Format

15. JSONL format: The replay file shall be newline-delimited JSON (JSONL), one event per line:

    {"type":"HEADER","version":"x.y.z","config_hash":"sha256...","start_ts":1234567890000000000,"start_wall":"2026-07-27T12:00:00Z"}
    {"type":"SNAPSHOT","ts":1234567890000000000,"data":{...full core state...}}
    {"type":"CONFIG_PUSH","ts":1234568000000000000,"payload":{...}}
    {"type":"JOBS_PUSH","ts":1234568100000000000,"payload":{"njobs":2,"jobs":[...]}}
    {"type":"BEGIN_ROUND","ts":1234568200000000000,"payload":{}}
    {"type":"DECISION","ts":1234568200000000000,"payload":{"op":"DISPATCH","jobid":"..."}}
    

    • type: event type name string
    • ts: nanosecond logical timestamp (monotonic, unsigned 64-bit integer)
    • payload: event-type-specific object
    • HEADER line: first line of every file, carries scheduler version and configuration hash for compatibility checking
    • SNAPSHOT lines: embed full core state in data (plain JSON, not base64-encoded — debuggable and directly extractable)
    • DECISION lines: recorded decisions (output), present only in files produced by the normal-mode driver's recording feature — the simulation driver ignores them unless --sim-verify is active
    • The event payload JSON schema is an open design question deferred to the design phase

State Snapshot

16. Snapshot serialization: The core shall support save_snapshot() producing a full state dump (plain JSON) containing all config, queues (with per-queue picker/preemption settings), jobs (with state, priority, queue-time, resreq), workers (with available/used resources), resources, resvals, users, projects, shared-resource tracking state, unsatisfiable-resreq cache, and session state. The core shall support load_snapshot() restoring to that exact state, producing a core that is indistinguishable from one that reached that state by replaying all events up to that timestamp.

Interaction with Existing Requirements

Reqs 1–9 describe logic inside the core — the picker loop, iterators, parjob allocator, preemption policy, resreq cache, and builtin plugins all operate within the core's event-driven boundary. Requirement 10 defines the boundary itself. Specifically:

• Req 5 (picker slots from config): config arrives via CONFIG_PUSH event, not via capnp decode inside the core
• Req 6 (preemption → controller): XCE_DECISION_PREEMPT is emitted through the decision callback, not sent directly to controller
• Req 7 (resreq cache, session lifecycle): BEGIN_ROUND replaces the implicit sleep(1)-delimited session boundary in main.c; the cache still lives on xce_sched_core_t and is cleared at session start
• Req 9 (HA soak regression): the same core code can be validated in simulation mode against recorded HA failover traces, catching regressions without a live cluster

Data Structure Changes Summary

xce_scheduler_t additions

char* job_pick_plugin;         // default: "builtin_fcfs" (Req 5)
// unsatisfiable resreq cache (Req 7)

xce_sched_queue_t additions

char*    project_name;          // owning project (picker uses internally)
char*    job_pick_plugin;       // per-queue picker override (NULL = use global)
char**   preemptible_queues;    // queues this queue CAN preempt from
int      n_preemptible_queues;
bool     preemptible;           // CAN this queue's jobs BE preempted?
char*    preempt_policy;        // victim selection plugin (NULL = default)

xce_sched_project_t additions

int32_t  priority;              // inter-project ordering (future spec)
// future: resource_quota, resource_used, parent_project (project-resource-ownership spec)

xce_job_picker_iface_t (new, Req 1)

typedef struct {
    int  (*init)(void* picker, xce_sched_core_t* core);
    void (*fini)(void* picker);
    int  (*begin_session)(void* picker);
    xce_sched_job_t* (*next_job)(void* picker);
    int  (*end_session)(void* picker);
    int  (*on_dispatch)(void* picker, uint64_t jobid);
    int  (*on_pend)(void* picker, uint64_t jobid);
} xce_job_picker_iface_t;

decision.h additions

typedef enum {
    XCE_DECISION_NONE = 0,
    XCE_DECISION_DISPATCH,
    XCE_DECISION_PEND,
    XCE_DECISION_PREEMPT,     // NEW
    XCE_DECISION_NEW_SESSION
} xce_decision_op_t;

typedef struct {
    uint64_t* victim_jobids;    // jobs to preempt
    int       n_victims;
    int       signal;           // SIGSTP (v1), SIGKILL (future)
    uint64_t  preempting_jobid; // job that triggered preemption
} xce_preempt_decision_t;

Core/Driver Separation (Req 10)

// Core event (driver → core)
typedef enum {
    XCE_CORE_EVENT_CONFIG_PUSH = 0,
    XCE_CORE_EVENT_JOBS_PUSH,
    XCE_CORE_EVENT_WORKERS_PUSH,
    XCE_CORE_EVENT_RESOURCES_PUSH,
    XCE_CORE_EVENT_RESVALS_FULL,
    XCE_CORE_EVENT_RESVALS_DELTA,
    XCE_CORE_EVENT_JOB_STATE_CHANGE,
    XCE_CORE_EVENT_JOB_UPDATE,
    XCE_CORE_EVENT_QUEUE_UPDATE,
    XCE_CORE_EVENT_JOB_RUSAGE,
    XCE_CORE_EVENT_BEGIN_ROUND,
} xce_core_event_type_t;

typedef struct {
    xce_core_event_type_t type;
    uint64_t              ts;       // driver-supplied logical time (ns)
    union {
        xce_conf4sched_t*     config;
        struct { int njobs; xce_job_t** jobs; } job_push;
        struct { int nworkers; xce_worker_t** workers; } worker_push;
        struct { int n; xce_resource_t** resources; } resource_push;
        struct { int n; xce_resval_t** vals; } resval_full;
        xce_resval_list_t*    resval_delta;
        struct { uint64_t jobid; uint32_t state; } job_state;
        xce_job_t*            job_update;
        xce_queue_t*          queue_update;
        xce_job_rusages_t*    job_rusage;
        // BEGIN_ROUND has no payload beyond ts
    } payload;
} xce_core_event_t;

// Decision callback (core → driver)
typedef int (*xce_core_decision_cb_t)(void* driver_ctx, xce_decision_t* decision);

// Core struct (pure scheduling state, no I/O)
typedef struct {
    // Scheduling state (moved from xce_scheduler_t)
    xce_conf4sched_t* config;
    uint64_t          configver;
    var               plugins;
    void*             seqjob_allocator;
    void*             parjob_allocator;
    void*             affinity_allocator;
    void*             gpu_allocator;
    var               resmgr, ugmgr, qgmgr, pgmgr, hgmgr;
    zarray_t          sharedInst, sharedInUse;
    var               jobs, queues, workers, users, projects;
    uint64_t          nextid;

    // New fields for core/driver boundary
    void*                  driver_ctx;     // opaque driver context for callback
    xce_core_decision_cb_t on_decision;    // decision output callback
    // Clock virtualization (Req 10 AC 3) — atomically updated by driver
    _Atomic uint64_t       virtual_clock_base;  // virtual-time anchor (ns)
    _Atomic uint64_t       real_clock_base;     // wall-clock anchor (ns, CLOCK_MONOTONIC)
    _Atomic uint64_t       speed_num;           // speed numerator
    _Atomic uint64_t       speed_den;           // speed denominator
    uint64_t               prng_seed;           // seeded by driver at init

    // Picker plugin slot (Req 5)
    char* job_pick_plugin;         // default: "builtin_fcfs"
    // per-queue overrides stored on xce_sched_queue_t.job_pick_plugin,
    // read by the picker internally — not separate core slots

    // Unsatisfiable resreq cache (Req 7)
    // ... (session-level fingerprint cache)

    // Session state
    _Atomic int         session_active;
    _Atomic int         eval_in_progress;
    select_eval_ctx_t*  sel_eval_ctx;
} xce_sched_core_t;

// Normal-mode driver
typedef struct {
    xce_sched_core_t     core;
    xcecli_ctx_t         apictx;          // controller API client
    xce_cluster_config_t cluster_cfg;     // HA candidate list
    dataq_t*             pend_updates[2]; // event buffer between rounds
    // Recording state
    FILE*                record_file;
    bool                 recording;
    uint64_t             record_snapshot_interval;
    uint64_t             events_since_snapshot;
} xce_sched_driver_t;

// Simulation-mode driver
typedef struct {
    xce_sched_core_t  core;
    FILE*             replay_file;
    uint64_t          virtual_clock;     // current logical time (ns)
    uint64_t          sim_speed;         // 0=unlimited, 1=real-time, N=N×
    uint64_t          pause_at_ts;       // 0=don't pause
    bool              verify;            // compare decisions vs recorded
    FILE*             alt_events_file;   // for what-if branching
} xce_sched_sim_driver_t;

// Core interface
int  xce_sched_core_init(xce_sched_core_t* core, xce_core_decision_cb_t cb,
                          void* driver_ctx, uint64_t prng_seed);
int  xce_sched_core_push_event(xce_sched_core_t* core, xce_core_event_t* event);
uint64_t xce_core_now(xce_sched_core_t* core);  // virtualized clock accessor
int  xce_sched_core_save_snapshot(xce_sched_core_t* core, FILE* out);
int  xce_sched_core_load_snapshot(xce_sched_core_t* core, FILE* in);
void xce_sched_core_fini(xce_sched_core_t* core);

Architecture Diagram

Core Scheduling Loop

┌──────────────────────────────────────────────────────────┐
│              SCHEDULER CORE LOOP                           │
│                                                            │
│  begin_session()                                           │
│       │                                                    │
│       ▼                                                    │
│  ┌──────────────────────┐                                  │
│  │ picker->next_job()    │  ←─ plugin decides HOW          │
│  │ returns job or NULL   │    (FCFS, priority, fair-share,  │
│  └──────────┬───────────┘    deadline, ...)                │
│             │                                              │
│       job   │   NULL (exhausted)                           │
│             ▼              ▼                               │
│  ┌──────────────────┐   end_session() → sleep → next round │
│  │ try_allocate(job) │                                      │
│  └────────┬─────────┘                                      │
│           │                                                │
│   success │ fail                                           │
│           ▼        ▼                                       │
│  ┌────────────┐ ┌────────────────┐                        │
│  │ DISPATCH   │ │ can_preempt?   │                         │
│  │ on_dispatch│ │  yes → PREEMPT │                         │
│  └────────────┘ │  no  → PEND    │                         │
│                 │  on_pend       │                         │
│                 └────────────────┘                         │
│                      │                                     │
│                      ▼                                     │
│                 next_job()  (loop continues)               │
└──────────────────────────────────────────────────────────┘

builtin_fcfs Internal Design

The default picker plugin internally implements the equivalent of today's selection logic. This is not part of the core loop — it's the plugin's private implementation:

builtin_fcfs.next_job():
  ┌────────────────────────────────────────────┐
  │ Iterate queues by priority (descending)     │
  │   Skip queues with no ready jobs            │
  │                                             │
  │   For each queue:                           │
  │     If queue has per-queue picker override: │
  │       Delegate to that picker               │
  │     Else:                                   │
  │       Iterate jobs by priority desc,        │
  │       queue_time asc (FCFS)                 │
  │       Skip jobs not ready                   │
  │       Return first ready job                │
  │                                             │
  │ Return NULL if all queues exhausted         │
  └────────────────────────────────────────────┘

Core/Driver Architecture (Req 10)

┌──────────────────────────────────────────────────────────────────┐
│                        DRIVER LAYER                               │
│                                                                   │
│  ┌──────────────────────┐          ┌────────────────────────────┐ │
│  │   NORMAL MODE         │          │   SIMULATION MODE          │ │
│  │                        │          │                            │ │
│  │  controller capnp ────┐│          │  replay.jsonl ──────────┐  │ │
│  │  (scheduler_req_      ││          │  (event file)            │  │ │
│  │   handler)            ││          │                          │  │ │
│  │        │              ││          │        │                 │  │ │
│  │        ▼              ││          │        ▼                 │  │ │
│  │  ┌──────────────┐     ││          │  ┌──────────────┐       │  │ │
│  │  │ event queue   │     ││          │  │ virtual clock│       │  │ │
│  │  │ (pend_updates)│     ││          │  │ (discrete    │       │  │ │
│  │  └──────┬───────┘     ││          │  │  event-driven)│       │  │ │
│  │         │             ││          │  └──────┬───────┘       │  │ │
│  │         │ events      ││          │         │ events        │  │ │
│  │         │ + BEGIN_ROUND││          │         │ + BEGIN_ROUND │  │ │
│  │         ▼             ││          │         ▼               │  │ │
│  │  ┌──────────────┐     ││          │  ┌──────────────┐       │  │ │
│  │  │ publish       │     ││          │  │ record/log    │       │  │ │
│  │  │ decisions to  │     ││          │  │ decisions     │       │  │ │
│  │  │ controller    │     ││          │  │ (--sim-verify)│       │  │ │
│  │  └──────▲───────┘     ││          │  └──────▲───────┘       │  │ │
│  │         │ decisions   ││          │         │ decisions     │  │ │
│  └─────────┼─────────────┘│          └─────────┼───────────────┘  │
│            │              │                    │                   │
│            │   xce_core_decision_cb_t          │                   │
│            │   (callback)                      │                   │
└────────────┼──────────────────────────────────┼───────────────────┘
             │                                  │
             ▼                                  ▼
┌──────────────────────────────────────────────────────────────────┐
│                      SCHEDULER CORE                               │
│                      (xce_sched_core_t)                            │
│                                                                    │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │  xce_sched_core_push_event(core, event)                      │ │
│  │                                                               │ │
│  │  Event types: CONFIG_PUSH, JOBS_PUSH, WORKERS_PUSH,          │ │
│  │  RESOURCES_PUSH, RESVALS_FULL, RESVALS_DELTA,                │ │
│  │  JOB_STATE_CHANGE, JOB_UPDATE, QUEUE_UPDATE,                 │ │
│  │  JOB_RUSAGE, BEGIN_ROUND                                     │ │
│  └──────────────────────────┬──────────────────────────────────┘ │
│                             │                                     │
│                             ▼                                     │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │  Internal Scheduling Logic (Reqs 1–9)                        │ │
│  │                                                               │ │
│  │  picker->next_job() → try_allocate(job)                       │ │
│  │    │                              │                            │ │
│  │    │  xce_alloc_plan_iter_t       │                            │ │
│  │    │  → xce_host_iter_t           │                            │ │
│  │    │  → try alloc                 │                            │ │
│  │    │                              │                            │ │
│  │    │  success → DISPATCH          │                            │ │
│  │    │  fail → PREEMPT or PEND     │                            │ │
│  │    │  → picker->on_dispatch()    │                            │ │
│  │    │  → picker->on_pend()        │                            │ │
│  │    │                              │                            │ │
│  │  Preemption (if alloc fails):                                │ │
│  │  select victims → SIGSTP → pend job → retry next round      │ │
│  │  Resreq fingerprint cache (per-session)                       │ │
│  └──────────────────────────┬──────────────────────────────────┘ │
│                             │                                     │
│                             ▼                                     │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │  on_decision(driver_ctx, decision)                            │ │
│  │  DISPATCH / PEND / PREEMPT                                    │ │
│  └─────────────────────────────────────────────────────────────┘ │
│                                                                    │
│  ZERO dependencies on: controller, capnp, network, wall-clock    │
└──────────────────────────────────────────────────────────────────┘

Simulation Replay Sequence

                   load snapshot    replay delta events
                       │                    │
                       ▼                    ▼
  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
  │ HEADER   │───▶│ SNAPSHOT │───▶│ EVENT @t1 │───▶│ EVENT @t2 │───▶ ...
  └──────────┘    └──────────┘    └──────────┘    └──────────┘
                       │               │               │
                       ▼               ▼               ▼
                  restore core    push + BEGIN_     push + BEGIN_
                  state via       ROUND(t1) →      ROUND(t2) →
                  load_snapshot   emit decisions   emit decisions

  Virtual clock:  t_snap ────────▶ t1 ───────────▶ t2 ───────────▶
                  (discrete jumps, no wall-clock delay, monotonic)

Preemption Flow

scheduler                          controller                  worker
───────────                        ──────────                  ──────
alloc fails for high-prio job J

preempt_policy.select_victims(J):
  collect running jobs from
    Q_aggressor.preemptible_queues
  filter: victim.prio < J.prio
  filter: victim.queue.preemptible
  rank: lowest priority first
  select minimum set satisfying J.resreq

  → [victim1, victim2]

XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
  .victim_jobids = [v1, v2]        deliver to each               job → JOB_SUSPENDED
  .signal = SIGSTP                 victim's worker               release resources
  .preempting_jobid = J

J pended → next scheduling round → allocation retries on freed resources

Future: Project Resource Ownership

Deferred to a separate project-resource-ownership spec. When implemented:

• Projects form a tree with resource quota at each node
• A project-aware picker plugin selects which project gets resources (fair-share, priority, proportional), then iterates queues scoped to that project, then jobs within each queue
• Project quota enforcement: projects over quota are skipped by the picker
• The core scheduling loop (Requirement 1) requires zero code changes — only a new picker plugin replaces builtin_fcfs. The core still calls picker->next_job() with no knowledge of projects.
• queue->project_name (added here) provides the queue→project mapping the v2 picker consumes
• xce_sched_project_t scaffolding (added here) provides the struct the v2 picker extends with quota/usage fields

The architecture is designed so that v1 (flat, builtin_fcfs ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. The only difference is which picker plugin is configured.