diff --git a/.kiro/specs/scheduler-allocation/brief.md b/.kiro/specs/scheduler-allocation/brief.md new file mode 100644 index 00000000..08b01d5d --- /dev/null +++ b/.kiro/specs/scheduler-allocation/brief.md @@ -0,0 +1,210 @@ +# Brief: scheduler-allocation + +## Problem + +The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: `sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: `parjob_allocator.c`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` -- any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()`) produces no allocation at all. Third, allocation is eager and one-shot: the sibling `seqjob_allocator.c`'s `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails. + +The first gap (hardcoded picker) is addressed in the sibling **scheduler-core** spec. This spec, **scheduler-allocation**, closes the other two: the parjob stub and one-shot allocation. + +## Current State + +### What exists (allocation surface) + +- **Seqjob allocator** (`seqjob_allocator.c`): handles `maxTasks == 1`. Checks shared resources, collects workers via `sched_collect_workers()`, tries one host, returns first match. No retry with alternative plans. +- **Parjob allocator stub** (`parjob_allocator.c`): registered as a plugin but `my_alloc()` is empty (`return 0`). Any job with `maxTasks > 1` silently fails allocation. +- **Resource request types** (`resreq.h`): PRIMARY (minTasks/maxTasks/select/rusages/span/same/affinity), COMPOUND (list of primaries), ALTER (list of compounds). The type hierarchy is well-defined but allocation at the PRIMARY leaf lacks iteration abstractions. +- **Select expression evaluator** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC. Already reusable for host filtering. +- **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session. +- **Allocation dispatch** (`scheduler.c:sched_make_allocation()`, `sched_make_primary_alloc_()`): routes to seqjob or parjob allocator based on `maxTasks`. + +### What's missing + +- **Lazy plan generation**: no `xce_alloc_plan_iter_t` -- plans are not generated at all, let alone lazily +- **Lazy host iteration**: no `xce_host_iter_t` -- host collection is eager via `sched_collect_workers()`, materializing the full list before trying any +- **Parallel job allocation**: `parjob_allocator` is a stub. No combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1), no multi-slot host matching +- **Unsatisfiable resreq caching**: a batch of similarly-shaped unsatisfiable jobs each re-pays the full allocation cost within a session + +## Desired Outcome + +### Iterator-Driven Allocation Architecture + +A lazy `xce_alloc_plan_iter_t` (candidate `(nhosts, slots_per_host)` plans, most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session. + +### Three-Layer Iterator Composition + +``` +ALTER -> OR-of-compounds: first compound that fully allocates wins (unchanged) + └─ COMPOUND -> AND-of-primaries: ALL primaries must allocate (unchanged) + └─ PRIMARY -> plan_iter -> host_iter -> try_alloc (NEW iterator layer) +``` + +The iterator composition operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today. + +### Iterator Pipeline + +``` +xce_alloc_plan_iter_t -- init(resreq) -> next() -> plan -> next() -> ... -> NULL -> fini() + │ plan + ▼ +xce_host_iter_t -- init(plan, workers) -> next() -> worker -> ... -> NULL -> fini() + │ worker + ▼ +try_alloc(worker, plan) -- check slots, shared res, commit or advance +``` + +### Key Properties + +- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization. +- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points. +- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. +- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf are unchanged from today. +- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host x 1 slot). This validates the abstraction before applying it to the complex parjob case. + +## Alternatives Considered + +### Approach A: Incremental Enhancement (rejected) + +| Aspect | Detail | +|--------|--------| +| **Plan generation** | Pre-computed plan array. Eager generation for parjob. | +| **Host iteration** | No new abstraction. Simple for-loop over `sched_collect_workers()` result. | +| **Unsatisfiable resreq** | `xlist_t unsatisfiable_resreqs` in session state | + +**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost. + +### Approach B: Iterator-Driven Allocation (chosen) + +| Aspect | Detail | +|--------|--------| +| **Plan iterator** | `xce_alloc_plan_iter_t` — lazy, cancel-safe, generates one plan per `next()` call | +| **Host iterator** | `xce_host_iter_t` — lazy per-plan host filtering via existing select-expression evaluator | +| **Parallel jobs** | Combinatorial plan generation: 1host x N -> 2hosts x ceil(N/2) -> ... -> Nhhosts x 1 | +| **Unsatisfiable resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` | +| **Seqjob path** | Trivial rewrap — single plan (1x1) through the same iterator interfaces | + +**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the resreq type hierarchy naturally. Seqjob rewrap validates the abstraction on the simple case before applying it to the complex parjob case. The cache eliminates redundant work for batches of unsatisfiable jobs. + +### Decision Rationale + +Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The two-iterator composition (plan -> host -> try_alloc) is a proven pattern: the plan iterator answers "how many hosts, how many slots each?", the host iterator answers "which specific hosts can fulfill this plan?", and the allocator commits or advances. This separation of concerns makes each layer independently testable. + +## Approach + +**Iterator-Driven Allocation**: + +### PRIMARY leaf allocation pseudocode + +```c +int allocate_primary(sched, primary_resreq, ctx) { + plan_iter = plan_iter_init(primary_resreq, sched); + while ((plan = plan_iter->next(plan_iter)) != NULL) { + host_iter = host_iter_init(plan, sched->workers); + while ((worker = host_iter->next(host_iter)) != NULL) { + if (try_alloc_on_host(worker, plan, ctx) > 0) { + host_iter->fini(host_iter); + plan_iter->fini(plan_iter); + return ALLOC_SUCCESS; + } + } + host_iter->fini(host_iter); + } + plan_iter->fini(plan_iter); + return ALLOC_FAIL; +} +``` + +### Who uses this composition + +| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior | +|-----------|------------|---------------|---------------|----------| +| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host x 1 slot) | Full: filters hosts by select expression | Unchanged from today — first matching host wins | +| `parjob_allocator` | `> 1` | Full combinatorial: 1hostxN, 2hosts x ceil(N/2), ..., Nhosts x 1 | Full: filters hosts with enough available slots | New — walks plans until one succeeds | +| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` | + +### Unsatisfiable resreq cache + +```c +// Per-session cache on xce_scheduler_t +// Key: fingerprint of {select, rusage, user, queue, project} +// Cleared at begin_session() +// Populated when a job's plan iterator exhausts without success +// Checked before starting allocation for each job — hit = immediate pend +``` + +### What stays unchanged + +- `seqjob_allocator` externally observable behavior — rewrap is internal +- COMPOUND and ALTER resreq allocation paths — dispatch is unchanged +- Select expression system — no changes +- Shared resource tracking — no changes +- Existing plugin framework — no changes + +### What's new + +- `xce_alloc_plan_iter_t` — lazy plan iterator interface +- `xce_host_iter_t` — lazy host iterator interface +- `parjob_allocator` combinatorial plan generation +- Session-level unsatisfiable resreq fingerprint cache +- Seqjob rewrap onto iterator interfaces (trivial case, behavior unchanged) + +## Scope + +### In + +- `xce_alloc_plan_iter_t` interface: `init(resreq, sched)`, `next() -> plan`, `fini()` +- `xce_host_iter_t` interface: `init(plan, workers)`, `next() -> worker`, `fini()` +- `parjob_allocator` combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) via plan iterator +- `seqjob_allocator` rewrap onto iterator interfaces (trivial 1-host/1-slot case, behavior unchanged) +- Session-level unsatisfiable resreq fingerprint cache (keyed on `{select, rusage, user, queue, project}`) +- COMPOUND and ALTER resreq allocation paths unchanged — they compose over the PRIMARY leaf allocator +- Unit tests: plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence +- Integration test: end-to-end multi-slot parallel job allocation + +### Out + +- Pluggable job picker interface (`xce_job_picker_iface_t`) — in **scheduler-core** +- `builtin_fcfs` picker — in **scheduler-core** +- Configurable picker plugin with per-queue override — in **scheduler-core** +- Pickers in general (Req 1, 5, 8 of scheduler-framework) — in **scheduler-core** +- Preemption (Req 6 of scheduler-framework) — in **scheduler-preemption** +- Core/driver separation (Req 10 of scheduler-framework) — in **scheduler-core** +- Queue vs Project section — in **scheduler-core** +- New job selection policies beyond what the picker interface enables — deferred +- Resreq template registry (named/shared resreq objects) — deferred +- Select expression language or evaluator changes +- GPU/topology/affinity-aware allocation — left pluggable via future allocator plugins +- Controller-side scheduling-state replication / commit-barrier wiring + +## Upstream / Downstream + +### Upstream (dependencies) + +- **scheduler-core** — the plan and host iterators consume the core event interface (`xce_core_event_t`) for worker/resource state updates, but are independently unit-testable with constructed state +- Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed +- Existing select expression evaluator (`resreq.h`) — NO changes needed +- Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed +- Existing resreq type hierarchy (PRIMARY/COMPOUND/ALTER) — NO changes needed + +### Downstream (future consumers) + +- **Affinity/GPU allocators** — register as allocator plugins, implement custom plan iterators for topology-aware placement, reuse `xce_host_iter_t` +- **Bin-packing allocator** — custom plan iterator strategy +- **Resreq template registry** — the fingerprint cache's key can incorporate template identity when sharing is added + +## Existing Spec Touchpoints + +### Depends on +- **scheduler-core** — the iterators operate within the core's scheduling session, consuming worker and resource state pushed through the core event interface. The iterators do NOT depend on the core's driver or I/O layer — they are purely computational, operating on in-memory state. + +### Adjacent +- **scheduler-preemption** — preemption fires after allocation failure. The unsatisfiable resreq cache is populated after both allocation and preemption fail (or preemption is inapplicable). The iterator interfaces are not preemption-aware. +- **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume. + +## Constraints + +- **C (gnu17)** — no C++, no external libraries beyond existing (pthreads) +- **Cancel-safe** — `plan_iter->next()` and `host_iter->next()` must have `pthread_cancel` cancellation points +- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session` +- **Backward compatible** — existing seqjob allocation behavior unchanged. COMPOUND/ALTER dispatch unchanged. New functionality is additive. +- **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M x N candidate evaluations (acceptably bounded). +- **No controller dependency** — iterators are purely computational, operating on in-memory worker/resource state. They do not perform I/O or call into the controller. diff --git a/.kiro/specs/scheduler-allocation/requirements.md b/.kiro/specs/scheduler-allocation/requirements.md new file mode 100644 index 00000000..b60505ae --- /dev/null +++ b/.kiro/specs/scheduler-allocation/requirements.md @@ -0,0 +1,619 @@ +# Requirements Document + +## Introduction + +The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. The first gap -- job selection policy is hardcoded (`sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal) -- is addressed in the sibling **scheduler-core** spec. + +This spec closes the remaining two allocation gaps: + +1. **Parallel job allocation is a stub**: `parjob_allocator.c`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` -- any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()` at `scheduler.c:93-103`) produces no allocation at all. + +2. **Allocation is eager and one-shot**: the sibling `seqjob_allocator.c`'s `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops -- there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails. + +This spec introduces an **iterator-driven allocation architecture** with these components: + +- A lazy `xce_alloc_plan_iter_t` -- candidate `(nhosts, slots_per_host)` plans, most-preferred first +- A lazy `xce_host_iter_t` -- candidate hosts for a given plan, filtered by the existing select-expression evaluator +- A `parjob_allocator` that walks the plan iterator to do real combinatorial multi-slot allocation +- A session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session +- A seqjob_allocator rewrap onto the iterator interfaces as a trivial 1-plan/1-slot case, validating the abstraction before applying it to the complex parjob case + +> Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in `.kiro/specs/scheduler-allocation/brief.md`. + +## Boundary Context + +- **In scope**: `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1) for `maxTasks > 1`; `xce_alloc_mode_t` on `xce_sched_ctx_t` for allocation mode (NORMAL/PREEMPT); session-level unsatisfiable-resreq fingerprint cache keyed by `xce_cache_key_t` (O(1) content hash from parsed resreq + identity); unit tests for plan generation, host iteration, cache hit/miss, seqjob rewrap equivalence; integration test for end-to-end multi-slot parallel job allocation. + +- **Out of scope**: Pluggable job picker interface (`xce_job_picker_iface_t`) -- in **scheduler-core**; `builtin_fcfs` picker -- in **scheduler-core**; configurable picker plugin with per-queue override -- in **scheduler-core**; preemption (Req 6 of scheduler-framework) -- in **scheduler-preemption**; core/driver separation (Req 10 of scheduler-framework) -- in **scheduler-core**; Queue vs Project roles -- in **scheduler-core**; new job selection policies beyond the builtin FCFS picker; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins). + +- **Adjacent expectations**: depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking (`xce_scheduler_t.sharedInst`/`sharedInUse`) -- none of these change. Depends on scheduler-core for the core event interface that pushes worker and resource state; the iterators are independently unit-testable with constructed state. `ha-consistency-invariants`' commit barrier is a future consumer of scheduling decisions; this spec does not wire it. + +## Allocation Architecture (Reqs 2-4) + +Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER): + +### Resreq Type Hierarchy + +``` +ALTER -> OR-of-compounds: first compound that fully allocates wins + └─ COMPOUND -> AND-of-primaries: ALL primaries must allocate for success + └─ PRIMARY -> plan_iter -> host_iter -> try_alloc (leaf level) +``` + +The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators -- they call into the same primary allocator, unchanged from today's `sched_make_compound_allocation()` and `sched_make_alter_allocation()` dispatch. + +### Iterator Composition (PRIMARY leaf) + +``` + ┌──────────────────────────┐ + │ xce_alloc_plan_iter_t │ Req 2 + │ │ + │ init(primary_resreq, sched) │ + │ next() -> plan │ generates (nhosts, slots_per_host) + │ fini() │ candidates, most-preferred first + └────────────┬───────────────┘ + │ plan + ▼ + ┌──────────────────────────┐ + │ xce_host_iter_t │ Req 3 + │ │ + │ init(plan, workers) │ for a given plan, iterate + │ next() -> worker │ candidate hosts matching + │ fini() │ select expression + availability + └────────────┬───────────────┘ + │ worker + ▼ + ┌──────────────────────────┐ + │ try_alloc(worker, plan) │ Req 4 (parjob) or + │ │ existing seqjob logic + │ check slots, shared res, │ + │ commit or advance │ + └────────────────────────────┘ +``` + +### Composition Algorithm (All Three Types) + +```c +// === PRIMARY leaf: iterator composition === +// Returns >0 on success, 0 on failure +int allocate_primary(sched, primary_resreq, ctx) { + plan_iter = plan_iter_init(primary_resreq, sched); + while ((plan = plan_iter->next(plan_iter)) != NULL) { + host_iter = host_iter_init(plan, sched->workers); + while ((worker = host_iter->next(host_iter)) != NULL) { + if (try_alloc_on_host(worker, plan, ctx) > 0) { + host_iter->fini(host_iter); + plan_iter->fini(plan_iter); + return ALLOC_SUCCESS; + } + } + host_iter->fini(host_iter); + } + plan_iter->fini(plan_iter); + return ALLOC_FAIL; +} + +// === COMPOUND: AND of primaries === +// ALL primaries must allocate. On any failure, roll back prior allocations +// and return failure. Same logic as today's sched_make_compound_allocation(). +int allocate_compound(sched, compound_resreq, ctx) { + for (int i = 0; i < compound_resreq->primaries.count; i++) { + primary = compound_resreq->primaries[i]; + if (allocate_primary(sched, primary, ctx) <= 0) { + // Roll back all primaries allocated so far in this compound + rollback_compound_allocations(ctx, i); + return ALLOC_FAIL; + } + } + return ALLOC_SUCCESS; +} + +// === ALTER: OR of compounds === +// First compound that fully allocates wins. Same logic as today's +// sched_make_alter_allocation(). On failure, no rollback needed +// (prior compounds were never committed -- they failed their own checks). +int allocate_alter(sched, alter_resreq, ctx) { + for (int i = 0; i < alter_resreq->compounds.count; i++) { + compound = alter_resreq->compounds[i]; + if (allocate_compound(sched, compound, ctx) > 0) { + return ALLOC_SUCCESS; + } + // This compound failed -- advance to next alternative + } + return ALLOC_FAIL; +} + +// === Top-level dispatch === +// Unchanged from today's sched_make_allocation() dispatch on obj->tag. +int allocate_job(sched, job, ctx) { + switch (job->resreq_type) { + case PRIMARY: + return allocate_primary(sched, job->primary_resreq, ctx); + case COMPOUND: + return allocate_compound(sched, job->compound_resreq, ctx); + case ALTER: + return allocate_alter(sched, job->alter_resreq, ctx); + } +} +``` + +**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back -- the compound is atomic. Today this is handled by `sched_make_compound_allocation()` tracking allocations in `ctx` and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract. + +### Who Uses This Composition + +| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior | +|-----------|------------|---------------|---------------|----------| +| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host x 1 slot) | Full: filters hosts by select expression | Unchanged from today -- first matching host wins | +| `parjob_allocator` | `> 1` | Full combinatorial: 1host x N, 2hosts x ceil(N/2), ..., Nhosts x 1 | Full: filters hosts with enough available slots | New -- walks plans until one succeeds | +| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` | + +### Key Design Properties + +- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization. +- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points. +- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. `fini()` on one does not affect the other. +- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside `allocate_primary()`. +- **Seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case -- its plan iterator always returns the single plan (1 host x 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case. +- **Pluggable plan generation, shared host iteration**: plan generation is allocator-specific — each `job_alloc_fn` brings its own strategy (trivial, combinatorial, topology-aware, bin-packing) and creates its own `xce_alloc_plan_iter_t` internally. Host iteration is shared infrastructure — `xce_host_iter_t` has one implementation reused by all allocators, parameterized by sort order (`HOST_ORDER_ANY`, `HOST_ORDER_MOST_FREE`, etc.). Swapping allocation strategy = swapping the plugin; no core loop change. + +## Requirements + +### Requirement 2: Lazy allocation plan iterator + +**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front. + +#### Acceptance Criteria + +1. The Scheduler shall define an `xce_alloc_plan_iter_t` interface with `init(resreq, sched)`, `next() -> plan`, and `fini()` operations. The `init()` signature passes the full `xce_primary_resreq_t` (including `span`, `same`, `affinity`) and scheduler state — sufficient for any allocator to derive its plan space without future interface changes. + + **Plan-shaping fields** (on `xce_primary_resreq_t`, consumed by plan iterator internally): + - `span`: bounds host count (`nhosts_min`/`nhosts_max`) and per-host task cap (`max_per_host`, analogous to LSF `ptile`). v1 combinatorial generation ignores these (empty structs); fill when ptile/nhosts support is needed. + - `same`: co-location constraints ("all tasks on same rack"). v1 ignores; future. + - `affinity`: restrict to named host group. v1 ignores; future. + - `minTasks`/`maxTasks`: task count bounds — the primary driver of plan space. Used in v1. + - `select`: host filter expression — consumed by host iterator (Req 3), not plan iterator. + - `rusages`: resource requests — consumed by `try_alloc_on_host`, not plan iterator. + + v1 implements simple combinatorial generation: `1host×N, 2hosts×ceil(N/2), ..., Nhosts×1`. Future allocators read `span`/`same`/`affinity` to constrain the plan space without touching the `init()` signature. + +2. Each call to `next()` shall generate exactly one candidate `(nhosts, slots_per_host, host_filter)` plan without generating or materializing any other plan in the same call. + +3. The Scheduler shall order plans returned by successive `next()` calls from most-preferred to least-preferred, per the owning allocator's preference policy. + +4. `next()` shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion. + +5. Calling `fini()` shall release all resources held by the iterator, and the interface shall define no other required cleanup step. + +### Requirement 3: Lazy host candidate iterator + +**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering. + +#### Acceptance Criteria + +1. The Scheduler shall define an `xce_host_iter_t` interface with `init(plan, workers)`, `next() -> worker`, and `fini()` operations. + +2. Each call to `next()` shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator. + +3. `next()` shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call `next()` again after receiving NULL for that plan. + +4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()`. + +5. Calling `fini()` shall release all resources held by the host iterator, independent of the plan iterator's lifecycle. + +### Requirement 4: Parallel job combinatorial allocation + +**Objective:** As a cluster operator submitting a job that needs multiple task slots, I want `parjob_allocator` to actually allocate across hosts, so that jobs with `maxTasks > 1` are no longer silently unallocatable. + +#### Acceptance Criteria + +1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host x N slots, 2 hosts x ceil(N/2) slots, ..., N hosts x 1 slot, using Requirement 2's plan iterator. + +2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan. + +3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job. + +4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected. + +5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents -- the current stub behavior -- for any job with `maxTasks > 1`. + +### Requirement 7: Session-level unsatisfiable resreq cache + +**Objective:** As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost. + +#### Acceptance Criteria + +1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by `xce_cache_key_t` — a value type composing `xce_resreq_fingerprint_t` (pre-computed `{minTasks, maxTasks, select_node_id, rusage_hash}`) and identity (`{user, queue, project}` as xstring pointers). The resreq fingerprint is stored on the scheduling object (`xce_primary_schedobj_t`) and computed once on job ingest. The full cache key is constructed on the stack at lookup time — a single `kvtab_lookup()` with `cache_key_hash()` and `cache_key_eq()`. Fully O(1): no list walks, no string traversals, no allocations. This captures everything the allocator and limit plugin consume. + +2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`). + +3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job. In v1, cache population is preemption-agnostic: it fires on allocation failure alone. When the scheduler-preemption spec ships, preemption fires between allocation failure and cache population — if preemption succeeds, no cache entry is created; if preemption fails or is inapplicable, the cache entry is populated as normal. The allocation layer does not call into preemption; the caller (scheduling loop) orchestrates the sequence. + +4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption. + +5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped. + +### Requirement 9: No regression to existing allocators or HA soak + +**Objective:** As a maintainer, I want the iterator-driven rework scoped so that existing single-slot allocation and COMPOUND/ALTER dispatch are unaffected, so that this framework change is safe to ship independently of any new allocator. + +#### Acceptance Criteria + +1. **Seqjob regression**: `seqjob_allocator`'s externally observable allocation behavior (which host a `maxTasks == 1` job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the `xce_host_iter_t` interface. + +2. **COMPOUND/ALTER unchanged**: The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today. + +3. **Unit tests**: Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the eager-one-shot behavior or stub return value this feature replaces. + +4. **HA soak**: A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. + +## Data Structure Changes Summary + +### `xce_alloc_plan_iter_t` (new, Req 2) + +```c +typedef struct xce_alloc_plan_iter_s xce_alloc_plan_iter_t; + +struct xce_alloc_plan_iter_s { + // Initialize the iterator for a given primary resreq and scheduler state. + // Returns 0 on success, nonzero on error. + int (*init)(xce_alloc_plan_iter_t* iter, xce_primary_resreq_t* resreq, + xce_scheduler_t* sched); + + // Return the next candidate plan, or NULL when exhausted. + // The returned plan pointer is valid until the next next() or fini() call. + xce_alloc_plan_t* (*next)(xce_alloc_plan_iter_t* iter); + + // Release all resources held by the iterator. + void (*fini)(xce_alloc_plan_iter_t* iter); + + // Private state: current position, strategy-specific data + void* private; +}; + +// A plan describes how many hosts and slots to attempt +typedef struct { + int nhosts; // number of hosts needed + int slots_per_host; // slots required on each host + // host_filter: derived from primary_resreq->select, plus slot availability +} xce_alloc_plan_t; +``` + +### `xce_host_iter_t` (new, Req 3) + +```c +typedef struct xce_host_iter_s xce_host_iter_t; + +struct xce_host_iter_s { + // Initialize the iterator for a given plan and worker set. + // Returns 0 on success, nonzero on error. + int (*init)(xce_host_iter_t* iter, xce_alloc_plan_t* plan, + var workers); + + // Return the next candidate worker matching the plan's host filter, + // or NULL when exhausted. The returned worker pointer is valid until + // the next next() or fini() call. + xce_worker_t* (*next)(xce_host_iter_t* iter); + + // Release all resources held by the iterator. + void (*fini)(xce_host_iter_t* iter); + + // Private state: current position, filtering state + void* private; +}; +``` + +### New types (Req 7) + +```c +// Resreq portion of the cache key — pre-computed, shareable across jobs +// with identical resreqs. Fits in a single cache line (24 bytes). +typedef struct { + int min_tasks; + int max_tasks; + uint32_t select_node_id; // CSE-interned root; 0 if select is NULL + uint64_t rusage_hash; // computed at parse time, stable for life of resreq +} xce_resreq_fingerprint_t; + +// Full cache key — resreq fingerprint + identity. +// Identity fields are xstring pointers (not owned, not freed). +// Pointer equality = content equality for xstring-interned strings. +typedef struct { + xce_resreq_fingerprint_t resreq; + const char* user; // xstring pointer + const char* queue; // xstring pointer + const char* project; // xstring pointer +} xce_cache_key_t; + +// Hash — single call site, no ad-hoc field assembly +static inline uint64_t cache_key_hash(const xce_cache_key_t* k) { + uint64_t h = FNV64_INIT; + hash_update(&h, &k->resreq, sizeof(k->resreq)); + hash_update_ptr(&h, k->user); + hash_update_ptr(&h, k->queue); + hash_update_ptr(&h, k->project); + return h; +} + +// Equality — memcmp on fixed-size resreq portion + pointer equality on identity +static inline bool cache_key_eq(const xce_cache_key_t* a, const xce_cache_key_t* b) { + return memcmp(&a->resreq, &b->resreq, sizeof(a->resreq)) == 0 + && a->user == b->user + && a->queue == b->queue + && a->project == b->project; +} +``` + +### `xce_primary_resreq_t` addition (Req 7) + +One new field, set once at parse time: + +```c +// In xce_primary_resreq_t (resreq.h): +uint64_t rusage_hash; // set by parse_resreq_primary(), stable for life of resreq +``` + +### `xce_sched_obj_t` / `xce_primary_schedobj_t` addition (Req 7) + +Store the resreq portion of the fingerprint on the scheduling object so it is computed once and reused for every cache lookup of that job: + +```c +// On xce_primary_schedobj_t (or xce_sched_obj_t): +xce_resreq_fingerprint_t resreq_fp; // populated on job ingest, read by cache lookup +``` + +### Lookup site (pseudocode) + +```c +// Before allocation: construct key on stack, single hash + lookup +xce_cache_key_t key = { + .resreq = obj->u.primary->resreq_fp, + .user = obj->user, + .queue = obj->queue, + .project = obj->project, +}; +if (kvtab_lookup(sched->unsatisfiable_cache, &key, cache_key_hash, cache_key_eq)) { + pend_job(obj, original_reason); + return; +} +``` + +### Key design decisions + +- **struct-based**: hash and equality live next to the type. Single call site, no scattered field assembly. +- **`memcmp` on resreq portion**: all fields are fixed-size integers/IDs — no pointers, no deep compare. +- **pointer equality on identity**: xstring guarantees identical content → identical pointer. +- **select via `node_id`**: CSE intern table deduplicates identical select expressions. O(1). +- **rusages via pre-computed hash**: set once at parse time, stable for life of resreq. O(1). +- **span/same/affinity deferred**: currently empty structs; add to `xce_resreq_fingerprint_t` when they gain fields. + +### `xce_sched_ctx_t` addition (allocation mode) + +A mode field that controls which resource pool the allocator draws from. Defaults to `XCE_ALLOC_NORMAL`. Consumed by `try_alloc_on_host()` and any resource-availability check — applies to all resource types (slots, shared resources, limit instances, licenses, etc.), not just CPU slots. + +```c +// In xce_sched_ctx_t (sched/types.h): +typedef enum { + XCE_ALLOC_NORMAL = 0, // only truly free resources + XCE_ALLOC_PREEMPT, // count preemptible job resources as available + // Future: + // XCE_ALLOC_RESERVE, // count reserved-but-idle resources as available + // XCE_ALLOC_BACKFILL, // count reserved + preemptible resources +} xce_alloc_mode_t; + +xce_alloc_mode_t alloc_mode; // XCE_ALLOC_NORMAL by default +``` + +When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the full resource footprint of victim-candidate jobs (not just slots) as borrowable. The allocator logic is otherwise identical — no fork in `try_alloc_on_host()`. + +### `xce_preempt_victim_iter_t` (new, scheduler-preemption spec) + +Same lazy-iterator pattern as plan iterator, applied to victim selection: + +```c +typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t; + +struct xce_preempt_victim_iter_s { + int (*init)(xce_preempt_victim_iter_t*, xce_sched_job_t* job, xce_scheduler_t*); + xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t*); + void (*fini)(xce_preempt_victim_iter_t*); + void* private; +}; +``` + +The default policy (`lowest_priority_first`) generates victim sets lazily: smallest sufficient set of lowest-priority running jobs from `preemptible_queues` that would free enough resources. Listed here because the preemption flow wraps `allocate_primary()` and consumes `xce_alloc_mode_t` — both defined in this spec. + +### `parjob_allocator` changes (Req 4) + +- `my_alloc()` is replaced from a stub (`return 0`) to a real implementation that creates a plan iterator, walks candidate plans, and commits on first success. +- The allocator's registration in the plugin framework is unchanged -- only the `job_alloc_fn` implementation changes. + +### `seqjob_allocator` changes (Req 9 AC 1) + +- Internal rewrap: `my_alloc()` is restructured to use the plan and host iterator interfaces internally as a trivial case (1 host x 1 slot plan, then host iteration matching today's `sched_collect_workers()` + `xlist_find()` logic). +- Externally observable behavior is unchanged -- same host selection given same state. + +## Architecture Diagram + +### Allocation Flow (PRIMARY leaf) + +``` +┌──────────────────────────────────────────────────────────┐ +│ ALLOCATE PRIMARY (Req 2+3+4) │ +│ │ +│ plan_iter_init(primary_resreq) │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────┐ │ +│ │ plan_iter->next() │ ←─ generates (nhosts, │ +│ │ returns plan or NULL │ slots_per_host) candidates │ +│ └──────────┬───────────┘ most-preferred first │ +│ │ │ +│ plan │ NULL (exhausted) │ +│ ▼ ▼ │ +│ ┌──────────────────┐ return ALLOC_FAIL │ +│ │ host_iter_init() │ │ +│ └────────┬─────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────┐ │ +│ │ host_iter->next() │ ←─ filters workers by │ +│ │ returns worker or NULL│ select expression + │ +│ └──────────┬───────────┘ slot availability │ +│ │ │ +│ worker │ NULL (no host for plan) │ +│ ▼ ▼ │ +│ ┌──────────────────┐ host_iter->fini() │ +│ │ try_alloc(worker, │ plan_iter->next() (next plan) │ +│ │ plan, ctx) │ │ +│ └────────┬─────────┘ │ +│ │ │ +│ success │ fail │ +│ ▼ ▼ │ +│ ┌────────────┐ host_iter->next() (next worker for plan) │ +│ │ commit │ │ +│ │ return │ │ +│ │ ALLOC_SUCCESS│ │ +│ └────────────┘ │ +└──────────────────────────────────────────────────────────┘ +``` + +### Unsatisfiable Resreq Cache (Req 7) + +``` +┌──────────────────────────────────────────────────────────┐ +│ SESSION START (begin_session) │ +│ │ +│ Clear unsatisfiable resreq fingerprint cache │ +│ │ +│ For each job from picker: │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────────────┐ │ +│ │ Lookup fingerprint in cache │ │ +│ │ key = hash(minTasks, │ │ +│ │ maxTasks, select_node_id, │ │ +│ │ rusage_hash, user, queue, │ │ +│ │ project) │ │ +│ └──────────┬───────────────────┘ │ +│ │ │ +│ hit │ miss │ +│ ▼ │ ▼ │ +│ ┌────────┐ │ ┌──────────────────┐ │ +│ │ PEND │ │ │ Run full │ │ +│ │ immedi- │ │ │ allocation │ │ +│ │ ately │ │ │ (Req 2+3+4) │ │ +│ └────────┘ │ └────────┬─────────┘ │ +│ │ │ │ +│ │ success │ fail (exhausted) │ +│ │ ▼ ▼ │ +│ │ ┌────────────┐ ┌──────────────────────┐ │ +│ │ │ DISPATCH │ │ Populate cache with │ │ +│ │ │ (no cache │ │ resreq fingerprint │ │ +│ │ │ entry) │ │ then PEND job │ │ +│ │ └────────────┘ └──────────────────────┘ │ +└──────────────────────────────────────────────────────────┘ +``` + +## Integration Note: Preemption Interaction + +Preemption (scheduler-preemption spec) fires when allocation fails for a preemptible job. It follows the same lazy-iterator pattern as allocation, applied to victim selection instead of host selection. + +### Preemption Iterator + +Same shape as plan iterator, different domain: + +```c +// Preemption victim iterator — lazy generation of victim sets +typedef struct xce_preempt_victim_iter_s xce_preempt_victim_iter_t; + +struct xce_preempt_victim_iter_s { + int (*init)(xce_preempt_victim_iter_t* iter, xce_sched_job_t* job, + xce_scheduler_t* sched); + xce_preempt_victim_set_t* (*next)(xce_preempt_victim_iter_t* iter); + void (*fini)(xce_preempt_victim_iter_t* iter); + void* private; +}; +``` + +The default policy (`lowest_priority_first`) generates victim sets in order: smallest sufficient set of lowest-priority running jobs from `preemptible_queues` that would free enough resources for the preempting job. + +### Allocation Mode + +A `xce_alloc_mode_t` enum on `xce_sched_ctx_t` controls which resource pool the allocator draws from — applies to all resource types (slots, shared resources, limit instances, licenses), not just CPU slots: + +```c +typedef enum { + XCE_ALLOC_NORMAL = 0, // only truly free resources + XCE_ALLOC_PREEMPT, // count preemptible job resources as available +} xce_alloc_mode_t; +``` + +When `alloc_mode != XCE_ALLOC_NORMAL`, resource-availability checks count the full resource footprint of victim-candidate jobs as borrowable. The allocator logic is otherwise identical — no fork in `try_alloc_on_host()`. + +### Full Preemption Flow + +```c +int allocate_job(sched, job) { + // 1. Cache check + if (cache_lookup(job)) return pend(job); + + // 2. Normal allocation + if (allocate_primary(sched, job->resreq, ctx) > 0) return dispatch(job); + + // 3. Preemption (if applicable) + if (job->queue->preemptible_queues == NULL) { + cache_insert(job); + return pend(job); + } + + victim_iter_init(&viter, job, sched); + while ((victims = viter.next(&viter)) != NULL) { + ctx.alloc_mode = XCE_ALLOC_PREEMPT; + if (allocate_primary(sched, job->resreq, &ctx) > 0) { + viter.fini(&viter); + return commit_preempt(victims, job); + } + ctx.alloc_mode = XCE_ALLOC_NORMAL; + } + viter.fini(&viter); + + cache_insert(job); + return pend(job); +} +``` + +Key properties: +- v1 cache is preemption-agnostic (populated on allocation failure alone). +- When preemption ships, it fires between allocation failure and cache population. +- If preemption succeeds → no cache entry, preempting job is re-pended for next round. +- If preemption fails/exhausted → cache entry populated, job pended normally. +- Plan and host iterators are not preemption-aware — they simply report success/failure. +- `xce_alloc_mode_t alloc_mode` is the only new allocator surface needed. + +## Implementation Order + +Scheduler sub-specs are designed for sequential delivery: + +``` +Phase 1: scheduler-allocation (this spec) + ├── plan_iter + host_iter foundation iterators + ├── seqjob rewrap validates abstraction on simple case + ├── parjob combinatorial real parallel allocation + └── resreq cache performance optimization + +Phase 2: scheduler-core + ├── core/driver split extracts xce_sched_core_t from xce_scheduler_t + ├── picker interface xce_job_picker_iface_t + builtin_fcfs + └── session loop wires core_go() calls allocate_job() + +Phase 3: scheduler-preemption + ├── victim_iter lazy victim-set generator + ├── XCE_ALLOC_PREEMPT mode tells allocator to count borrowed resources + └── preempt_policy plugin lowest_priority_first default +``` + +| Phase | Why there | +|---|---| +| Allocation first | Self-contained. Unit-testable with fake workers. No core, no preemption needed. Iterators are the building block everything else sits on. | +| Core second | Wires allocation into a real scheduling loop. Picker feeds jobs → allocator places them → decisions flow. Without allocation, core has nothing to call. | +| Preemption last | Wraps allocation. Needs `allocate_primary()` to exist. Needs core session loop to re-pend preempted job. Is a "retry with borrowing" layer on top of normal allocation. | diff --git a/.kiro/specs/scheduler-allocation/spec.json b/.kiro/specs/scheduler-allocation/spec.json new file mode 100644 index 00000000..6bca5a89 --- /dev/null +++ b/.kiro/specs/scheduler-allocation/spec.json @@ -0,0 +1,26 @@ +{ + "feature_name": "scheduler-allocation", + "created_at": "2026-07-27T00:00:00Z", + "updated_at": "2026-07-27T00:00:00Z", + "language": "en", + "phase": "requirements-generated", + "parent_spec": "scheduler-framework", + "description": "Lazy iterator-driven allocation architecture with parallel job support and resreq caching for the XCE scheduler", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "blocked_on": ["scheduler-core"], + "sibling_specs": ["scheduler-core", "scheduler-preemption"] +} diff --git a/.kiro/specs/scheduler-core/brief.md b/.kiro/specs/scheduler-core/brief.md new file mode 100644 index 00000000..53f1452f --- /dev/null +++ b/.kiro/specs/scheduler-core/brief.md @@ -0,0 +1,168 @@ +# Brief: scheduler-core + +## Problem + +The XCE scheduler has a plugin architecture but critical foundational gaps: + +1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation. + +2. **Monolithic scheduler tied to controller I/O** — `xce_scheduler_t` holds both scheduling state and controller API context (`apictx`, `cluster_cfg`). No way to replay recorded events for debugging or test scheduling policies without a live cluster. + +3. **No picker plugin interface** — the existing plugin framework has `job_elect_fn` and `job_alloc_fn`, but no defined contract for how the picker receives state-change events, manages sessions, or communicates decisions back. The FCFS picker is tightly coupled to the scheduler internals. + +4. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. + +Operators need configurable scheduling policies, and developers need a way to debug/tune scheduling behavior without a live cluster. + +## Current State + +### What exists +- **Plugin framework** (`plugin.h`, `plugin.c`): `xce_sched_plugin_t` with 9 hooks + `job_elect_fn` / `job_alloc_fn` +- **FCFS job picker** (`fcfs.c`): dedicated `jobpicker` thread. Iterates queues by priority, jobs by (priority desc, queue_time asc). Communicates via `jbuf` cbuf. +- **Scheduling loop** (`scheduler.c`): pick job → allocate → pend or dispatch → next +- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`. +- **Select expression memoization** (`select_eval_ctx_t`): per-session AST caching + +### What's missing +- Pluggable job picker with defined interface contract +- Configurable picker name (no recompilation) +- Core/driver separation: scheduling logic decoupled from controller I/O +- Simulation mode: replay recorded events, virtual clock, deterministic execution +- Runtime recording for debugging/tuning +- State snapshot/restore for replay efficiency + +## Desired Outcome + +### 1. Pluggable Job Picker + +The core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no `jbuf` cbuf. + +Picker plugin interface (`xce_job_picker_iface_t`): + +| Category | Hooks | Purpose | +|----------|-------|---------| +| Lifecycle | `init`, `fini` | Setup and teardown | +| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called when events arrive | +| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle | +| Post-allocation | `on_dispatch`, `on_pend` | After allocation attempt | +| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) | + +### 2. Core/Driver Separation + +Scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it: + +| Mode | Driver | Input | Output | +|------|--------|-------|--------| +| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller | +| Simulation | Replay file | JSONL event file → core events | Core decisions → log | + +- **Mode-agnostic core**: no mode flag, no mode-specific code paths +- **Virtualized clock**: `xce_core_now(core)` with two-base + speed formula +- **Discrete event-driven virtual clock**: jumps to next event timestamp +- **Snapshot + delta replay**: load snapshot, replay delta events +- **Deterministic guarantee**: same initial state + same events = same decisions +- **Runtime recording control**: xceadmin `scheduler record start/stop` + +### 3. Single Builtin Picker + +`builtin_fcfs` internally implements queue-priority + FCFS job ordering — replicates today's behavior. +Per-queue picker overrides supported. + +### Queue vs Project — Roles + +| | Queue | Project | +|---|---|---| +| **Role** | Scheduling mechanism | Resource ownership | +| **Question** | HOW are jobs ordered? | WHO owns resources? | +| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy (future) | +| **Cardinality** | 1 job → 1 queue | 1 job → 1 project | + +## Approach + +**Event-Driven Core with Pluggable Picker + Dual-Mode Driver**: + +### Core scheduling loop + +```c +picker->begin_session(picker); +while ((job = picker->next_job(picker)) != NULL) { + if (try_allocate(job) > 0) { + picker->on_dispatch(picker, job->jobid); + on_decision(driver_ctx, DISPATCH, job); + } else if (can_preempt(job)) { + on_decision(driver_ctx, PREEMPT, job, victims); + picker->on_pend(picker, job->jobid); + } else { + on_decision(driver_ctx, PEND, job); + picker->on_pend(picker, job->jobid); + } +} +picker->end_session(picker); +``` + +Core knows nothing about project/queue/job ordering. `next_job()` hides all selection. + +### Clock virtualization + +```c +uint64_t xce_core_now(core) { + wall_ns = clock_gettime(CLOCK_MONOTONIC); + elapsed = wall_ns - core->real_clock_base; + return core->virtual_clock_base + (elapsed * speed_num) / speed_den; +} +``` + +Normal: bases=0, speed=1/1 → returns wall clock. Simulation discrete: speed=0/1 → frozen at event ts. + +### What stays unchanged +- Plugin framework hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved +- Allocation logic — unchanged (in scheduler-allocation sibling spec) +- Controller communication in driver — extended, not replaced + +### What's new +- `xce_job_picker_iface_t` — pluggable picker with full hook specification +- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t` +- `xce_core_event_t` — typed event interface +- `xce_core_decision_cb_t` — decision callback +- JSONL replay format + snapshot/restore +- `xce_core_now()` — virtualized clock accessor +- `builtin_fcfs` — single builtin picker +- `job_pick_plugin` on scheduler + per-queue override +- xceadmin recording commands + +## Scope + +### In +- Pluggable job picker interface with full hook specification +- Single builtin picker (`builtin_fcfs`) +- Configurable picker plugin with per-queue override +- Core/driver separation: passive core, normal driver, simulation driver +- Virtualized clock (`xce_core_now()` with two-base + speed formula) +- Discrete event-driven virtual clock +- JSONL replay format (HEADER + SNAPSHOT + events + DECISIONS) +- State snapshot/restore +- Deterministic replay guarantee +- Runtime recording control via xceadmin commands +- Data structures: `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t` +- Unit tests: picker hook behavior, simulation replay, snapshot/restore, deterministic verification +- HA soak regression for picker changes + +### Out +- Allocation iterators and parjob allocator (→ scheduler-allocation) +- Preemption policy and SIGSTP pipeline (→ scheduler-preemption) +- New picker policies beyond builtin_fcfs +- Hierarchical project resource quotas (→ project-resource-ownership) +- Dead field on `xce_sched_job_t` + +## Sibling Specs +- **scheduler-allocation**: Iterator-driven allocation, parjob allocator, resreq cache +- **scheduler-preemption**: Preemption policy, victim selection, SIGSTP pipeline + +## Constraints +- **C (gnu17)** — no C++, no external libraries beyond existing +- **Cancel-safe** — `next_job()` has pthread_cancel cancellation points +- **Session isolation** — picker state reset on begin_session +- **Backward compatible** — existing FCFS behavior unchanged +- **Deterministic replay** — same state + events → same decisions +- **Mode-agnostic core** — no mode flag, no mode-specific code paths +- **No wall-clock time in core** — all time through xce_core_now() diff --git a/.kiro/specs/scheduler-core/requirements.md b/.kiro/specs/scheduler-core/requirements.md new file mode 100644 index 00000000..cbcf7c86 --- /dev/null +++ b/.kiro/specs/scheduler-core/requirements.md @@ -0,0 +1,1311 @@ +# Requirements Document + +## Introduction + +The XCE scheduler has a plugin architecture but critical foundational gaps. Job selection policy is hardcoded (`sched_job_picker()` sets `job_pick_plugin = "xce_sched_fcfs"` as a literal). The scheduler is tightly coupled to controller I/O (`apictx`, `cluster_cfg` embedded in `xce_scheduler_t`). There is no way to swap picking policies without recompilation, and no way to debug or tune scheduling behavior without a live cluster. + +This spec establishes one central design invariant and three architectural foundations to achieve it. + +**Central invariant — Normal↔Sim equivalence**: Given identical initial configuration and identical event sequences, the scheduler shall produce identical scheduling decisions regardless of whether it runs in production (normal mode, controller-connected, wall-clock time) or in simulation (replay file, virtual clock). Record a live run → replay it in simulation → the decisions match byte-for-byte. This is NOT a debugging convenience — it is the **validation contract** for the entire architecture. If normal and simulation diverge, the core/driver separation is broken. This invariant enables every simulation-mode use case: debugging is faithful, tuning is predictive, regression testing is trustworthy, and what-if analysis reflects real behavior. + +Three foundations deliver this invariant: + +**Pluggable job picker**: A `xce_job_picker_iface_t` — the picker thread calls `picker->next_job()` to get the next schedulable job and hands it to the allocator thread via cbuf, with zero knowledge of how jobs are selected. The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior. Per-queue picker overrides are supported. The picker runs in a dedicated thread communicating with the allocator via cbuf (existing producer/consumer pattern preserved). + +**Core/driver separation**: The scheduler core is a passive, event-driven engine with zero controller I/O dependencies. A normal-mode driver connects it to a live controller; a simulation-mode driver replays recorded cluster events through a discrete event-driven virtual clock, enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. + +**Core is mode-agnostic**: The core has no concept of which mode it runs in. It processes events identically whether the driver is in normal mode or simulation mode. There is no mode flag, no mode-specific code path, and the same init function is used for both modes. + +> Full discovery context is in `.kiro/specs/scheduler-core/brief.md`. Sibling specs: `scheduler-allocation`, `scheduler-preemption`. + +## Queue vs Project — Roles + +These are fundamentally different concerns: + +| | Queue | Project | +|---|---|---| +| **Role** | Scheduling mechanism | Resource ownership | +| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? | +| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster | +| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy, members (future) | +| **Cardinality to job** | 1 job → 1 queue | 1 job → 1 project | +| **Cardinality to each other** | Many-to-many | Many-to-many | + +- **Queue** = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, priority (for inter-queue ordering within the picker), per-queue picker override, and preemption config. +- **Project** = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2). + +## Boundary Context + + - **In scope**: Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic behind a single plugin; single builtin picker (`builtin_fcfs`) replicating today's behavior through the new interface; configurable picker plugin with per-queue override (`queue->job_pick_plugin`); **core/driver separation** with passive event-driven core interface, mode-agnostic core (no mode flag, no mode-specific code paths), normal-mode driver (controller-connected), simulation-mode driver as a **standalone investigation tool** (replay recorded events with discrete event-driven virtual clock, zero controller dependencies, self-contained `.replay` file — single capnp binary stream with header + snapshot + events + decisions + periodic checkpoints in one artifact, what-if editing via event file substitution, diff-based comparison of simulated vs recorded decisions via `--sim-output`), virtualized clock access (`xce_core_now()` with two-base + speed formula), runtime recording control via xceadmin commands, `.replay` capnp stream format, state snapshot/restore for replay efficiency, **interactive REPL** with single-step, breakpoints, state inspection, and snapshot capture via `--interactive`, and deterministic replay guarantee for debugging/tuning; **normal↔sim equivalence guarantee** (same config + same events → identical decisions regardless of mode, AC 7.1); `xceadmin scheduler dump ` for human-readable replay file inspection; documentation of picker plugin interface contract (hooks, cancel-safe, session-aware); unit tests for picker hook behavior, **event recording/replay, snapshot/restore, and deterministic replay**; HA soak regression for picker changes. + +- **Out of scope**: Allocation iterators and parjob allocator (→ `scheduler-allocation` spec); preemption policy and SIGSTP pipeline (→ `scheduler-preemption` spec); new job selection policies beyond the builtin FCFS picker — the pluggable picker interface enables them, but none ship in this spec beyond `builtin_fcfs`; hierarchical project resource quotas (deferred to `project-resource-ownership` spec — only `project_name` on queue and `xce_sched_project_t` scaffolding ship here); decision payload capnp schema for `DispatchDecision`/`PendDecision`/`PreemptDecision` (deferred to design phase — replay file format schema is in AC 20); distributed/cluster-mode simulation (v1 is single-scheduler replay only); visual inspection / UI tool for replay files (deferred — `xceadmin scheduler dump` provides human-readable output, `diff` of dumps is the primary comparison workflow for v1). + +- **Adjacent expectations**: Depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking — none of these change. Sibling spec `scheduler-allocation` provides the `try_allocate()` function this spec's core loop calls. Sibling spec `scheduler-preemption` provides the `can_preempt()` check and preemption decision type. + +## Implementation Phases + +The spec is implemented in three sequential phases. Each phase ships independently, passes HA soak regression, and preserves existing scheduling behavior. Phases are ordered by dependency — earlier phases produce primitives that later phases consume. + +### Phase 1: Pluggable Picker (Requirements 1–4) + +**Scope**: Replace the hardcoded `"xce_sched_fcfs"` literal with a configurable `xce_job_picker_iface_t`. Implement `builtin_fcfs` as the default plugin, replicating today's FCFS behavior exactly through the new interface hooks. + +**Changes to existing code**: +- Add `xce_job_picker_iface_t` with lifecycle, state-change, session, and post-allocation hooks (Req 1) +- Implement `builtin_fcfs` plugin with private `ordq_t` state, all MUST hooks (Req 3) +- Replace `sched_job_picker()` hardcoded string with `plugin_register`/`key_find` lookup (Req 1 AC 7) +- Add `job_pick_plugin` to `xce_scheduler_t` and `xce_sched_queue_t` (Req 2) +- Add `xce_sched_project_t` scaffolding + `project_name` on queue (Req 2 AC 3) + +**What does NOT change**: +- Today's main loop (`sched_update` → `sched_begin_session` → `sched_go` → decision drain → `sched_end_session`) is untouched +- Today's threads (picker, allocator) and cbuf communication are untouched +- Today's controller integration (`scheduler_req_handler`, capnp) is untouched +- Today's build system, test suite structure is untouched + +**Verification**: HA soak — same job-picking order as today, same decisions, zero divergence. + +**Why first**: Phase 2's `core_push_events()` must call picker hooks (`on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update`). Implementing the picker first means Phase 2 only moves existing hook call sites — same calls, different location. No dead wiring, no incomplete interfaces. + +### Phase 2: Core/Driver Separation (Requirement 5, ACs 1–8) + +**Scope**: Extract the scheduling core into a passive, mode-agnostic engine (`xce_sched_core_t`). Introduce the three-call driver↔core protocol (`core_push_events`, `core_go`, `on_decision`). Build the normal-mode driver that recreates today's controller-connected behavior. + +**Changes to existing code**: +- Extract `xce_sched_core_t` from `xce_scheduler_t` — all scheduling state, zero controller deps (AC 1) +- Implement `core_push_events()` — batch-push events, call picker hooks synchronously (AC 2, AC 4) +- Implement `core_go()` — non-blocking signal to picker thread via `cbuf_put(&wbuf)` (AC 6) +- Implement `on_decision()` callback — per-decision emission to driver (AC 6) +- Build normal-mode driver: capnp → core events → decisions → xcecli (AC 8) +- Replace today's `main.c` decision switch with driver callback + +**What does NOT change**: +- Picker interface and `builtin_fcfs` plugin (Phase 1 already done) +- Threading: picker → jbuf → allocator → decisions cbuf pattern preserved +- Controller integration: same capnp messages, same xcecli API, same behavior +- Simulation: not yet wired — core is mode-agnostic but no sim driver exists + +**Verification**: HA soak — same decisions as Phase 1, same controller interaction, zero regression. + +**Why second**: Core extraction needs the picker hooks to exist (Phase 1 done). Core extraction is a prerequisite for Phase 3 — simulation needs a mode-agnostic core that is already tested and passing HA soak in production. + +### Phase 3: Simulation Mode + Recording + Interactive REPL (Requirement 5, ACs 9–22) + +**Scope**: Add virtual clock (`xce_core_now()`), snapshot/restore (`save_snapshot`/`load_snapshot`), simulation driver (replay file → events → decisions → record file), interactive REPL (step, breakpoints, state inspection, snapshot capture), and runtime recording hooks in the normal-mode driver. + +**Changes to existing code**: +- Add clock virtualization atoms to core: `virtual_clock_base`, `real_clock_base`, `speed_num`, `speed_den` (AC 3) +- Implement `save_snapshot()` / `load_snapshot()` with picker rebuild from core state (AC 21) +- Build simulation driver: load HEADER+SNAPSHOT, replay events, emit decisions to `--sim-output` (AC 13, AC 19) +- Build interactive REPL: stdin/stdout command loop, breakpoint matching, state inspection queries (AC 22) +- Add recording state to normal driver: `scheduler record start/stop`, event writing, periodic snapshots (AC 10–11, AC 12) +- Add simulation CLI flags: `--sim`, `--sim-speed`, `--sim-verify`, `--sim-output`, `--interactive` / `-i` + +**What does NOT change**: +- Core is already mode-agnostic (Phase 2 done) — no `if (simulation)` branches added +- Picker is already clean (Phase 1 done) — no changes to picker interface +- Normal driver is already split (Phase 2 done) — recording hooks are boolean-gated, zero overhead when off +- Production scheduling behavior is identical — simulation is an opt-in mode + +**Verification**: HA soak (normal mode — no regression). **Normal↔sim equivalence** (AC 7.1): record a live run, replay in simulation, `diff <(xceadmin scheduler dump recorded.replay) <(xceadmin scheduler dump simulated.replay)` → decision content identical. **Deterministic replay** (AC 7): same `.replay` file twice → same output. **Recording**: file written correctly, simulation replays recorded file correctly. **Interactive REPL** (AC 22): `--interactive` mode steps, breaks, and completes to identical decisions as batch `--sim` mode. State inspection commands report accurate live state (verified against known-good snapshot dumps). + +**Why last**: Simulation needs Phase 1 (picker hooks exist) and Phase 2 (core is mode-agnostic, driver boundary is clean). It is the only phase that adds net-new capability rather than restructuring existing code — all risk is in Phase 1 and 2. By Phase 3, the new architecture has already been validated in production via HA soak. + +### Phase Dependency Graph + +``` +Phase 1: Pluggable Picker ──► Phase 2: Core/Driver Split ──► Phase 3: Simulation + (Req 1-4) (Req 5 AC 1-8) (Req 5 AC 9-22) + │ │ │ + │ No deps │ Needs picker hooks │ Needs mode-agnostic core + │ │ from Phase 1 │ + clean driver boundary + │ │ │ from Phase 2 + ▼ ▼ ▼ + HA soak pass HA soak pass HA soak pass +``` + +Each phase: TDD (red → green → refactor) → HA soak regression → merge. No phase gates on unfinished work from a later phase. Each phase ships value independently. + +## Requirements + +### Requirement 1: Pluggable job picker interface + +**Objective:** As a maintainer of the scheduler, I want the core scheduling loop to call a single `next_job()` plugin interface with no knowledge of how jobs are selected, so that picking strategies (FCFS, priority, fair-share, deadline, etc.) are truly pluggable and the core loop never changes regardless of the picking algorithm. + +#### Acceptance Criteria +1. The Scheduler core loop shall call `picker->next_job(picker)` to get the next schedulable job (or NULL if none ready), and shall not contain any logic about project iteration, queue iteration, or job ordering — those concerns are entirely encapsulated inside the picker plugin. + +2. The job picker plugin interface (`xce_job_picker_iface_t`) shall define these operations, organized by category: + + **Lifecycle hooks** (MUST): + + | Operation | Signature | Semantics | + |-----------|-----------|-----------| + | `init` | `int (*init)(void* picker, xce_sched_core_t* core)` | One-time setup, receives core reference for state queries | + | `fini` | `void (*fini)(void* picker)` | Teardown and cleanup of all picker-owned resources | + + **State-change hooks** (MUST — called by core when events arrive from driver): + + | Operation | Signature | Semantics | Triggered by | + |-----------|-----------|-----------|-------------| + | `on_queue_update` | `int (*on_queue_update)(void* picker, xce_sched_queue_t* queue)` | Add/update queue in picker's internal ordering | `QUEUE_UPDATE` event | + | `on_job_push` | `int (*on_job_push)(void* picker, xce_sched_job_t* job)` | Add new pending job to picker's candidate set | `JOBS_PUSH` event | + | `on_job_state_change` | `int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t new_state)` | Remove job from candidates if no longer pending; re-add if returning to pending | `JOB_STATE_CHANGE` event | + | `on_job_update` | `int (*on_job_update)(void* picker, xce_sched_job_t* job)` | Reorder job in candidates if priority/queue changed | `JOB_UPDATE` event | + + **Session hooks** (MUST — called at round boundaries): + + | Operation | Signature | Semantics | + |-----------|-----------|-----------| + | `begin_session` | `int (*begin_session)(void* picker)` | Reset iteration cursor; called at start of each scheduling round | + | `next_job` | `xce_sched_job_t* (*next_job)(void* picker)` | Return next schedulable job per picker's ordering, or NULL when exhausted | + | `end_session` | `int (*end_session)(void* picker)` | End-of-round notification; called before core sleeps between rounds | + + **Post-allocation hooks** (MUST — called after allocation attempt): + + | Operation | Signature | Semantics | + |-----------|-----------|-----------| + | `on_dispatch` | `int (*on_dispatch)(void* picker, uint64_t jobid)` | Job was allocated; remove from candidates permanently | + | `on_pend` | `int (*on_pend)(void* picker, uint64_t jobid)` | Job was pended; keep for next round but may deprioritize | + + **Optional hooks** (MAY — NULL if not needed): + + | Operation | Signature | Semantics | When needed | + |-----------|-----------|-----------|------------| + | `on_config` | `int (*on_config)(void* picker, xce_conf4sched_t* config)` | Rebuild picker strategy from config | Config-driven pickers (deadline, fair-share weights) | + | `on_job_rusage` | `int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages)` | Track per-job resource usage | Resource-aware pickers (fair-share, backfill) | + + Hooks not defined on the picker interface (worker updates, resource updates, resval updates) are not forwarded to the picker — picking is about job ordering, not resource availability. + +3. **Threading model**: The core uses three threads communicating via cbuf (bounded FIFO with mutex + condition variables, `src/include/mylock.h`): + + | Thread | Role | cbuf In | cbuf Out | Blocked On | + |--------|------|---------|----------|------------| + | Picker | Iterates queues/jobs via `next_job()`, pushes jobids | `wbuf` (from core main) | `jbuf` (to allocator) | `cbuf_get(&wbuf)` between sessions | + | Allocator | Calls `try_allocate()`/`can_preempt()`, builds decisions | `jbuf` (from picker) | `decisions` (to driver) | `cbuf_get(&jbuf)` between jobs | + | Core Main (driver context) | Calls `core_push_events()`, signals `core_go()`, drains decisions | `decisions` (from allocator) | `wbuf` (to picker) | `cbuf_get(&decisions)` during session | + + **Driver → Core event delivery**: State-change hooks (`on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update`) are called synchronously during `core_push_events()` in the driver thread, before the call returns. This is safe because `core_push_events()` is only called between sessions when the picker thread is idle (blocked on `cbuf_get(&wbuf)`) and the allocator thread is idle (blocked on `cbuf_get(&jbuf)`). No mid-session event injection. + + **Session hooks** (`begin_session`, `next_job`, `end_session`) run in the picker thread. **Post-allocation hooks** (`on_dispatch`, `on_pend`) run in the allocator thread. + +4. The core scheduling loop shall use a **producer/consumer pattern across three threads** communicating via cbuf: + + **Picker thread** (`fcfs_elect_job` equivalent): + ```c + while (1) { + int go = 0; + cbuf_get(&core->wbuf, &go); // block until core_go() signals + picker->begin_session(picker); + xce_sched_job_t* job; + while ((job = picker->next_job(picker)) != NULL) { + cbuf_put(&core->jbuf, &job->jobid); // hand off to allocator + } + uint64_t end_marker = 0; + cbuf_put(&core->jbuf, &end_marker); // end-of-session signal + picker->end_session(picker); + } + ``` + + **Allocator thread**: + ```c + while (1) { + uint64_t jobid = 0; + cbuf_get(&core->jbuf, &jobid); // block until picker sends job + if (jobid == 0) { + // end-of-session marker + cbuf_put(&core->decisions, NULL); + continue; + } + xce_sched_job_t* job = sched_get_job(core, jobid); + int rc = try_allocate(job); + if (rc > 0) { + picker->on_dispatch(picker, job->jobid); + decision = make_dispatch_decision(job->allocs); + } else if (can_preempt(job)) { + decision = make_preempt_decision(job, victims); + picker->on_pend(picker, job->jobid); + } else { + decision = make_pend_decision(job->reason); + picker->on_pend(picker, job->jobid); + } + cbuf_put(&core->decisions, &decision); + } + ``` + + **Main loop** (driver thread context — `core.go()`): + ```c + atomic_store(&core->session_active, 1, memory_order_release); + core_begin_session_internal(core); // select eval ctx, reset workers + cbuf_put(&core->wbuf, &go); // NON-BLOCKING: signal picker, return + // Driver then drains decisions cbuf: + while (1) { + cbuf_get(&core->decisions, &decision); + if (decision == NULL) break; // session exhausted + core->on_decision(core->driver_ctx, decision); // publish immediately + } + core_end_session_internal(core); // GC, reset workers + atomic_store(&core->session_active, 0, memory_order_release); + ``` + + The picker and allocator contain zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`. The allocator contains zero knowledge of how the driver publishes decisions — it only calls `cbuf_put(&decisions, &decision)`. + +5. The picker plugin shall be cancel-safe: `next_job()` shall have defined `pthread_cancel` cancellation points. The picker shall be session-aware: `begin_session()` resets iteration state, `end_session()` is called before sleep-between-rounds. + +6. The hardcoded `"xce_sched_fcfs"` literal at `scheduler.c:43` (in `sched_job_picker()`) shall be replaced by loading the configured `sched->job_pick_plugin` via the existing `plugin_register`/`key_find` mechanism. If the configured plugin name is not registered, the Scheduler shall fail with a clear configuration error. + +7. The existing multi-threaded producer/consumer architecture (picker thread → `jbuf` cbuf → allocator thread → `decisions` cbuf → main loop) shall be preserved. The driver thread signals the picker thread via `wbuf` cbuf (non-blocking `cbuf_put`). The only thread eliminated is the conceptual one: today's FCFS picker is a dedicated thread, and it stays a dedicated thread — but the core scheduling loop in the driver is simplified because decision publishing moves from a hardcoded `main.c` switch to a driver-registered `on_decision` callback. + +### Requirement 2: Configurable picker plugin + +**Objective:** As a cluster operator, I want to configure which job-picking policy the scheduler uses without recompiling, and override it per-queue, so that FCFS, priority, or custom pickers can be swapped operationally. + +#### Acceptance Criteria +1. The Scheduler shall add a `job_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_fcfs"` when unset — this is the global picker plugin name. + +2. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_queue_t`, defaulting to NULL (inherit from global). When non-NULL, the configured global picker plugin shall use this per-queue override for jobs within that queue. How the picker plugin uses per-queue overrides is an implementation detail of the plugin, not the core loop. + +3. The Scheduler shall add a `project_name` field to `xce_sched_queue_t`, set from queue configuration, establishing the queue→project relationship. The picker plugin may use this field internally for project-scoped iteration. + +4. The Scheduler shall accept the picker plugin name from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild. + +5. When the configured plugin name is not registered via the existing `plugin_register`/`key_find` mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default. + +6. The picker plugin interface contract (`xce_job_picker_iface_t`) shall be documented: each picker must be cancel-safe (responds to `pthread_cancel` at defined points in `next_job()`) and session-aware (resets its iteration state on `begin_session()`). + +### Requirement 3: Builtin default picker + +**Objective:** As a maintainer, I want the new architecture to ship with a builtin picker plugin that replicates today's exact scheduling behavior, so that operators who don't configure a custom picker see no behavioral change. + +#### Acceptance Criteria + +1. **Registration**: `builtin_fcfs` shall be registered automatically at scheduler startup as `"builtin_fcfs"` and set as the default `sched->job_pick_plugin`. + +2. **Hook implementation** — `builtin_fcfs` shall implement every MUST and SHOULD hook from `xce_job_picker_iface_t` (Req 1 AC 2) as follows: + + | Hook | FCFS Implementation | + |------|-------------------| + | `init` | Allocate `fcfs_plugin_data_t`. Initialize `ordq_t queues` (ordered by `queue->priority desc`). Set iteration cursor to NULL. | + | `fini` | Free all per-queue `ordq_t pendJobs`, the `ordq_t queues`, and plugin data. | + | `on_queue_update` | Upsert queue into `ordq_t queues`. If queue exists, update its priority and reorder. If new, allocate `fcfs_queue_t` with empty `ordq_t pendJobs`. | + | `on_job_push` | If `job->state & JOB_STAT_PEND`: find the job's queue in `ordq_t queues`, add `fcfs_job_t{job}` to that queue's `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). | + | `on_job_state_change` | If `new_state` is not pending (EXIT/FAIL/DONE): remove job from its queue's `pendJobs` by jobid. If `new_state` returns to PEND: re-add to `pendJobs`. | + | `on_job_update` | If job's `priority` or `queue` field changed: remove from old position in `pendJobs`, re-insert at correct position per ordering. | + | `begin_session` | Reset iteration cursor to head of `ordq_t queues`. | + | `next_job` | Walk `ordq_t queues` from cursor. For each queue, skip if `sched_queue_is_ready()` returns false. Within queue, walk `ordq_t pendJobs`. Return first job where `sched_job_is_ready()` returns true. Advance cursor. Return NULL when all queues and jobs exhausted. | + | `end_session` | No-op for FCFS (iteration state reset in `begin_session`). | + | `on_dispatch` | Remove job from its queue's `ordq_t pendJobs` by jobid. | + | `on_pend` | No-op for FCFS — job remains in `pendJobs`, will be retried next round in same position. | + | `on_config` | NULL — FCFS is config-agnostic. | + | `on_job_rusage` | NULL — FCFS is not resource-aware. | + +3. **Internal data structures** — `builtin_fcfs` shall maintain its own private state, not query the core's `sched->queues`/`sched->jobs` during `next_job()`. All candidate management happens in the state-change hooks. This ensures: + - `next_job()` is fast — no core lock acquisition, no hash table scans + - The picker is independently testable — feed hooks, verify `next_job()` order + - Simulation snapshots capture picker state naturally via the hooks already recorded in the event stream + +4. **Ordering equivalence** — `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`: inter-queue by descending `queue->priority`, intra-queue by `priority desc, queue_time asc`. Behavior with per-queue picker overrides is as specified in Req 2 AC 2. + +5. **Dedicated picker thread retained** — `builtin_fcfs` shall implement `next_job()` as a synchronous call within the picker's dedicated thread. The picker thread communicates with the allocator thread via `jbuf` cbuf (existing pattern). `pthread_cancel` cancellation points shall exist in the `next_job()` iteration loop and in `cbuf_get(&wbuf)` where the picker blocks between sessions. + +6. **Session state** — `end_session()` is a no-op. Iteration state is reset in `begin_session()`. Undispatched jobs remain in `pendJobs` across sessions (they carry over to the next round). + +### Requirement 4: No regression to existing picker behavior or HA soak + +**Objective:** As a maintainer, I want the picker rework and core/driver separation scoped so that existing FCFS behavior is unaffected, making the framework change safe to ship independently of any new policy. + +#### Acceptance Criteria +1. `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`, given the same queue and job state — same inter-queue priority ordering and same intra-queue (priority desc, queue_time asc) ordering. +2. Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string and the refactored `sched_go()` / decision publishing path this feature replaces. The picker thread, allocator thread, and cbuf communication pattern are preserved. +3. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. + +### Requirement 5: Core/Driver Separation with Dual-Mode Operation + +**Objective:** As a scheduler developer, I want the scheduling core separated from all external I/O so that the same core logic runs deterministically in production (driven by a live controller) and in simulation (driven by recorded cluster events with a discrete event-driven virtual clock), enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. + +#### Acceptance Criteria + +**Driver ↔ Core Protocol (overview)** + +The driver and core communicate through a three-call synchronous interface with cbuf-based internal threading. The driver buffers controller events externally between sessions, pushes them as a batch before each scheduling round, and receives decisions one-by-one as they are produced. + +| Direction | Call | Semantics | Thread | +|-----------|------|-----------|--------| +| Driver → Core | `core_push_events(core, events, n)` | Batch-push events before session. State-change hooks called synchronously. Only valid when picker is idle (between sessions). | Driver thread | +| Driver → Core | `core_go(core)` | Start scheduling session. **NON-BLOCKING** — signals picker thread via cbuf, returns immediately. Returns `-EINVAL` if session already active. | Driver thread | +| Core → Driver | `on_decision(ctx, decision)` | Each scheduling decision emitted immediately as produced. Called from allocator thread context. | Allocator thread | +| (implicit) | picker sends `jobid=0` through `jbuf` | End-of-session marker. Allocator forwards as `NULL` through `decisions` cbuf. | Picker → Allocator → Driver | + +**Event buffering discipline**: The driver accumulates controller events (capnp push/notify messages from the handler thread) into `pend_updates` (existing dataq swap mechanism). Between sessions, the driver drains `pend_updates` into `xce_core_event_t` array and calls `core_push_events()`. No events are injected mid-session — the picker thread is blocked on `cbuf_get(&wbuf)` and the allocator thread on `cbuf_get(&jbuf)`, so `core_push_events()` has exclusive access to picker state. + +**Decision publishing**: Each decision is published immediately to the controller (not batched at session end). The driver's `on_decision` callback calls `xcecli_jobdispatch()` / `xcecli_set_pend_reason()` / `xcecli_preempt()` synchronously. This matches today's `main.c:181-196` behavior. + +**Mode selection**: Mode is chosen at startup via CLI flags. The core has zero knowledge of which mode is active: + +```c +// main.c — mode selection (single branch) +if (arguments.sim_file) { + sched_sim_driver_run(&sim_driver, arguments); +} else { + sched_driver_run(&driver, &config, cluster_cfg, handler); +} +``` + +| Flag | Mode | Purpose | +|------|------|---------| +| (none) | Normal | Controller-connected (current behavior) | +| `--sim FILE` | Simulation | Replay `.replay` file | +| `--sim-speed N` | Simulation | 0=fast, 1=real-time, N=N× | +| `--sim-start-ts NS` | Simulation | Start replay from logical time | +| `--sim-verify` | Simulation | Compare decisions against recorded | +| `--interactive` / `-i` | Simulation | Open REPL for step-through debugging | +| `--sim-alt-events FILE` | Simulation | What-if branching | +| `--sim-output FILE` | Simulation | Write simulated decisions to `.replay` file | + +**Migration from old `job_elect_fn`**: The existing `xce_sched_plugin_t->job_elect_fn` (which `fcfs_elect_job` implements) is superseded by `xce_job_picker_iface_t->next_job()`. The old `job_elect_fn` shall be deprecated with a warning at plugin register time. The new `builtin_fcfs` picker implements the new interface exclusively. Existing plugins that only implement `job_elect_fn` continue to work via a compatibility wrapper during the transition period, but new pickers shall implement `xce_job_picker_iface_t`. + +--- + +**Core Interface Contract** + +1. **Mode-agnostic core**: The Scheduler core shall have no concept of which mode it runs in. It processes events identically whether the driver is in normal mode (controller-connected, wall-clock time) or simulation mode (replay file, virtual clock). Specifically: + - There shall be no mode flag (`bool simulation_mode` or equivalent) anywhere in the core + - There shall be no `if (simulation) { ... } else { ... }` code path in the core + - `xce_sched_core_init()` shall take identical parameters regardless of mode + - The core's decision callback shall be the same function pointer type in both modes + - The core's `save_snapshot()` and `load_snapshot()` shall be identical in both modes + +2. **Passive event-driven interface**: The core shall expose a pure event-driven interface — it receives typed events from the driver, updates internal scheduling state, and emits decisions through a driver-registered callback. The core shall have zero direct dependency on: controller communication, capnp serialization, network I/O, filesystem access, or wall-clock time sources. + +3. **Virtualized clock access**: The core shall access time exclusively through `xce_core_now()`. The core shall never call `time(NULL)`, `clock_gettime()`, `gettimeofday()`, or any other wall-clock time function directly. + + **Clock virtualization mechanism — two-base + speed formula:** + + The core stores four fields (atomically updated by the driver as a group): + + | Field | Type | Description | + |-------|------|-------------| + | `virtual_clock_base` | `_Atomic uint64_t` | Virtual-time anchor (nanoseconds) | + | `real_clock_base` | `_Atomic uint64_t` | Wall-clock anchor captured when `virtual_clock_base` was set (nanoseconds, `CLOCK_MONOTONIC`) | + | `speed_num` / `speed_den` | `_Atomic uint64_t` | Speed multiplier as rational fraction | + + `xce_core_now(core)` computes: + ```c + uint64_t wall_ns = clock_gettime(CLOCK_MONOTONIC); // only wall-clock call in the core + uint64_t elapsed = wall_ns - core->real_clock_base; + return core->virtual_clock_base + (elapsed * core->speed_num) / core->speed_den; + ``` + + | Mode | `virtual_clock_base` | `real_clock_base` | `speed` | Behavior | + |------|----------------------|-------------------|---------|----------| + | Normal | 0 | 0 | 1/1 | `xce_core_now()` = wall clock (degraded: `0 + (wall - 0) × 1`) | + | Simulation discrete | event `ts` | wall at load | 0/1 | `xce_core_now()` = frozen at event `ts` (time does not advance between events) | + | Simulation real-time | snapshot `ts` | wall at load | 1/1 | `xce_core_now()` tracks wall clock from virtual anchor | + | Simulation fast/slow | snapshot `ts` | wall at load | N/1 or 1/N | `xce_core_now()` runs N× faster or slower than wall clock | + + The driver atomically updates all four fields before each event dispatch. **Multi-thread safety**: `xce_core_now()` reads atomic fields lock-free. Both picker and allocator threads independently compute the same virtual time from the shared bases — no mutex needed. + + All internal time uses go through `xce_core_now()`: scheduling logic, logging (via tlog format function using `userptr`), error messages, debug traces. + Job `sub_time` and `queue_time` are set by the driver before pushing `JOBS_PUSH`. + + **Third-party library constraint**: any library linked into the core that needs time must obtain it through `xce_core_now()` or a core-provided wrapper. Specifically: + - **Logging (tlog)**: The core shall register its own format function via `tlog_reg_format_func(fn, core_ptr)` where `core_ptr` is the `xce_sched_core_t*`. This format function shall ignore `info->time` and instead read `xce_core_now(core_ptr)`, converting nanoseconds to `struct tm` fields via `gmtime_r()` (deterministic — pure computation). + - **libxce subset**: The core shall link only against the minimal data-structure subset of libxce (`ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti`) — not against `netio`, `delay`, `xceapi`, or `idgen` which call wall-clock time internally. + - **pthread timed-blocking**: The core is multi-threaded. In simulation mode, thread interleaving is made deterministic — `pthread_cond_timedwait()` and `pthread_mutex_timedlock()` are not needed in the deterministic execution path. + - Libraries that cannot avoid calling wall-clock time directly must not be linked into the core; such concerns stay in the driver layer. + +4. **Core event types — detailed payload specification**: The core interface shall accept at minimum these input events. Each event carries a timestamp (`ts`, nanoseconds since epoch) and a type-specific payload. The field set below is normative for implementation; the exact capnp message schema is deferred to the design phase. + + **`CONFIG_PUSH`** — Full configuration push (scheduler startup or config reload): + | Field | Type | Description | + |-------|------|-------------| + | `config` | `xce_conf4sched_t*` | All queues, resource thresholds, preemption config, time windows, global policy settings | + | → Picker hook: `on_config(picker, config)` | | | + + **`JOBS_PUSH`** — Newly submitted jobs: + | Field | Type | Description | + |-------|------|-------------| + | `njobs` | `int` | Number of jobs in this batch | + | `jobs` | `xce_job_t*[njobs]` | Job records: jobid, user, project, queue, priority, state, queue_time, resreq | + | → Picker hook: `on_job_push(picker, job)` for each job | | | + + **`WORKERS_PUSH`** — Worker list update (nodes added/removed): + | Field | Type | Description | + |-------|------|-------------| + | `nworkers` | `int` | Number of workers in this update | + | `workers` | `xce_worker_t*[nworkers]` | Worker records: workerid, host, resources, state, assigned_jobs | + + **`RESOURCES_PUSH`** — Resource definitions (new resource types registered): + | Field | Type | Description | + |-------|------|-------------| + | `n` | `int` | Number of resource definitions | + | `resources` | `xce_resource_t*[n]` | Resource records: name, type, total, flags | + + **`RESVALS_FULL`** — Full resource value snapshot (periodic or after large change): + | Field | Type | Description | + |-------|------|-------------| + | `n` | `int` | Number of resource value entries | + | `resvals` | `xce_resval_t*[n]` | Each: workerid, resource_name, available, used, reserved | + + **`RESVALS_DELTA`** — Incremental resource value update (fine-grained changes): + | Field | Type | Description | + |-------|------|-------------| + | `deltas` | `xce_resval_list_t*` | Linked list of individual resval changes (add/subtract) | + + **`JOB_STATE_CHANGE`** — Job started, exited, or failed: + | Field | Type | Description | + |-------|------|-------------| + | `jobid` | `uint64_t` | Job identifier | + | `old_state` | `uint32_t` | Previous job state (for context) | + | `new_state` | `uint32_t` | New job state: RUNNING, EXIT, FAIL, DONE, PEND | + | `exit_code` | `int32_t` | Exit code (meaningful only for EXIT/FAIL) | + | → Picker hook: `on_job_state_change(picker, job, new_state)` | | | + + **`JOB_UPDATE`** — In-place job field modification (priority change, queue reassignment): + | Field | Type | Description | + |-------|------|-------------| + | `jobid` | `uint64_t` | Job identifier | + | `changed_fields` | `uint32_t` | Bitmask of fields that changed (JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ) | + | `job` | `xce_job_t*` | Full job record with updated fields | + | → Picker hook: `on_job_update(picker, job)` | | | + + **`QUEUE_UPDATE`** — Queue definition added or modified: + | Field | Type | Description | + |-------|------|-------------| + | `queue` | `xce_queue_t*` | Queue record: name, priority, RUNLIMIT, MAX_RUNNING, state, time_window, job_pick_plugin, project_name | + | → Picker hook: `on_queue_update(picker, queue)` | | | + + **`JOB_RUSAGE`** — Job resource usage update (for resource-aware pickers): + | Field | Type | Description | + |-------|------|-------------| + | `jobid` | `uint64_t` | Job identifier | + | `rusages` | `xce_job_rusages_t*` | Resource usage metrics: cpu_time, mem_bytes, io_bytes | + | → Picker hook: `on_job_rusage(picker, job, rusages)` (optional — NULL if not registered) | | | + + **`BEGIN_ROUND`** — Trigger a scheduling round (session boundary, see AC 18): + | Field | Type | Description | + |-------|------|-------------| + | `timestamp_ns` | `uint64_t` | Logical time at which the round should execute | + + **`END_SESSION`** — Explicit end-of-recording marker (written to replay file, never sent to core): + | Field | Type | Description | + |-------|------|-------------| + | `timestamp_ns` | `uint64_t` | Logical time when recording stopped | + +5. **Decision output types**: The core shall emit decisions through the `on_decision` callback. Three decision types exist: + + | Decision Type | Fields | Driver Action | + |---------------|--------|---------------| + | `DISPATCH` | `jobid`, `workerid`, `allocations[]` (resource, amount per allocation) | `xcecli_jobdispatch()` | + | `PEND` | `jobid`, `reason` (code + message) | `xcecli_set_pend_reason()` | + | `PREEMPT` | `jobid`, `victims[]` (jobid, workerid per victim), `reason` | `xcecli_preempt()` | + + These are recorded as `ReplayDecision` messages within `ReplaySession.decisions`. The exact decision payload schema is deferred to the design phase. + +6. **Decision output callback**: The core shall emit all scheduling decisions (`DISPATCH`, `PEND`, `PREEMPT`) through a `xce_core_decision_cb_t` callback registered by the driver at init time. Signature: `int (*on_decision)(void* driver_ctx, xce_decision_t* decision)`. This replaces the current internal `decisions` cbuf poll loop in `main.c`. + +7. **Deterministic guarantee**: Given identical initial state and identical event sequences with identical timestamps, the core shall produce identical decision sequences. The core is multi-threaded (picker + allocator threads communicating via cbuf). Deterministic replay is achieved through: + - **cbuf capacity=1 serialization**: The `jbuf` cbuf between picker and allocator has capacity 1 (`cbuf_init(&jbuf, 1, sizeof(uint64_t))`). This acts as a rendezvous — the picker blocks on `cbuf_put` until the allocator takes the current jobid. Jobs flow through the pipeline one at a time in fully determined order. + - **Stabilized hash table iteration**: All hash table / key-value store iteration that affects scheduling output order shall be stabilized (sorted by key before iterating). + - **PRNG for tie-breaking**: Any required randomness (e.g., tie-breaking) shall use a PRNG seeded by the driver at init time (`core->prng_seed`). + - **Same events → same picker state**: Since `core_push_events()` is called between sessions (picker idle), the picker's internal data structures (`ordq_t queues`, `ordq_t pendJobs`) are populated identically given identical event sequences. The picker iteration then produces jobids in identical order. + + **7.1 Normal-mode ↔ simulation equivalence**: Given identical initial configuration (`xce_conf4sched_t`) and identical event sequences (same job submissions, state changes, resource updates at the same logical positions), the core shall produce identical decision sequences in both normal mode and simulation mode. Specifically: + + - Record a normal-mode scheduling session (AC 10) → produce `.replay` file R + - Run simulation on R (AC 13) → produce `.replay` file S + - `diff <(xceadmin scheduler dump R) <(xceadmin scheduler dump S)` → only non-decision metadata may differ (timestamps in `ReplayHeader`, wall-clock annotations in `SessionSummary`). Every `ReplayDecision` shall match identically: same jobid, same workerid, same allocation amounts, same pend reasons. + + This constraint is the **validation contract** for the entire core/driver architecture. If normal mode and simulation mode diverge, one of three things is broken: + - The core has a hidden dependency on wall-clock time or controller I/O (violates AC 1) + - The event recording is lossy — dropping events that matter to scheduling decisions (violates AC 10) + - The simulation driver's input translation from capnp or `.replay` format is not faithfully reproducing the normal-mode event stream + + This constraint turns recording + replay from a debugging convenience into a **regression test**: after any scheduler change, record a live run, replay it in simulation, and verify the decisions are byte-for-byte identical. Any divergence is a bug. + +8. **Driver event queue**: The driver shall retain an internal pending-event queue to buffer events received from the controller between scheduling rounds. Events accumulate in the driver between `BEGIN_ROUND` invocations. + +**Normal-Mode Driver** + +9. **Controller integration**: The normal-mode driver shall receive capnp push/notify messages from the controller, translate each to the corresponding core event type, and invoke the core's event push interface. The driver shall publish core-emitted decisions back to the controller via the existing `xcecli_ctx_t` API. + +10. **Runtime recording control — detailed mechanism**: The scheduler shall support runtime recording of all scheduling events and decisions to a replay file for later simulation and analysis. + + **Admin command flow**: xceadmin CLI connects directly to the controller via TLS + capnp (same as `xceadmin config push` — no apiserver, no REST). The controller receives `OPCode_schedAdmin` and forwards it to the scheduler via the existing `scheduler_t.ep` endpoint. The scheduler processes the request and replies; the controller relays the reply back to xceadmin. This is a request-reply opcode, NOT a one-way notify — recording `status` returns live scheduler state that the controller does not own. + + **Why request-reply (not notify)**: The existing `OPCode_notify` is fire-and-forget — no reply path. Recording `start`/`stop` could theoretically use notify, but `status` must return live state (`recording`, `file`, `event_count`, `started_at`). Using two different mechanisms for three commands on the same subsystem is worse than one consistent request-reply path. Additionally, `start`/`stop` benefit from a synchronous reply: `start` returns the auto-generated filename, and `stop` returns the final event count. + + **Why a dedicated opcode (not reusing OP_upgradeConfig)**: `OP_upgradeConfig` carries a config blob through the controller's own config-replication pipeline. Recording control is a scheduler-scoped admin action — it touches no controller state, needs no WAL persistence, and should be forwarded directly to the scheduler without controller-side interpretation. A dedicated `OPCode_schedAdmin` keeps these concerns separate. + + **Cap'n Proto schema additions** (in `controller.capnp`): + + New OPCode: + ```capnp + enum OPCode { + ... + schedAdmin @42; + } + ``` + + New types: + ```capnp + enum SchedAdminAction { + recordStart @0; + recordStop @1; + recordStatus @2; + } + + struct SchedAdminRequest $C.mapname("sched_admin_req_t") { + action @0: SchedAdminAction; + recordFile @1: Text; # path for recordStart; empty = auto-generate + } + + struct SchedAdminReply $C.mapname("sched_admin_reply_t") { + ok @0: Bool; + error @1: Text; # human-readable error if !ok + + # Fields populated for recordStatus (also returned by recordStart/recordStop + # for confirmation — caller can verify state after each action): + recording @2: Bool; # true between start and stop + file @3: Text; # record file path (auto-generated name returned by start) + eventCount @4: UInt64; # events written since recording started + startedAt @5: UInt64; # logical time recording started (nanoseconds since epoch) + } + ``` + + New Request.body union entry: + ```capnp + schedadminmsg @31: SchedAdminRequest; + ``` + + New RequestReply.data union entry: + ```capnp + schedadminreplymsg @19: SchedAdminReply; + ``` + + **Controller forwarding** (in `myreq_handler`): + ``` + case OPCode_schedAdmin: + → validate xceadmin auth (admin RBAC, same as config push) + → lock(&c->sched.lock) + → if sched.ep == NULL: + unlock; reply {ok=false, error="scheduler not connected"} + → forward to sched.ep via existing capnp push/request + → await scheduler reply (timeout: 5s) + → unlock + → relay SchedAdminReply back to xceadmin via RequestReply + ``` + + The controller is a **pass-through** — it owns no recording state, interprets no fields beyond the opcode dispatch. All recording logic lives in the scheduler's normal-mode driver. In the new architecture, recording is a driver-layer concern — the core has zero knowledge of whether recording is active. + + **Edge cases**: + - **Scheduler not connected** (`sched.ep == NULL`): Controller replies `{ok=false, error="scheduler not connected"}` immediately. No capnp round-trip needed. + - **Scheduler disconnects mid-request**: The 5s timeout triggers; controller replies `{ok=false, error="request timeout"}`. + - **Controller is not primary**: xceadmin's existing redirect-following logic handles this — if the contacted controller is not the primary, it redirects to the primary (same as `config push`). The scheduler is a singleton attached to the primary controller, so recording commands always reach the primary's scheduler. + + **xceadmin CLI surface**: + ``` + xceadmin scheduler record start [FILE] + → SchedAdminRequest{action=recordStart, recordFile=FILE or ""} + → Success: {ok=true, file="", recording=true, + eventCount=0, startedAt=} + → Error: {ok=false, error="already recording", recording=true, file=""} + + xceadmin scheduler record stop + → SchedAdminRequest{action=recordStop} + → Success: {ok=true, recording=false, eventCount=14273, file="", + startedAt=} + → Error: {ok=false, error="not recording", recording=false} + + xceadmin scheduler record status + → SchedAdminRequest{action=recordStatus} + → Success: {ok=true, recording=true|false, file="", + eventCount=N, startedAt=} + ``` + + **Reply field semantics by action**: + | Field | recordStart (ok) | recordStart (err) | recordStop (ok) | recordStop (err) | recordStatus | + |-------|-------------------|--------------------|------------------|--------------------|--------------| + | `ok` | true | false | true | false | true | + | `error` | "" | reason (e.g. "already recording") | "" | reason (e.g. "not recording") | "" | + | `recording` | true | true (unchanged) | false | false (unchanged) | true/false | + | `file` | current path | current path | last path (informational) | "" | current or "" | + | `eventCount` | 0 | current count | final count | 0 | current | + | `startedAt` | start time | original start | original start (historical) | 0 | original or 0 | + + **Error codes reference**: + | Condition | `ok` | `error` | + |-----------|------|---------| + | `record start` while already recording | false | `"already recording"` | + | `record stop` while not recording | false | `"not recording"` | + | Scheduler not connected to this controller | false | `"scheduler not connected"` | + | Request timeout (5s) | false | `"request timeout"` | + | Disk full / write error (start) | false | `"cannot write to : "` | + + Status output (xceadmin formats the capnp reply): + ``` + recording: yes + file: /var/log/xce/scheduler-replay-prod-cluster-1-20260729-020900.replay + events: 14273 + started at: 2026-07-29T02:09:00 + ``` + + **Permissions**: All three commands require admin-level authentication (same RBAC role as `config push`). The controller validates the xceadmin auth before forwarding to the scheduler — the scheduler trusts the controller's forwarding (same trust model as existing notify events from the controller). + + **Recording state** (in `xce_sched_driver_t`): + ```c + bool recording; // true between start and stop + char* record_file_path; // strdup'd path for status queries + FILE* record_file; // replay file handle (NULL when not recording) + uint64_t record_event_count; // counter for periodic snapshot + uint64_t record_snapshot_interval; // N events between snapshots (0 = disabled) + uint64_t record_start_ts; // logical time when recording started (0 = not started) + ``` + + `record_file_path` is a `strdup`'d copy of the auto-generated or user-specified path, owned by the record file lifecycle: `strdup` on start, `free`+NULL on stop or error. This field exists so `SchedAdminReply.file` can be populated even after `record_file` is closed — it persists for informational queries (e.g., `status` after a disk-full auto-stop). + + **Recording lifecycle — what gets written at each step:** + + 1. **On `record start`**: + - Take full core snapshot via `save_snapshot(core, &snapshot_data)` — fills `SnapshotData` struct in memory + - Write `ReplayHeader` as first message of the file (recorder version, config hash, cluster identity, timestamps, sched policy) + - Serialize `SnapshotEntry{ts, data=snapshot_data}` and write as second message + - Set `recording = true`, `record_event_count = 0` + + 2. **On each `core_push_events()` call** (driver thread, between sessions): + - For each event in the batch, serialize and write the corresponding `ReplayEvent` message to the `.replay` file + - Increment `record_event_count` for each event written + + 3. **On `core_go()`** (driver thread, start of session): + - Begin building a `ReplaySession` message with `beginRound = xce_core_now(core)` + + 4. **In the decision drain loop** (driver thread, during session): + - For each decision from `cbuf_get(&decisions)`: append a `ReplayDecision` to the current `ReplaySession.decisions` list + - Decisions are in allocator output order, before the NULL end-of-session marker + + 5. **On `core_end_session()`** (driver thread): + - Finalize the `ReplaySession` with session summary, write the complete message to the `.replay` file + + 6. **On periodic checkpoint** (driver thread): + - If `record_event_count > 0 && record_event_count % record_snapshot_interval == 0`: + - Serialize and write `SnapshotEntry` message from `save_snapshot(core, &snapshot_data)` + + 7. **On `record stop`**: + - Write `ReplayEvent{endSession}` message + - `fflush()` + `fclose()` the record file + - Set `recording = false`, `record_file = NULL` + + **Recording insert points in the normal-mode driver main loop** (annotated): + ```c + while (!sched.stop) { + sched_update(&sched); // process capnp push/notify + // [RECORDING] write each buffered event as ReplayEvent message + // [RECORDING] periodic checkpoint check + + sched_begin_session(&sched); + // [RECORDING] build ReplaySession{beginRound = now} + sched_go(&sched); + + while (cbuf_get(&sched.decisions, &d), d) { + // [RECORDING] append ReplayDecision to session + switch (d->op) { + case DISPATCH: sched_publish_dispatch(...); break; + case PEND: sched_publish_pend_reason(...); break; + case PREEMPT: sched_publish_preempt(...); break; + } + } + + sched_end_session(&sched); + // [RECORDING] finalize + write ReplaySession message + // [RECORDING] periodic checkpoint after session + sleep(1); + } + ``` + + **Performance**: When recording is inactive (`recording == false`), the scheduler shall incur zero overhead — no file I/O, no extra allocations, no event buffering beyond what the non-recording path already does. The recording check is a single boolean branch per session. + + **Error handling**: If writing to the record file fails (disk full, permission error), the driver shall log the error, set `recording = false`, `fflush()` + `fclose()` the record file, `free(record_file_path)` + set `record_file_path = NULL`, and continue normal scheduling operation. Recording failure shall not cause the scheduler to stop or crash. + +11. **Periodic snapshot during recording**: While recording is active, the driver shall periodically append fresh snapshots to the replay file (default: every `N` events, configurable via `--record-snapshot-interval N`, N=0 disables periodic snapshots). + +12. **Session response time metrics**: The core shall track session duration for diagnostics. At `core_go()`, the core stores `session_start = xce_core_now()`. At `core_end_session()`, the core computes `session_duration_ns = xce_core_now() - session_start`. The duration is written to the `ReplaySession.summary` field as a `SessionSummary` struct. The `SessionSummary` is optional for replay correctness (the simulation driver ignores it) but provides valuable diagnostics: + - **Normal mode**: reveals real scheduling latency — is the picker slow? is allocation the bottleneck? + - **Simulation mode**: in real-time mode, compare virtual vs wall-clock session duration; in fast mode, the virtual clock freezes during `core_go()` so `duration_ns` is typically 0 (all decisions at same `ts`) + - **Tuning**: diff session durations before and after a policy change to quantify performance impact + + The core stores `session_start` internally — no new API functions, no change to the `on_decision` callback, no overhead beyond a single `xce_core_now()` call per session. + +**Simulation-Mode Driver** + +13. **Simulation mode startup**: The scheduler shall support simulation mode activated by `--sim FILE` and `--sim-start-ts TIMESTAMP`. The startup sequence is: + + **Step 1 — Open and validate replay file:** + ``` + sim->replay_file = fopen(arguments.sim_file, "r"); + read first capnp message → validate it is ReplayHeader + ``` + If the file is missing, unreadable, or the first message is not a valid `ReplayHeader`, the simulation driver shall fail with a clear error message. + + **Step 2 — Locate starting snapshot:** + ``` + scan forward from ReplayHeader, decode each SnapshotEntry + find the most recent SnapshotEntry with ts <= --sim-start-ts + ``` + If no `SnapshotEntry` exists after `ReplayHeader`, the simulation driver shall reject the file. Without initial state, simulation has no starting point. + + **Step 3 — Load snapshot and rebuild picker:** + ``` + xce_sched_core_load_snapshot(&sim->core, snapshot_entry.data) + // Walk core state to rebuild picker private state: + for each queue in core.queues: + picker->on_queue_update(picker, queue) + for each job in core.jobs where job.state & JOB_STAT_PEND: + picker->on_job_push(picker, job) + ``` + After this step, core state and picker state are fully restored. The core is ready to run. + + **Step 4 — Set virtual clock:** + ``` + sim->core.virtual_clock_base = snapshot_ts + sim->core.real_clock_base = clock_gettime(CLOCK_MONOTONIC) // wall anchor + sim->core.speed_num = arguments.sim_speed // 0=fast, 1=real-time, N>1=N× + sim->core.speed_den = 1 + ``` + From this point, `xce_core_now()` returns virtual time computed from the snapshot's logical timestamp. + + **Step 5 — Enter event loop (AC 18):** accumulate events → BEGIN_ROUND → `core_push_events` → `core_go` → drain decisions → `core_end_session` → repeat until EOF. + + The `ReplayHeader` + `SnapshotEntry` together form the complete self-contained starting point. Nothing outside the replay file is needed to begin simulation. + +14. **Discrete event-driven virtual clock**: The simulation driver shall implement a virtual clock that advances by jumping directly to the timestamp of the next queued event — no fixed tick interval, no wall-clock waiting. Strictly monotonically increasing. Same-timestamp events delivered in file order. Supports: + - `--sim-speed N` — N=0 as-fast-as-possible, N=1 real-time, N>1 N× real-time + - `--interactive` / `-i` — open REPL for step-through debugging, breakpoints, state inspection, and snapshot capture (AC 22) + +15. **Deterministic replay verification**: In simulation mode with `--sim-verify`, the driver shall compare each core-emitted decision against the recorded decisions in the replay file. Any mismatch shall be reported as a regression with exact logical-time and decision-detail context. + +16. **What-if branching**: The simulation driver shall accept `--sim-alt-events FILE`. When both a base `.replay` file (for snapshot loading) and an alt-events file are provided, the driver loads the snapshot from the base file, then replays events from the alt-events file (same `.replay` capnp stream format) — enabling "what-if" analysis. + +17. **Simulation isolation**: In simulation mode, the scheduler process shall not establish any connection to the controller. The `xce_sched_sim_driver_t` shall contain no `xcecli_ctx_t`, no `xce_cluster_config_t`, no capnp socket, and no RPC to any controller. All scheduling input comes exclusively from the replay file. All scheduling decisions go to the simulation record file (see AC 19) and/or stdout. Attempting to use a controller-facing API from simulation mode is a programming error that shall be caught at compile time (via struct field absence) or fail with a clear error at startup. + +18. **BEGIN_ROUND as session boundary**: The simulation driver shall treat each `ReplaySession` message as the trigger for one complete scheduling session. The driver accumulates `ReplayEvent` messages into a batch, and upon encountering `ReplaySession`, executes the full session cycle: + + ```c + // Simulation driver event loop + xce_core_event_t buf_events[MAX_EVENTS_PER_ROUND]; + int buf_n = 0; + + while (decode_next_message(replay_file, &msg) == 0) { + switch (msg.which) { + case ReplayEvent: + buf_events[buf_n++] = replay_event_to_core_event(&msg.event); + break; + + case ReplaySession: // ← SESSION BOUNDARY + // 1. Push accumulated events (picker hooks called synchronously) + core_push_events(&sim->core, buf_events, buf_n); + + // 2. Kick scheduling round (non-blocking signal to picker thread) + core_go(&sim->core); + + // 3. Drain decisions until end-of-session marker (NULL) + xce_decision_t* d; + int di = 0; + while (cbuf_get(&sim->core.decisions, &d) == 0 && d != NULL) { + // 4. Compare against recorded decisions (if verify enabled) + if (sim->verify && di < msg.session.decisions.len) { + xce_decision_t expected = from_replay_decision(msg.session.decisions[di]); + if (!decision_eq(&expected, d)) + log_mismatch(sim, &expected, d); // --sim-verify + di++; + } + if (sim->record_file) + write_replay_decision(sim->record_file, d); // record output + free_decision(d); + } + + // 5. End session (GC, reset workers) + core_end_session(&sim->core); + buf_n = 0; // reset for next round + break; + + case SnapshotEntry: + // Mid-replay snapshots (periodic checkpoints): skipped during replay. + // Only the initial SnapshotEntry (loaded at startup) is consumed. + break; + } + } + ``` + + The driver thread blocks during the decision drain loop (step 3-4) because `cbuf_get(&decisions)` blocks until the allocator produces a decision or the end-of-session marker. The `core_go()` call itself is non-blocking (it only signals the picker thread via `cbuf_put(&wbuf)` and returns immediately). After all decisions are drained and the NULL marker arrives, `core_end_session()` runs cleanup, and the driver resumes reading the next message from the `.replay` file. + + **Decision ordering constraint**: `ReplayDecision` messages appear within the `ReplaySession.decisions` list. Their order matches the allocator thread's output order through the `decisions` cbuf. A `ReplayDecision` appearing outside a `ReplaySession` is a format error. + +19. **Simulation record file**: In simulation mode with `--sim-output FILE`, the driver shall write a `.replay` file in the identical capnp stream format as the input replay file (`ReplayHeader` → `SnapshotEntry` → `ReplayEvent` → `ReplaySession` cycles). This record file captures the simulation's actual decisions, enabling: + + - **Diff-based debugging**: `diff <(xceadmin scheduler dump original.replay) <(xceadmin scheduler dump simulated.replay)` reveals exactly which decisions changed + - **Tuning feedback loop**: Modify picker parameters, re-run simulation, diff the record files to quantify the impact + - **What-if comparison**: Compare `--sim-alt-events` run output against the base run output + + The record file starts with a `ReplayHeader` (carrying scheduler version + config hash from the sim run, NOT copied from the replay file), followed by the same `SnapshotEntry` that was loaded, then events + `ReplaySession` cycles exactly as produced by the core. If `--sim-output` is not specified, decisions are printed to stdout only and no record file is written. + + **Symmetry with normal-mode recording**: Both the normal-mode driver (when recording is active, AC 10) and the simulation-mode driver (with `--sim-output`) produce `.replay` files in the identical capnp stream format. The same file layout, same message types, same decision encoding. This symmetry is intentional: + + | Aspect | Normal-mode recording | Simulation `--sim-output` | + |--------|----------------------|---------------------------| + | Trigger | `scheduler record start` (runtime) | `--sim-output FILE` (CLI at startup) | + | Input source | Controller capnp events | Replay file events | + | Writes events? | Yes — records all input events as `ReplayEvent` messages | Yes — passes through events from replay file as `ReplayEvent` messages | + | Writes decisions? | Yes — records scheduler decisions in `ReplaySession` messages | Yes — records simulated decisions in `ReplaySession` messages | + | Output file | `.replay` file (for later simulation) | `.replay` file (for diff analysis) | + | `ReplayHeader` origin | Live cluster metadata | Sim run metadata | + | `SnapshotEntry` origin | `save_snapshot(core)` at record start | `SnapshotEntry` from replay file | + | Purpose | Capture reality | Capture simulation result | + + This means `diff <(xceadmin scheduler dump original.replay) <(xceadmin scheduler dump simulated.replay)` works out of the box — both files have the same structure, same timestamps, same event types. Only the decisions may differ, which is exactly what you want to compare. + +**Replay File Format** + +20. **Single-file capnp stream format (`.replay`)**: The replay file shall be a single file containing consecutive Cap'n Proto messages in the standard framing (segment table + data per message). Each message is one of the types below. The file is appendable — recording writes each message as it occurs; no need to know final size upfront. + + **File suffix**: `.replay` — self-documenting, no collision with schema files (`.capnp`), configuration blobs (`.bin`), or text formats (`.json`, `.jsonl`). + + **Schema** (in `controller.capnp`): + + ```capnp + struct ReplayHeader { + recorderVersion @0: Text; # scheduler binary version that wrote this file + configHash @1: Data; # sha256 of xce_conf4sched_t + cluster @2: ReplayCluster; + recordingStartTs @3: UInt64; # logical time recording started (ns) + recordingStartWall @4: Text; # wall-clock time recording started (ISO 8601) + schedPolicy @5: Text; # active picker plugin name (e.g. "builtin_fcfs") + } + + struct ReplayCluster { + name @0: Text; + controllers @1: List(Text); # ["host1:9991", "host2:9992", "host3:9993"] + recordingHost @2: Text; # controller hostname that made the recording + } + + struct SnapshotEntry { + ts @0: UInt64; # logical time of snapshot + data @1: SnapshotData; # full core state + } + + struct ReplayEvent { + ts @0: UInt64; # logical time + body @1: union { + configPush @2: Conf4Sched; + jobsPush @3: SchedJobData; + workersPush @4: WorkerData; + resourcesPush @5: ResourceData; + resvalsFull @6: ResValList; + resvalsDelta @7: ResValList; + jobStateChange @8: JobStateChangeEvent; + jobUpdate @9: JobUpdateEvent; + queueUpdate @10: QueueConf; + jobRusage @11: SchedJobRusages; + endSession @12: Void; # recording stop marker + } + } + + struct ReplaySession { + beginRound @0: UInt64; # logical time (ts value) + decisions @1: List(ReplayDecision); + summary @2: SessionSummary; # optional diagnostic metadata + } + + struct ReplayDecision { + ts @0: UInt64; + body @1: union { + dispatch @2: DispatchDecision; + pend @3: PendDecision; + preempt @4: PreemptDecision; + } + } + + struct SessionSummary { + durationNs @0: UInt64; + nDecisions @1: UInt32; + nDispatched @2: UInt32; + nPended @3: UInt32; + } + ``` + + **File structure**: The `.replay` file is a single stream of consecutive capnp messages: + + ``` + [msg 1] ReplayHeader + [msg 2] SnapshotEntry ← initial state (MUST be present, MUST be first after header) + [msg 3] ReplayEvent ← input event + [msg 4] ReplayEvent ← input event + ... + [msg N] ReplaySession ← BEGIN_ROUND → decisions → SESSION_SUMMARY + [msg N+1] ReplayEvent + ... + [msg M] SnapshotEntry ← periodic checkpoint (every N events, AC 11) + ... + [msg K] ReplayEvent{endSession} ← recording stop marker + ``` + + - **`ReplayHeader`**: first message. Identifies the file type (`.replay` files start with a known capnp struct, not an ambiguous byte sequence). Carries metadata about the recording origin — recorder version, config hash, cluster identity, timestamps, active scheduling policy. + - **The `.replay` file is fully self-contained**: the `ReplayHeader` identifies which config was active (`configHash`), and the first `SnapshotEntry` carries the full config payload plus all scheduling state. No external config file, controller, or database is needed to replay — one file has everything. + - **`SnapshotEntry`**: the first data message after `ReplayHeader` MUST be a `SnapshotEntry`. The simulation driver shall reject any `.replay` file whose second message is not a `SnapshotEntry` — without initial state, simulation has no starting point. Mid-stream `SnapshotEntry` messages (from periodic recording, AC 11) provide seek points for `--sim-start-ts`. + - **`ReplayEvent`**: input event carrying payload in a discriminated union. The `body` union tag selects the event type (`configPush`, `jobsPush`, etc.). No separate `"type"` field needed — the capnp union tag is the type discriminator. + - **`ReplaySession`**: groups a `BEGIN_ROUND` trigger with its resulting decisions and session summary. A `ReplaySession` between `ReplayEvent` messages marks a session boundary; a `ReplayDecision` outside a `ReplaySession` is a format error. Decision order within `decisions` matches allocator output order. + - **`ReplayEvent{endSession}`**: final message in a completed recording. Signals clean shutdown. The simulation driver ignores it (replay runs until EOF). + - No text parsing, no JSON schema, no two-format code paths — the same capnp decode path reads every message. The simulation driver decodes each message sequentially from the stream. + + **Human inspection**: `xceadmin scheduler dump ` decodes a `.replay` file to human-readable text. xceadmin already bundles `controller.capnp` schemas for `config push`/`config build` — `scheduler dump` adds one new subcommand reusing the same schema infrastructure. + + **Recording (append)**: The `.replay` format is append-friendly — `save_snapshot()` and event writes serialize one capnp message at a time. On `record start`: write `ReplayHeader` + initial `SnapshotEntry`. Each event: write one `ReplayEvent`. Each session: write one `ReplaySession`. Periodic checkpoint: write one `SnapshotEntry`. On `record stop`: write `ReplayEvent{endSession}` + fflush + fclose. Recording failure mid-write: the file is truncated to the last complete message — no partial-message corruption. + + **Simulation (read)**: `--sim-start-ts NS` seeks forward through `SnapshotEntry` messages (deserialize each, check `ts <= NS`, keep latest match), loads the matching snapshot, then replays all messages with `ts >= NS`. + +**State Snapshot** + +21. **Snapshot serialization**: The core shall support `save_snapshot()` producing a `SnapshotData` capnp struct containing all scheduler configuration (`xce_conf4sched_t*` — queues, priorities, resource thresholds, preemption config, time windows), jobs, workers, resources, resvals, users, projects, shared-resource tracking state, unsatisfiable-resreq cache, PRNG state, and session state. The core shall support `load_snapshot()` restoring from `SnapshotData` to that exact state. + + **The snapshot is fully self-contained for simulation.** After `load_snapshot()` and picker rebuild, the core is ready to process `core_push_events()` + `core_go()` and produce decisions — no other state, external config, or preceding events are needed. Given a `.replay` file, the simulation driver can start from any `SnapshotEntry` (via `--sim-start-ts`): load the snapshot, rebuild picker, set virtual clock to snapshot `ts`, then replay all messages with `ts >= snapshot.ts`. The result is identical to replaying from the beginning — the snapshot is a complete checkpoint. + + **Picker state is NOT stored in snapshot.** The builtin picker's private state (`ordq_t queues`, per-queue `ordq_t pendJobs`) is derived from core state: every queue → `on_queue_update`, every pending job → `on_job_push`. After `load_snapshot()` restores core state, the driver shall rebuild the picker's private state by walking the core's queues and jobs and calling the picker's state-change hooks. This ensures: + - Snapshots capture only canonical core state (no duplicate of picker internals) + - Picker rebuild is deterministic: same core state + same hook sequence = same picker state + - Custom pickers with additional state can rebuild from the same core-state walk + + **Snapshot contents checklist** — everything required to run a scheduling session: + + | Category | Contents | + |----------|----------| + | Config | `xce_conf4sched_t*` — all queues, priorities, resource thresholds, preemption config, time windows | + | Jobs | All job records with current state, priority, queue assignment, queue_time | + | Queues | All queue definitions with RUNLIMIT, MAX_RUNNING, state, per-queue picker override | + | Workers | All worker records with resources, state, assigned jobs | + | Resources | Resource definitions (resreq templates) | + | Resvals | Current resource value snapshots for all workers | + | Users | User records with resource limits, group membership | + | Projects | Project scaffolding with priority (v1: flat, v2: hierarchical quota) | + | Shared-resource | Shared resource tracking state across workers | + | Unsatisfiable cache | Resreqs that could not be satisfied in previous rounds (for early skip) | + | Session state | Current scheduling round, iterator positions, GC state | + | PRNG state | Current PRNG value (for deterministic tie-breaking resume mid-file) | + +22. **Interactive simulation REPL**: In simulation mode with `--interactive` (or `-i`), the scheduler shall open a read-eval-print loop on stdin/stdout after loading the replay file but before the first event is processed. The REPL gives the operator full control over simulation execution — single-step, breakpoints, state inspection, and snapshot capture — turning the simulation driver into a real debugger for scheduling policy. + + **Interaction model**: The simulation process blocks on stdin at startup (pre-replay pause) and at each breakpoint hit. No controller, no network, no authentication — this is a local investigation tool. The REPL prompt is `(sim) `. + + **Minimum command set**: + + **Execution control**: + | Command | Description | + |---------|-------------| + | `step [N]` | Advance N sessions (default 1), pause after. Processes all events up to and including the Nth `BEGIN_ROUND` from the current position, then returns to REPL. | + | `continue` | Run until next breakpoint hit or EOF. If no breakpoints are set, runs to completion. | + | `quit` | Exit the scheduler immediately. No cleanup guarantees beyond `fclose` on open files. | + + **Breakpoints**: + | Command | Description | + |---------|-------------| + | `break [key=value ...]` | Set breakpoint on event type (e.g., `JOB_STATE_CHANGE`, `JOB_PUSH`, `BEGIN_ROUND`, `DECISION`). Optional key=value filters narrow the match: `break JOB_STATE_CHANGE jobid=12345`, `break JOB_STATE_CHANGE state=RUN`, `break DECISION type=DISPATCH queue=normal`. Multiple filters are AND'd. Key names match the field names in the capnp schema for each message type. | + | `info break` | List all breakpoints with their IDs, event types, and filters. | + | `delete ` | Delete breakpoint by ID. `delete all` removes all breakpoints. | + + **State inspection**: + | Command | Description | + |---------|-------------| + | `info session` | Show current simulation position: logical time (`xce_core_now()`), event index (Nth message processed), session count, last `SessionSummary` if available. | + | `info job ` | Dump one job: state, queue, priority, queue_time, resreq, allocations. Jobid uses the same format as xcecli (e.g., `12345`). | + | `info queue ` | Dump one queue: priority, state (open/closed), RUNLIMIT, MAX_RUNNING, job_pick_plugin, list of queued job IDs with priorities. `info queue all` lists all queues. | + | `info worker ` | Dump one worker: state, resources (total + available), assigned jobs. `info worker all` lists all workers. | + | `info resource ` | Dump one resource: type, total capacity, available, allocation count. `info resource all` lists all resources. | + | `info summary` | Quick overview: `N_jobs (M pending, K running), N_workers (M online), N_queues, N_resources`. | + + **Snapshots**: + | Command | Description | + |---------|-------------| + | `snapshot [FILE]` | Write a full `save_snapshot()` to FILE as a standalone `.replay` file (contains `ReplayHeader` + `SnapshotEntry`). If FILE is omitted, auto-generate: `-snap-.replay`. The snapshot is a complete state dump (same format as periodic recording checkpoints). The operator can restart simulation from this snapshot via `--sim --sim-start-ts `. | + + **Help**: + | Command | Description | + |---------|-------------| + | `help` | Print available commands with one-line descriptions. Covers all REPL commands: execution control (`step`, `continue`, `quit`), breakpoints (`break`, `info break`, `delete`), state inspection (`info session/job/queue/worker/resource/summary`), snapshots (`snapshot`), sim parameters (`set`, `info config`). | + + **Simulation parameters** (interactive `set` / `info config`): + | Command | Description | + |---------|-------------| + | `set speed ` | Set simulation speed from this point forward. `0` = as-fast-as-possible (no wall-clock delay), `1` = real-time, `N` = N× real-time. Overrides the `--sim-speed` CLI value. | + | `set output ` | Start writing simulation decisions to FILE in the same `.replay` capnp stream format as `--sim-output`. Opens `ReplayHeader`+`SnapshotEntry` at the current position if the file is new. File is written incrementally — each subsequent session's decisions land in the file. | + | `set output off` | Close the current output file. Subsequent decisions are not recorded. `info config` confirms the file path and event count at close time. | + | `set verify on\|off` | Enable or disable decision verification against the recorded replay file. When enabled, each decision is compared against the corresponding recorded decision at the same logical time. Mismatches are printed immediately. Default: matches `--sim-verify` CLI flag. | + | `info config` | Show current sim parameters: `speed=N, output=, verify=on|off, events_processed=N, current_ts=, sessions=N`. | + + **Why interactive parameters matter**: Without these, tuning simulation requires restart: change `--sim-speed`, re-run from snapshot, re-check results. With `set speed` and `set output`, the operator can fast-forward through known-good regions (`set speed 0`), slow down for suspicious sections (`set speed 0.1`), and only start recording output when the interesting behavior starts — all in one session. This turns the REPL from a passive inspector into an active investigation tool where parameters co-evolve with the operator's understanding of the trace. + + **REPL integration with the simulation loop**: + ``` + load replay file + snapshot + if --interactive: + print "[sim] loaded: N events, snapshot @ ts=..." + print "[sim] ready. type 'help' for commands." + repl_pause() // pre-replay pause — operator can set breakpoints + + for each event in replay_file: + // ── sim parameter: speed ── + if sim->sim_speed_num > 0: + wall_clock_delay(delta_ns * sim->sim_speed_den / sim->sim_speed_num) + + if check_breakpoints(event): + print "[sim] hit breakpoint @ event #I: ..." + repl_pause() + + core_push_events(core, event) + + if msg.which == ReplaySession: + core_go(core) + for each decision: + // ── sim parameter: verify ── + if sim->verify: + compare_and_report(decision, expected) // AC 15; REPL 'set verify on/off' + + // ── sim parameter: output ── + if sim->output_file: + write_replay_decision(sim->output_file, decision) // AC 19; REPL 'set output' + + if check_breakpoints(decision): + print "[sim] hit breakpoint @ decision: job=..." + repl_pause() + + if --interactive: + print "[sim] replay complete. N_sessions processed." + if sim->output_file: + print "[sim] output written to: " + ``` + + **What is NOT in v1**: + - **Write-side mutations to scheduler state**: No `set queue priority`, `set job state`, `add job`, `remove job`. The `set` command in v1 only tunes simulation parameters (`speed`, `output`, `verify`) — it never mutates core scheduling state. Operator inspects state but cannot alter it. What-if analysis uses `--sim-alt-events` (AC 16) instead. State mutation from REPL is a follow-up. + - **Reverse execution**: No `step back` or time-travel undo. Operator can restart from snapshots captured at earlier points. + - **Expression evaluation**: No `print core->sessions_count` or arbitrary C evaluation. Commands are fixed. + - **Scripting/batch mode**: No `source FILE` or `-c "step 5; info summary"`. The REPL is interactive-only. For automated simulation runs, use the batch flags (`--sim`, `--sim-output`, `--sim-verify`) without `--interactive`. + - **No semantic `snapshot diff` in xceadmin**: `xceadmin scheduler dump` produces human-readable output. Comparing two dumps (`diff <(xceadmin scheduler dump snap1) <(xceadmin scheduler dump snap2)`) works but is generic text diff — no scheduler-aware semantic diff ("queue priority changed from 10 to 20", "3 new jobs added"). Semantic diff is a v2 feature. + + **Deterministic replay is preserved**: The REPL never alters core state or event stream. It only controls *when* events are processed (by pausing), not *what* events are processed. The same replay file with the same breakpoints ignored (treated as skip) produces identical decisions as batch mode. Interactive use with `step` and `continue` that runs to completion produces the same output as `--sim` without `--interactive`. + + **Implementation note**: The REPL is a single `fgets` loop in the simulation driver. It is not a separate thread — when the REPL blocks on stdin, the simulation loop is parked. This is correct and intentional: the operator is in control, and the simulation has no wall-clock deadlines. + +Full object sharing (resreq template registry, pend reason dedup pool) is deferred to a follow-up spec. However, the architecture must not block sharing from being added later: + +1. **Resreq immutability**: Resreq objects shall be treated as immutable once created. No code path shall mutate a resreq in place. This ensures sharing via `const` pointer is safe to introduce later. + +2. **Pend reason factory**: All pend reason allocation shall go through `make_pendreason()` (the single factory in `lib/preason.c`). Callers never `calloc(sizeof(xce_pendreason_t))` directly. + +3. **Pointer-identity fast path**: Code comparing two resreqs or pend reasons shall use pointer equality (`a == b`) as a fast path before deep comparison. + +4. **Select expression memoization**: The existing `select_eval_ctx_t` per-session memoization shall be preserved and extended. Parsed/evaluated form attached to resreq on first evaluation, reused for subsequent evaluations. + +5. **Snapshot forward-compatibility**: The snapshot format shall not preclude object sharing. Shared objects shall be serializable by identity (object table with IDs). + +These invariants impose zero overhead today but rule out design choices that would make sharing expensive or impossible to retrofit. + +## Data Structure Changes Summary + +### `xce_scheduler_t` additions +```c +char* job_pick_plugin; // default: "builtin_fcfs" (Req 2) +``` + +### `xce_sched_queue_t` additions +```c +char* project_name; // owning project (picker uses internally) +char* job_pick_plugin; // per-queue picker override (NULL = use global) +``` + +### `xce_sched_project_t` additions +```c +int32_t priority; // inter-project ordering (future spec) +// future: resource_quota, resource_used, parent_project +``` + +### `xce_job_picker_iface_t` (new) +```c +typedef struct { + int (*init)(void* picker, xce_sched_core_t* core); + void (*fini)(void* picker); + int (*begin_session)(void* picker); + xce_sched_job_t* (*next_job)(void* picker); + int (*end_session)(void* picker); + int (*on_dispatch)(void* picker, uint64_t jobid); + int (*on_pend)(void* picker, uint64_t jobid); + int (*on_queue_update)(void* picker, xce_sched_queue_t* queue); + int (*on_job_push)(void* picker, xce_sched_job_t* job); + int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state); + int (*on_job_update)(void* picker, xce_sched_job_t* job); + int (*on_config)(void* picker, xce_conf4sched_t* config); + int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); +} xce_job_picker_iface_t; +``` + +### Core/Driver split +```c +// Core event (driver → core) +typedef enum { XCE_CORE_EVENT_CONFIG_PUSH = 0, XCE_CORE_EVENT_JOBS_PUSH, + XCE_CORE_EVENT_WORKERS_PUSH, XCE_CORE_EVENT_RESOURCES_PUSH, + XCE_CORE_EVENT_RESVALS_FULL, XCE_CORE_EVENT_RESVALS_DELTA, + XCE_CORE_EVENT_JOB_STATE_CHANGE, XCE_CORE_EVENT_JOB_UPDATE, + XCE_CORE_EVENT_QUEUE_UPDATE, XCE_CORE_EVENT_JOB_RUSAGE, + XCE_CORE_EVENT_BEGIN_ROUND } xce_core_event_type_t; + +typedef struct { + xce_core_event_type_t type; + uint64_t ts; + union { /* event-type-specific payload */ } payload; +} xce_core_event_t; + +typedef int (*xce_core_decision_cb_t)(void* driver_ctx, xce_decision_t* decision); + +typedef struct { + // Scheduling state (moved from xce_scheduler_t) + xce_conf4sched_t* config; var plugins; var jobs, queues, workers, users, projects; + // ... (full state) + + // Core/driver boundary + void* driver_ctx; + xce_core_decision_cb_t on_decision; + + // Internal threading — cbuf producer/consumer + pthread_t picker_thread, allocator_thread; + cbuf_t wbuf; // driver→picker trigger (int, cap=1) + cbuf_t jbuf; // picker→allocator job handoff (uint64_t, cap=1) + cbuf_t decisions; // allocator→driver decision output (xce_decision_t*, cap=5) + _Atomic int session_active; // 1 during session, 0 between + + // Clock virtualization — atomically updated by driver + _Atomic uint64_t virtual_clock_base, real_clock_base, speed_num, speed_den; + uint64_t prng_seed; + + // Picker plugin slot + char* job_pick_plugin; +} xce_sched_core_t; + +typedef struct { + xce_sched_core_t core; + xcecli_ctx_t apictx; + xce_cluster_config_t cluster_cfg; + dataq_t* pend_updates[2]; + // Recording state (driver-layer only — core is unaware) + char* record_file_path; // strdup'd path for status queries (AC 10) + FILE* record_file; // replay file handle (NULL when not recording) + bool recording; + uint64_t record_event_count; + uint64_t record_snapshot_interval; + uint64_t record_start_ts; +} xce_sched_driver_t; + +typedef struct { + xce_sched_core_t core; + FILE *replay_file, *alt_events_file, *record_file; + // sim parameters (mutable via REPL 'set' command — AC 22) + uint64_t sim_speed_num, sim_speed_den; // 0/1=fast, 1/1=real-time, N/1=Nx + bool verify; // decision comparison on/off + bool interactive; // REPL active +} xce_sched_sim_driver_t; + +// Core interface +int xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed); +int xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents); // batch push before session +int xce_core_go(xce_sched_core_t*); // NON-BLOCKING: signal picker thread +int xce_core_end_session(xce_sched_core_t*); // GC, reset workers +uint64_t xce_core_now(xce_sched_core_t*); // virtualized clock accessor +int xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out); // fills out with core state +int xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in); // restores from snapshot +void xce_sched_core_fini(xce_sched_core_t*); +``` + +## Architecture Diagrams + +### Internal Core Thread Architecture (Multi-Threaded, cbuf-Based) + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ CORE (process) │ +│ │ +│ [PICKER THREAD] [ALLOCATOR THREAD] │ +│ │ +│ cbuf_get(&wbuf) ──┐ cbuf_get(&jbuf) ──┐ │ +│ begin_session() │ try_allocate() │ │ +│ while next_job(): │ can_preempt() │ │ +│ cbuf_put(&jbuf,─┼──► cbuf_put( │ │ +│ &jobid) │ &decisions,──► on_decision callback │ +│ cbuf_put(&jbuf, 0) │ &decision) │ │ +│ end_session() │ (NULL on EOS) │ │ +│ ┌─────────────────┘ ┌─────────────────┘ │ +│ │ │ │ +│ │ wbuf jbuf │ decisions │ +│ │ (int) (uint64_t) │ (xce_decision_t*) │ +│ │ cap=1 cap=1 │ cap=5 │ +│ └─────────────────────────┴──────────────────────────────────────│ +│ │ +│ [Clock atoms — lock-free] [Core state] │ +│ virtual_clock_base queues, jobs, workers │ +│ real_clock_base plugins, resources, resvals │ +│ speed_num, speed_den picker plugin instance │ +│ prng_seed users, projects │ +└──────────────────────────────────────────────────────────────────┘ +``` + +### Core Scheduling Loop (per-round) + +``` + ┌─────────────────────┐ + │ DRIVER THREAD │ + │ │ + │ core_push_events() │ ← batch: CONFIG_PUSH, JOBS_PUSH, + │ picker hooks │ JOB_STATE_CHANGE, QUEUE_UPDATE, + │ called sync │ WORKERS_PUSH, RESVALS_*, etc. + │ │ + │ core_go() ──────────┼── cbuf_put(&wbuf, &go) + │ NON-BLOCKING │ + │ │ + │ while (1): │ + │ cbuf_get(&decisions)── block per decision + │ if NULL: break │ ← end-of-session marker + │ on_decision(d) ───┼── publish to controller NOW + │ │ (xcecli_jobdispatch / + │ core_end_session() │ xcecli_set_pend_reason / + │ │ xcecli_preempt) + │ sleep(1) or next │ + │ event trigger │ + └─────────────────────┘ +``` + +### Core/Driver Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ DRIVER LAYER │ +│ Normal Mode Simulation Mode │ +│ controller capnp ──┐ replay.replay ──┐ │ +│ handler → events │ virtual clock │ │ +│ → publish decisions│ → record/log │ │ +│ │ ▲ │ │ ▲ │ │ +└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘ + │ │ decisions │ │ decisions + ▼ │ ▼ │ +┌──────────────────────────────────────────────────────────────┐ +│ SCHEDULER CORE │ +│ core_push_event() → picker hooks → picker->next_job() │ +│ → try_allocate() → on_decision() → DISPATCH / PREEMPT / PEND │ +│ ZERO dependencies on: controller, capnp, network, wall-clock, filesystem │ +└──────────────────────────────────────────────────────────────┘ +``` + +## Future: Project Resource Ownership + +Deferred to a separate `project-resource-ownership` spec. When implemented: + +- Projects form a tree with resource quota at each node +- A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue +- The core scheduling loop requires zero code changes — only a new picker plugin replaces `builtin_fcfs` +- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes +- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends + +The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. The only difference is which picker plugin is configured. diff --git a/.kiro/specs/scheduler-core/spec.json b/.kiro/specs/scheduler-core/spec.json new file mode 100644 index 00000000..2e662d7b --- /dev/null +++ b/.kiro/specs/scheduler-core/spec.json @@ -0,0 +1,26 @@ +{ + "feature_name": "scheduler-core", + "created_at": "2026-07-27T00:00:00Z", + "updated_at": "2026-07-27T00:00:00Z", + "language": "en", + "phase": "requirements-generated", + "parent_spec": "scheduler-framework", + "description": "Core/driver separation, pluggable job picker interface, and simulation mode for the XCE scheduler", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "blocked_on": [], + "sibling_specs": ["scheduler-allocation", "scheduler-preemption"] +} diff --git a/.kiro/specs/scheduler-framework/brief.md b/.kiro/specs/scheduler-framework/brief.md index e556d22d..82a7e105 100644 --- a/.kiro/specs/scheduler-framework/brief.md +++ b/.kiro/specs/scheduler-framework/brief.md @@ -2,13 +2,14 @@ ## Problem The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework: -1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation. The picker is monolithic — it both chooses which queue to serve and which job within the queue. +1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. The picker runs in a dedicated thread with `jbuf` cbuf communication. No way to swap policies without recompilation. 2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation. 3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails. 4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources. -5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. Queue and project roles are conflated. +5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. +6. **Monolithic scheduler** — the scheduler is tightly coupled to controller I/O (capnp, `apictx`). No way to replay recorded events for debugging or test scheduling policies without a live cluster. -Operators need configurable scheduling policies at multiple levels, parallel job support, and preemption. The scheduler must generate multiple candidate allocation plans and try them in preference order. +Operators need configurable scheduling policies, parallel job support, preemption, and a way to debug/tune scheduling behavior without a live cluster. ## Current State @@ -21,36 +22,49 @@ Operators need configurable scheduling policies at multiple levels, parallel job - **Select expression AST** (`resreq.h`): full parser/evaluator with per-session memoization (`select_eval_ctx_t`) and inter-session GC. - **Shared resources** (`xce_scheduler_t.sharedInst`/`sharedInUse`): zarray of resource values tracked per scheduling session. - **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next. +- **Controller integration** (`handler.c`): receives capnp push/notify → enqueues `xce_sched_update_request_t` → `sched_update()` dequeues. Decisions published via `sched->apictx`. ### What's missing -- Configurable job picker: the plugin name is a hardcoded string literal; a monolithic picker conflates inter-queue and intra-queue selection +- Configurable job picker: the plugin name is a hardcoded string literal - Parallel job allocation: no combinatorial plan generation, no multi-slot host matching - Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans -- Preemption: no mechanism to preempt lower-priority running jobs when a high-priority job cannot be scheduled -- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` — no resource ownership, quota, or scheduling policy +- Preemption: no mechanism to preempt lower-priority running jobs +- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` - Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session -- Picker composition: no way to assign different pickers to different queues or projects +- Scheduler simulation/debugging: no way to replay recorded events, no virtual clock, no deterministic mode +- Core/driver separation: scheduling logic is tightly coupled to controller I/O ## Desired Outcome -### Four-Level Scheduling Architecture +### Three Architectural Foundations -The scheduling loop is always `project → queue → job`, composing pickers at each level: +**1. Pluggable Job Picker** — the core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected. The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. The picker runs synchronously in the core's scheduling thread — no dedicated picker thread, no `jbuf` cbuf. -| Level | Picker | Plugin slot | Default (v1) | -|-------|--------|-------------|---------------| -| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` — single "default" project | -| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` | -| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` | -| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` | +The picker plugin interface (`xce_job_picker_iface_t`): -1. **Unified picker loop**: A single scheduling loop that always follows `project → queue → job`. In v1 (flat project space), the project picker `builtin_flat` yields one "default" project containing all queues — identical to today's behavior. In v2 (future), the project picker becomes quota-aware and hierarchical, without changing the loop code. -2. **Configurable job selection at every level**: each picker slot is a plugin. Level 1 (inter-queue) picks which queue to serve next. Level 2 (intra-queue) picks which job within that queue, with per-queue customization via the `queue → project → global` precedence chain. -3. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail. -4. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator. -5. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success. -6. **Preemption**: when allocation fails for a high-priority job, select victim jobs from lower-priority queues (LSF-style per-queue `preemptible_queues` list), send SIGSTP via controller→worker pipeline, and retry allocation next round. -7. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans + preemption exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session. +| Category | Hooks | Purpose | +|----------|-------|---------| +| Lifecycle | `init`, `fini` | Setup and teardown | +| State-change | `on_queue_update`, `on_job_push`, `on_job_state_change`, `on_job_update` | Called by core when events arrive from driver | +| Session | `begin_session`, `next_job`, `end_session` | Scheduling round lifecycle | +| Post-allocation | `on_dispatch`, `on_pend` | Notification after allocation attempt | +| Optional | `on_config`, `on_job_rusage` | Advanced pickers (NULL if unused) | + +**2. Iterator-Driven Allocation** — lazy `xce_alloc_plan_iter_t` (candidate (nhosts, slots_per_host) plans, most-preferred first) composed with lazy `xce_host_iter_t` (candidate hosts for a given plan). `parjob_allocator` implements real combinatorial multi-slot allocation. Session-level unsatisfiable resreq fingerprint cache avoids re-evaluating already-proven-unsatisfiable resreqs. + +**3. Core/Driver Separation** — the scheduler core is a passive, event-driven engine with zero controller I/O dependencies. The driver connects it to the outside world: + +| Mode | Driver | Input | Output | +|------|--------|-------|--------| +| Normal | Controller-connected | capnp push/notify → core events | Core decisions → controller | +| Simulation | Replay file | JSONL event file → core events | Core decisions → log (with `--sim-verify` diff against recorded) | + +- **Discrete event-driven virtual clock**: jumps directly to next event timestamp, no fixed tick, monotonic +- **Snapshot + delta replay**: load state snapshot, replay only subsequent delta events +- **Deterministic guarantee**: same initial state + same events = same decisions +- **Runtime recording control**: xceadmin `scheduler record start/stop` commands +- **JSONL replay format**: one event per line, debuggable, plain text +- **What-if branching**: load snapshot, replay alternative delta events ### Queue vs Project — Roles @@ -59,10 +73,10 @@ The scheduling loop is always `project → queue → job`, composing pickers at | **Role** | Scheduling mechanism | Resource ownership | | **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? | | **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster | -| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) | +| **Key attributes** | Priority, limits, time windows, per-queue picker override, preemption config | Quota, usage, hierarchy, members (future) | | **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many | -One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware). +One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware). The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2). ## Alternatives Considered @@ -71,237 +85,267 @@ One project submits to many queues (different hardware profiles). One queue rece | Aspect | Detail | |--------|--------| | **Job selection** | Add `job_pick_plugin` config field, same as chosen approach | -| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation — all plans materialized before first try | -| **Iterator** | No new abstraction. Simple for-loop over plan array, inner for-loop over hosts | -| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state, same as chosen approach | +| **Parallel jobs** | Implement parjob allocator with pre-computed plan list. Eager generation | +| **Iterator** | No new abstraction. Simple for-loop over plan array | +| **Shared resreq** | `xlist_t unsatisfiable_resreqs` in session state | -**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces (e.g., 64 slots on 128 hosts = thousands of plans). No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost. +**Rejected because:** Eager plan generation causes memory pressure for large combinatorial spaces. No reusable abstraction — each new allocator reimplements the loop. Short-term win, long-term rework cost. -### Approach B: Iterator-Driven Allocation Architecture ★ Chosen +### Approach B: Iterator-Driven Allocation + Picker Plugin ★ Chosen | Aspect | Detail | |--------|--------| -| **Job selection** | Configurable plugin name via `job_pick_plugin` field | +| **Job selection** | Pluggable `xce_job_picker_iface_t` with `next_job()`. Single `builtin_fcfs` default. Per-queue picker override. Synchronous, no dedicated thread. | | **Iterator abstraction** | `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy, cancel-safe, composable | | **Parallel jobs** | Combinatorial plan generation via plan iterator: 1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 | | **Host iterator** | Per-plan lazy candidate host iteration using existing select expression eval | +| **Preemption** | `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP via controller→worker | | **Shared resreq** | Session-level fingerprint cache: `{select, rusage, user, queue, project}` | +| **Core/driver split** | Passive event-driven core + dual-mode driver (normal/simulation) | +| **Simulation** | JSONL replay, snapshot + delta, discrete event-driven virtual clock, deterministic | -**Chosen because:** Lazy generation avoids memory pressure. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. Clean separation of concerns. Each iterator implementation is independently testable. +**Chosen because:** Lazy generation avoids memory pressure. Single `next_job()` interface makes pickers truly pluggable — the core loop never changes. Core/driver separation enables simulation mode for debugging/tuning without a live cluster. Iterator composition matches the user's vision: plan iterator → host iterator → try alloc. ### Approach C: Full Scheduler Framework (deferred) | Aspect | Detail | |--------|--------| -| **Job selection** | Policy chain — multiple plugins score/veto jobs. Composable: priority + fair-share + preemption | +| **Job selection** | Policy chain — multiple plugins score/veto jobs | | **Iterator** | Triple composition: job iterator → plan iterator → host iterator | -| **Resreq registry** | Named primary resreqs registered globally, referenced by name, reference-counted | -| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin, topology-aware | -| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT with well-defined transitions | +| **Resreq registry** | Named primary resreqs registered globally, reference-counted | +| **Allocation plugins** | Different strategies as plugins: greedy, bin-packing, round-robin | +| **Session state machine** | Formalized STAGING → ELECT → ALLOCATE → COMMIT | -**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's iterator abstraction is designed to compose upward — the plan iterator and host iterator are building blocks that a future job iterator or policy chain can consume. The template registry can layer on top without breaking the iterator interfaces. +**Deferred (not rejected):** The policy chain and resreq registry are valuable but lack current demand. Approach B's picker interface is designed to compose upward — a future policy-chain picker implements the same `next_job()` interface. -### Decision rationale +### Decision Rationale -Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). The four-level picker architecture (project → queue → job + preemption) extends Approach B with design elements from Approach C (policy composition, formalized level separation) that are needed NOW (per-queue pickers, preemption), while deferring the remaining Approach C layers (policy chains with scoring, resreq registry, formal state machine) until real use cases emerge. The iterator interfaces and picker slots are designed to compose upward when needed. +Approach B chosen over A because the iterator abstraction pays off immediately (parjob combinatorial allocation) and compounds (every future allocator reuses it). The picker plugin interface with `next_job()` keeps the core loop simple (5 lines) and makes pickers truly pluggable. Core/driver separation enables simulation mode — a capability that would be prohibitively expensive to retrofit later. -**Updated decision (2026-07-10):** The original Approach B treated the job picker as a single global plugin (`job_pick_plugin` on `xce_scheduler_t`). Design discussion identified that different queues/projects need different pickers, and that preemption is an essential missing mechanism. The architecture now specifies four composable levels (project, queue, job, preemption) with per-queue configuration, LSF-style cross-queue preemption, and a unified loop that works identically for v1 (flat projects) and v2 (hierarchical projects). The project picker level is deferred to a future spec but the architecture slot and interface contract are defined now. +**Design evolution (2026-07-10 → 2026-07-27):** The original design specified a four-level architecture with three separate builtin plugins (`builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`) and a three-level `project→queue→job` loop hardcoded in the core. Analysis of the existing FCFS plugin (`fcfs.c`) showed it already encapsulates all picking logic internally. The architecture was simplified to a single `next_job()` plugin interface with one builtin picker (`builtin_fcfs`) — the three-level decomposition is now an internal implementation detail of the plugin, not the core loop. Core/driver separation was added (Req 10) to enable simulation mode for policy debugging and tuning. ## Approach -**Iterator-Driven Allocation Architecture with Four-Level Picker Composition** (extended from original Approach B): +**Iterator-Driven Allocation with Pluggable Picker + Core/Driver Separation**: -### Core abstractions - -``` -┌──────────────────────────────────────────────────────────┐ -│ Level 0: Project Picker [sched->project_pick_plugin] │ -│ │ -│ v1 (builtin_flat): yields one "default" project │ -│ v2 (future): quota-aware project selection │ -└──────────────────────┬───────────────────────────────────┘ - │ project scope - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │ -│ │ -│ Iterates queues scoped to the selected project │ -│ Default: strict_priority (descending queue->priority) │ -└──────────────────────┬───────────────────────────────────┘ - │ queue - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Level 2: Intra-Queue Picker [queue→project→sched] │ -│ │ -│ Picks one job from the selected queue │ -│ Default: builtin_fcfs (priority desc, queue_time asc) │ -└──────────────────────┬───────────────────────────────────┘ - │ job - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Allocation │ -│ │ -│ xce_alloc_plan_iter_t xce_host_iter_t │ -│ ┌─────────────────────┐ ┌──────────────────────┐ │ -│ │ init(resreq, sched) │ │ init(plan, workers) │ │ -│ │ next() → plan │───────▶│ next() → worker │ │ -│ │ fini() │ │ fini() │ │ -│ └─────────────────────┘ └──────────────────────┘ │ -│ │ -│ success → dispatch │ -│ fail → Level 3 (preemption) │ -└──────────────────────┬──────────────────────────────────┘ - │ fail - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Level 3: Preemption │ -│ │ -│ if queue->preemptible_queues: │ -│ victims = preempt_policy.select_victims(job) │ -│ if victims: commit XCE_DECISION_PREEMPT │ -│ → controller → worker → SIGSTP victims │ -│ → victims → JOB_SUSPENDED, release resources │ -│ → pend job for next round │ -│ else: pend(job, "Resources") │ -│ else: pend(job, "Resources") │ -└──────────────────────────────────────────────────────────┘ -``` - -### Picker plugin interface - -Each picker level follows the same contract: -```c -typedef struct { - void* (*init)(xce_scheduler_t* sched, void* parent_scope); - void* (*next)(void* iter); // returns scope or NULL - void (*fini)(void* iter); -} picker_iface_t; -``` - -### Unified scheduling loop +### Core scheduling loop ```c void* sched_allocator(void* arg) { while (1) { - // Level 0: project scope - scope = project_picker.next(sched); - if (!scope) { sleep(1); continue; } + picker->begin_session(picker); - // Level 1: queue (scoped to project) - queue = queue_picker.next(sched, scope); - if (!queue) { project_picker.advance(); continue; } - - // Level 2: job (within queue) - // Precedence: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin - picker = resolve_job_picker(sched, queue); - job = picker->elect_one(queue); - - if (!job) { queue_picker.advance(); continue; } - - // Allocation + preemption - if (try_allocate(sched, job)) { - dispatch(sched, job); - } else if (can_preempt(sched, job)) { - preempt_and_pend(sched, job); - } else { - pend(sched, job, "Resources"); + xce_sched_job_t* job; + while ((job = picker->next_job(picker)) != NULL) { + if (try_allocate(job) > 0) { + picker->on_dispatch(picker, job->jobid); + emit_decision(DISPATCH, job); + } else if (can_preempt(job)) { + emit_decision(PREEMPT, job, victims); + picker->on_pend(picker, job->jobid); + } else { + emit_decision(PEND, job); + picker->on_pend(picker, job->jobid); + } } + + picker->end_session(picker); + sleep(1); } } ``` -This loop is identical for v1 and v2. v1 uses `builtin_flat` for Level 0. v2 replaces it with a quota-aware project picker. +The core loop has zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`. + +### Picker plugin interface + +```c +typedef struct xce_job_picker { + // Lifecycle + int (*init)(void* picker, xce_sched_core_t* core); + void (*fini)(void* picker); + + // State-change hooks (called by core when events arrive) + int (*on_queue_update)(void* picker, xce_sched_queue_t* queue); + int (*on_job_push)(void* picker, xce_sched_job_t* job); + int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state); + int (*on_job_update)(void* picker, xce_sched_job_t* job); + + // Session lifecycle + int (*begin_session)(void* picker); + int (*end_session)(void* picker); + + // Job selection + xce_sched_job_t* (*next_job)(void* picker); + + // Post-allocation notification + int (*on_dispatch)(void* picker, uint64_t jobid); + int (*on_pend)(void* picker, uint64_t jobid); + + // Optional hooks (NULL = not implemented) + int (*on_config)(void* picker, xce_conf4sched_t* config); + int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); +} xce_job_picker_iface_t; +``` + +### builtin_fcfs internal design + +The default picker internally implements the equivalent of today's selection: + +``` +builtin_fcfs.next_job(): + Iterate queues by priority (descending) + Skip queues with no ready jobs + For each queue: + If queue has per-queue picker override: + Delegate to that picker + Else: + Iterate jobs by priority desc, queue_time asc (FCFS) + Skip jobs not ready + Return first ready job + Return NULL if all queues exhausted +``` + +This is an implementation detail of the plugin, not visible to the core loop. ### Preemption flow ``` scheduler controller worker ─────────── ────────── ────── -preempt_policy.select_victims(J) +alloc fails for high-prio job J + +preempt_policy.select_victims(J): + collect running jobs from + Q_aggressor.preemptible_queues + filter: victim.prio < J.prio + filter: victim.queue.preemptible + rank: lowest priority first + select minimum set satisfying J.resreq + → [victim1, victim2] -XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim - .victim_jobids[] deliver to each job → JOB_SUSPENDED - .signal = SIGSTP victim's worker release resources +XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 + .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED + .signal = SIGSTP victim's worker release resources .preempting_jobid = J J pended → next round → retry alloc ``` -V1 uses SIGSTP (terminal stop — process groups paused, job stays in queue). Future escalation to SIGKILL kill-and-requeue is deferred. +### Core/Driver Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ DRIVER LAYER │ +│ │ +│ Normal Mode Simulation Mode │ +│ controller capnp ──┐ replay.jsonl ──┐ │ +│ handler → events │ virtual clock │ │ +│ → publish decisions│ → record/log │ │ +│ │ ▲ │ │ ▲ │ │ +└───────┼───────┼─────┴──────────────┼───────┼───┴──────────────┘ + │ │ decisions │ │ decisions + │ │ (callback) │ │ (callback) + ▼ │ ▼ │ +┌──────────────────────────────────────────────────────────────┐ +│ SCHEDULER CORE │ +│ │ +│ core_push_event() → picker hooks → picker->next_job() │ +│ → try_allocate() → DISPATCH / PREEMPT / PEND │ +│ │ +│ ZERO dependencies on: controller, capnp, network, wall-clock │ +└──────────────────────────────────────────────────────────────┘ +``` ### What stays unchanged -- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved +- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — preserved, but the picker no longer uses the dedicated `job_elect_fn` thread - `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot - Select expression system — no changes - COMPOUND and ALTER resreq types — unchanged (PRIMARY focus) -- Controller communication (handler.c opcodes, decision cbuf) — extended, not replaced +- Controller communication (handler.c opcodes, decision cbuf in driver) — extended, not replaced - `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications ### What's new -- `XCE_DECISION_PREEMPT` decision type -- `xce_preempt_decision_t` payload -- Three builtin plugins: `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs` +- `xce_job_picker_iface_t` — pluggable picker interface with full hook specification +- `xce_alloc_plan_iter_t` + `xce_host_iter_t` — lazy iterator abstractions +- `XCE_DECISION_PREEMPT` decision type + `xce_preempt_decision_t` payload +- `builtin_fcfs` — single builtin picker (replaces three separate builtins from original design) +- `xce_sched_core_t` / `xce_sched_driver_t` / `xce_sched_sim_driver_t` — core/driver struct split +- `xce_core_event_t` — typed event interface between driver and core +- `xce_core_decision_cb_t` — decision callback (core → driver) +- JSONL replay file format + snapshot/restore +- Discrete event-driven virtual clock - `project_name` on queue structs -- `job_pick_plugin` on queue + project structs +- `job_pick_plugin` on scheduler + queue structs (per-queue override) - `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct -- `project_pick_plugin`, `queue_pick_plugin`, `job_pick_plugin` on scheduler struct ## Scope ### In -- Unified picker loop: `project → queue → job` with plugin slots at each level -- Three builtin plugins: `builtin_flat` (Level 0), `builtin_strict_priority` (Level 1), `builtin_fcfs` (Level 2) -- Picker plugin interface: `init/next/fini` — cancel-safe, session-aware -- Picker precedence chain: `queue → project → global` -- `xce_alloc_plan_iter_t` interface (header + implementation) -- `xce_host_iter_t` interface (header + implementation) +- Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic +- Single builtin picker (`builtin_fcfs`) replicating today's behavior +- Configurable picker plugin with per-queue override (`queue->job_pick_plugin`) +- Picker hook specification: lifecycle, state-change, session, post-allocation, optional hooks +- `xce_alloc_plan_iter_t` + `xce_host_iter_t` interfaces (lazy, cancel-safe, composable) - `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1) -- Preemption: `XCE_DECISION_PREEMPT` decision type, `xce_preempt_decision_t` payload, LSF-style cross-queue victim selection, SIGSTP mechanism via controller→worker pipeline +- Preemption: `XCE_DECISION_PREEMPT` decision type, LSF-style cross-queue victim selection, SIGSTP mechanism - Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy` - Session-level unsatisfiable resreq fingerprint cache -- Data structure changes to `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h` -- Plugin interface contract documentation (cancel-safe, session-aware) -- Unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection +- Core/driver separation: passive event-driven core, normal-mode driver, simulation-mode driver +- Discrete event-driven virtual clock (monotonic, jumps to next event timestamp) +- Runtime recording control via xceadmin commands (`scheduler record start/stop`) +- JSONL replay file format (HEADER + SNAPSHOT + events + DECISIONS) +- State snapshot/restore for replay efficiency +- Deterministic replay guarantee for debugging/tuning +- What-if branching (load snapshot, replay alternative delta events) +- Data structure changes: `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`, `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`, `xce_core_event_t`, `xce_job_picker_iface_t` +- Picker plugin interface contract documentation +- Unit tests: plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, event recording/replay, snapshot/restore, deterministic replay - Integration test: multi-slot parallel job allocation and preemption flow end-to-end ### Out -- New job selection policy implementations beyond the three builtins (priority, fair-share, deadline, preemption, backfill, etc.) — the configurable slots enable them, but none ship in this spec -- Hierarchical project resource quotas (Level 0 full) — deferred to `project-resource-ownership` spec -- Resreq template registry (named/shared resreq objects) — future enhancement +- New job selection policies beyond the builtin FCFS picker — the pluggable interface enables them, but none ship here +- Hierarchical project resource quotas — deferred to `project-resource-ownership` spec +- Resreq template registry — future enhancement - Select expression language changes -- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item) +- Controller-side scheduling state replication (commit barrier wiring) - Compound/ALTER resreq allocation changes - GPU/affinity-aware allocation (pluggable later via allocator plugins) -- Kill-and-requeue preemption escalation (v1 is SIGSTP only, suspend-and-retry) -- Deadline field on `xce_sched_job_t` (needed for deadline-based pickers — deferred) +- Kill-and-requeue preemption escalation (v1 is SIGSTP only) +- Deadline field on `xce_sched_job_t` +- Event payload JSON schema (open design question — deferred to design phase) +- Distributed/cluster-mode simulation (v1 is single-scheduler replay only) ## Future: Project Resource Ownership Deferred to a separate `project-resource-ownership` spec. When implemented: - Projects form a tree with resource quota at each node -- Level 0 project picker selects which project gets resources (fair-share, priority, proportional) -- Level 1 queue picker is scoped to the selected project's queues -- Level 2 job picker sees only jobs from the selected project within each queue -- Project quota enforcement: projects at or over quota are skipped by Level 0 -- The unified picker loop (above) requires zero code changes — only a new project picker plugin replaces `builtin_flat` +- A project-aware picker plugin selects which project gets resources, then iterates queues scoped to that project, then jobs within each queue +- Project quota enforcement: projects over quota are skipped by the picker +- The core scheduling loop requires zero code changes — only a new picker plugin replaces `builtin_fcfs` +- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes +- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends -The architecture is designed so that v1 (flat project space) and v2 (hierarchical projects) share the identical scheduling loop. +The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. ## Boundary Candidates The work naturally splits into these implementation phases (single spec, sequential): -1. **Unified picker loop + builtin plugins** — implement the `project → queue → job` loop with `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`; retire `sched_job_picker()`; add plugin slots and precedence chain to structs -2. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction -3. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator -4. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline (scheduler→controller→worker), queue preemption config fields -5. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop (skip before allocation + preemption attempt) -6. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, HA soak regression +1. **Core/driver separation** — split `xce_scheduler_t` into `xce_sched_core_t` + `xce_sched_driver_t`; define `xce_core_event_t` and `xce_core_decision_cb_t`; implement passive event-driven core interface; wire normal-mode driver to existing handler.c +2. **Picker plugin interface + builtin_fcfs** — define `xce_job_picker_iface_t`; implement `builtin_fcfs` with all MUST/SHOULD hooks; retire dedicated picker thread + `jbuf`; make core loop call `next_job()` synchronously; add `job_pick_plugin` config with per-queue override +3. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction +4. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator +5. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline, queue preemption config fields +6. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop +7. **Simulation driver** — implement simulation-mode driver with JSONL replay, virtual clock, snapshot/restore, deterministic verification +8. **Recording + xceadmin commands** — implement runtime recording control, periodic snapshots during recording +9. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, simulation replay, HA soak regression ## Out of Boundary -- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec +- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that implement `xce_job_picker_iface_t` - Resreq template sharing across jobs (future: `resreq_registry` spec) - Commit barrier wiring for scheduler decisions (separate hardening item in roadmap) - Controller-side changes (the scheduler runs independently, receives updates via existing opcodes) @@ -312,10 +356,11 @@ The work naturally splits into these implementation phases (single spec, sequent - Existing plugin framework (`plugin.h`, `plugin.c`) — NO changes needed - Existing select expression evaluator (`resreq.h` eval/Gc) — NO changes needed - Existing shared resource tracking (`sharedInst`/`sharedInUse`) — NO changes needed -- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — minimal additions +- Existing scheduler data structures (`xce_scheduler_t`, `xce_sched_ctx_t`) — split, not replaced ### Downstream (future consumers) -- **New job selection policies** — priority, fair-share, preemption plugins register via the configurable picker +- **New job selection policies** — priority, fair-share, deadline, backfill plugins implement `xce_job_picker_iface_t` +- **Project-resource-ownership** — project-aware picker plugin consumes `queue->project_name` and `xce_sched_project_t` - **Affinity/GPU allocators** — register as allocator plugins, implement plan iterator for topology-aware placement - **Commit barrier wiring** — scheduling decisions flow through majority commit (separate hardening item) - **Resreq template registry** — named resreqs shared across jobs, reference-counted @@ -323,7 +368,7 @@ The work naturally splits into these implementation phases (single spec, sequent ## Existing Spec Touchpoints ### Extends -- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same `xce_scheduler_t` state. No conflicts; the msg-gap spec fills handler gaps this spec reads from. +- **scheduler-controller-msg-gap** — adjacent (message handling). The framework enhancement operates on the same scheduling state. No conflicts; the msg-gap spec fills handler gaps this spec reads from. ### Adjacent - **ha-consistency-invariants** — commit barrier that scheduling decisions will eventually flow through. This spec does NOT wire the barrier; it only makes allocation decisions that the barrier WILL consume. @@ -332,7 +377,8 @@ The work naturally splits into these implementation phases (single spec, sequent ## Constraints - **C (gnu17)** — no C++, no external libraries beyond existing (capnp, zlib, pthreads) - **Non-blocking in plugin hooks** — allocation happens on allocator thread, not iomgr. No new blocking concerns. -- **Cancel-safe** — job picker thread is `pthread_cancel`-able. Iterator `next()` calls must have cancellation points. -- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session`. +- **Cancel-safe** — `next_job()` must have `pthread_cancel` cancellation points +- **Session isolation** — iterator state and resreq cache are per-session, cleared on `begin_session` - **Backward compatible** — existing FCFS + seqjob behavior unchanged. New functionality is additive. +- **Deterministic replay** — same initial state + same event sequence → same decisions. No wall-clock time, no unseeded randomness, stable iteration order. - **Memory** — lazy generation avoids materializing all combinations. Worst-case for a job needing N slots on M hosts: M×N candidate evaluations (acceptably bounded). diff --git a/.kiro/specs/scheduler-framework/requirements.md b/.kiro/specs/scheduler-framework/requirements.md index 06e9b5d8..891c6283 100644 --- a/.kiro/specs/scheduler-framework/requirements.md +++ b/.kiro/specs/scheduler-framework/requirements.md @@ -4,18 +4,13 @@ The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: `sched_job_picker()` (`scheduler.c:39`) sets `job_pick_plugin = "xce_sched_fcfs"` as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: `parjob_allocator.c:73`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` — any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()` at `scheduler.c:93-103`) produces no allocation at all. Third, allocation is eager and one-shot: the sibling `seqjob_allocator.c:107` `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops — there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails. -This feature closes all three gaps with an iterator-driven allocation architecture: a lazy `xce_alloc_plan_iter_t` (candidate `(nhosts, slots_per_host)` plans, most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session. +This feature closes all three gaps and adds two architectural foundations: -Beyond the three gaps, this feature introduces a **four-level scheduling architecture** that separates concerns cleanly and composes upward: +**Allocation**: A lazy `xce_alloc_plan_iter_t` (candidate `(nhosts, slots_per_host)` plans, most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session. -| Level | Picker | Question it answers | Plugin slot | Default (v1) | -|-------|--------|---------------------|-------------|---------------| -| 0 | Project picker | WHO gets resources this round? | `sched->project_pick_plugin` | `"builtin_flat"` — single "default" project | -| 1 | Inter-queue picker | WHICH queue within the project? | `sched->queue_pick_plugin` | `"builtin_strict_priority"` | -| 2 | Intra-queue picker | WHICH job from this queue? | `queue→project→sched` precedence chain | `"builtin_fcfs"` | -| 3 | Preemption policy | WHAT to preempt when allocation fails? | `queue->preempt_policy` | `"lowest_priority_first"` | +**Job picking**: A pluggable `xce_job_picker_iface_t` — the core scheduling loop calls `picker->next_job()` to get the next schedulable job, with zero knowledge of how jobs are selected (FCFS, priority, fair-share, deadline, etc.). The default `builtin_fcfs` picker internally implements queue-priority ordering and FCFS job ordering, replicating today's behavior exactly. Per-queue picker overrides are supported. Preemption fires at Level 3 when allocation fails for a preemptible job. -The scheduling loop is always `project → queue → job`, with Level 0 collapsing to a single implicit "default" project in v1 and expanding to hierarchical resource ownership in a future spec (`project-resource-ownership`). Level 3 (preemption) fires only when allocation fails for a preemptible job. +**Core/driver separation**: The scheduler core is a passive, event-driven engine with zero controller I/O dependencies. A normal-mode driver connects it to a live controller; a simulation-mode driver replays recorded cluster events through a discrete event-driven virtual clock, enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. > Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in `.kiro/specs/scheduler-framework/brief.md`. @@ -32,30 +27,223 @@ These are fundamentally different concerns: | **Cardinality to job** | 1 job → 1 queue | 1 job → 1 project | | **Cardinality to each other** | Many-to-many | Many-to-many | -- **Queue** = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, inter-queue priority, and the intra-queue job picker. -- **Project** = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. One project submits to many queues; one queue receives from many projects. +- **Queue** = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, priority (for inter-queue ordering within the picker), per-queue picker override, and preemption config. +- **Project** = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. One project submits to many queues; one queue receives from many projects. The picker plugin may use `queue->project_name` internally for project-scoped iteration (v2). ## Boundary Context -- **In scope**: Unified picker loop (`project → queue → job`) with plugin slots at each level; `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1) for `maxTasks > 1`; three builtin plugins (`builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`) replicating today's behavior through the new architecture; configurable picker slots with precedence chain (`queue → project → global`); preemption decision type (`XCE_DECISION_PREEMPT`) flowing scheduler→controller→worker with SIGSTP mechanism; cross-queue victim selection (LSF-style per-queue `preemptible_queues` list); session-level unsatisfiable-resreq fingerprint cache keyed on `{select, rusage, user, queue, project}`; documentation of plugin interface contracts (cancel-safe, session-aware); unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection; integration test for end-to-end multi-slot parallel job allocation and preemption flow. +- **In scope**: Pluggable job picker interface (`xce_job_picker_iface_t` with `next_job()`) hiding all selection logic behind a single plugin; `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1) for `maxTasks > 1`; single builtin picker (`builtin_fcfs`) replicating today's behavior through the new interface; configurable picker plugin with per-queue override (`queue->job_pick_plugin`); preemption decision type (`XCE_DECISION_PREEMPT`) flowing scheduler→controller→worker with SIGSTP mechanism; cross-queue victim selection (LSF-style per-queue `preemptible_queues` list); session-level unsatisfiable-resreq fingerprint cache keyed on `{select, rusage, user, queue, project}`; **core/driver separation** with passive event-driven core interface, normal-mode driver (controller-connected), simulation-mode driver (replay recorded events with discrete event-driven virtual clock), runtime recording control via xceadmin commands, JSONL replay file format, state snapshot/restore for replay efficiency, and deterministic replay guarantee for debugging/tuning; documentation of picker plugin interface contract (hooks, cancel-safe, session-aware); unit tests for plan generation, host iteration, cache hit/miss, picker hook behavior, preemption victim selection, **event recording/replay, snapshot/restore, and deterministic replay**; integration test for end-to-end multi-slot parallel job allocation and preemption flow. -- **Out of scope**: New job selection policies beyond the three builtins (priority, fair-share, deadline, preemption, backfill) — the configurable slots enable them, but none ship in this spec beyond the builtins; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring for scheduling decisions (separate roadmap hardening item, tracked adjacent to `ha-consistency-invariants`); `COMPOUND`/`ALTER` resreq allocation changes beyond what composing over `PRIMARY` already provides; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins); hierarchical project resource quotas (Level 0 full — deferred to `project-resource-ownership` spec); kill-and-requeue preemption escalation (deferred — v1 is SIGSTP only); deadline field on `xce_sched_job_t` (deferred — needed for deadline-based pickers). +- **Out of scope**: New job selection policies beyond the builtin FCFS picker (priority, fair-share, deadline, backfill) — the pluggable picker interface enables them, but none ship in this spec beyond `builtin_fcfs`; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring for scheduling decisions (separate roadmap hardening item, tracked adjacent to `ha-consistency-invariants`); `COMPOUND`/`ALTER` resreq allocation changes beyond what composing over `PRIMARY` already provides; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins); hierarchical project resource quotas (deferred to `project-resource-ownership` spec — only `project_name` on queue and `xce_sched_project_t` scaffolding ship here); kill-and-requeue preemption escalation (deferred — v1 is SIGSTP only); deadline field on `xce_sched_job_t` (deferred — needed for deadline-based pickers); **event payload JSON schema** (open design question — deferred to design phase); **distributed/cluster-mode simulation** (v1 is single-scheduler replay only). - **Adjacent expectations**: depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking (`xce_scheduler_t.sharedInst`/`sharedInUse`) — none of these change. `scheduler-controller-msg-gap` (implemented) governs controller↔scheduler message handling this feature reads state through but does not modify. `ha-consistency-invariants`' commit barrier is a future consumer of scheduling decisions; this feature does not wire it. Controller hook `HOOK_DETAIL_JOB_STOP` (enum value 65 in `cluster_def.h`) already exists and is reused for preemption stop actions. +## Allocation Architecture (Reqs 2–4) + +Requirements 2, 3, and 4 form a three-layer allocation architecture. These layers compose according to the job's resource request type (PRIMARY, COMPOUND, or ALTER): + +### Resreq Type Hierarchy + +``` +ALTER → OR-of-compounds: first compound that fully allocates wins + └─ COMPOUND → AND-of-primaries: ALL primaries must allocate for success + └─ PRIMARY → plan_iter → host_iter → try_alloc (leaf level) +``` + +The iterator composition (`xce_alloc_plan_iter_t` + `xce_host_iter_t`) operates at the **PRIMARY leaf**. COMPOUND and ALTER are composition rules above the iterators — they call into the same primary allocator, unchanged from today's `sched_make_compound_allocation()` and `sched_make_alter_allocation()` dispatch. + +### Iterator Composition (PRIMARY leaf) + +``` + ┌──────────────────────────┐ + │ xce_alloc_plan_iter_t │ Req 2 + │ │ + │ init(primary_resreq) │ + │ next() → plan │ generates (nhosts, slots_per_host) + │ fini() │ candidates, most-preferred first + └────────────┬───────────────┘ + │ plan + ▼ + ┌──────────────────────────┐ + │ xce_host_iter_t │ Req 3 + │ │ + │ init(plan, workers) │ for a given plan, iterate + │ next() → worker │ candidate hosts matching + │ fini() │ select expression + availability + └────────────┬───────────────┘ + │ worker + ▼ + ┌──────────────────────────┐ + │ try_alloc(worker, plan) │ Req 4 (parjob) or + │ │ existing seqjob logic + │ check slots, shared res, │ + │ commit or advance │ + └────────────────────────────┘ +``` + +### Composition Algorithm (All Three Types) + +```c +// === PRIMARY leaf: iterator composition === +// Returns >0 on success, 0 on failure +int allocate_primary(sched, primary_resreq, ctx) { + plan_iter = plan_iter_init(primary_resreq, sched); + while ((plan = plan_iter->next(plan_iter)) != NULL) { + host_iter = host_iter_init(plan, sched->workers); + while ((worker = host_iter->next(host_iter)) != NULL) { + if (try_alloc_on_host(worker, plan, ctx) > 0) { + host_iter->fini(host_iter); + plan_iter->fini(plan_iter); + return ALLOC_SUCCESS; + } + } + host_iter->fini(host_iter); + } + plan_iter->fini(plan_iter); + return ALLOC_FAIL; +} + +// === COMPOUND: AND of primaries === +// ALL primaries must allocate. On any failure, roll back prior allocations +// and return failure. Same logic as today's sched_make_compound_allocation(). +int allocate_compound(sched, compound_resreq, ctx) { + for (int i = 0; i < compound_resreq->primaries.count; i++) { + primary = compound_resreq->primaries[i]; + if (allocate_primary(sched, primary, ctx) <= 0) { + // Roll back all primaries allocated so far in this compound + rollback_compound_allocations(ctx, i); + return ALLOC_FAIL; + } + } + return ALLOC_SUCCESS; +} + +// === ALTER: OR of compounds === +// First compound that fully allocates wins. Same logic as today's +// sched_make_alter_allocation(). On failure, no rollback needed +// (prior compounds were never committed — they failed their own checks). +int allocate_alter(sched, alter_resreq, ctx) { + for (int i = 0; i < alter_resreq->compounds.count; i++) { + compound = alter_resreq->compounds[i]; + if (allocate_compound(sched, compound, ctx) > 0) { + return ALLOC_SUCCESS; + } + // This compound failed — advance to next alternative + } + return ALLOC_FAIL; +} + +// === Top-level dispatch === +// Unchanged from today's sched_make_allocation() dispatch on obj->tag. +int allocate_job(sched, job, ctx) { + switch (job->resreq_type) { + case PRIMARY: + return allocate_primary(sched, job->primary_resreq, ctx); + case COMPOUND: + return allocate_compound(sched, job->compound_resreq, ctx); + case ALTER: + return allocate_alter(sched, job->alter_resreq, ctx); + } +} +``` + +**COMPOUND rollback note:** If primaries 1 and 2 of a compound allocate but primary 3 fails, primaries 1 and 2 must be rolled back — the compound is atomic. Today this is handled by `sched_make_compound_allocation()` tracking allocations in `ctx` and cleaning up on failure. This behavior is unchanged; the iterator interfaces don't alter the rollback contract. + +### Who Uses This Composition + +| Allocator | `maxTasks` | Plan iterator | Host iterator | Behavior | +|-----------|------------|---------------|---------------|----------| +| `seqjob_allocator` | `== 1` | Trivial: single plan (1 host × 1 slot) | Full: filters hosts by select expression | Unchanged from today — first matching host wins | +| `parjob_allocator` | `> 1` | Full combinatorial: 1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1 | Full: filters hosts with enough available slots | New — walks plans until one succeeds | +| Future allocators | any | Custom strategy (e.g., topology-aware, bin-packing) | Reuses same `xce_host_iter_t` | Pluggable via `job_alloc_fn` | + +### Key Design Properties + +- **Lazy**: `plan_iter->next()` generates one plan at a time. `host_iter->next()` returns one host at a time. No full materialization. +- **Cancel-safe**: both `next()` calls have `pthread_cancel` cancellation points. +- **Independent lifecycles**: a `host_iter` is created from a plan, used, and destroyed; the `plan_iter` is advanced to the next plan only after the `host_iter` is done. `fini()` on one does not affect the other. +- **COMPOUND/ALTER unchanged**: the composition rules above the PRIMARY leaf (AND for compound, OR for alter) are unchanged from today. The iterator interfaces only change what happens inside `allocate_primary()`. +- **seqjob rewrap**: `seqjob_allocator` is rewrapped onto the iterator interfaces as a trivial case — its plan iterator always returns the single plan (1 host × 1 slot), and the inner loop is identical to today's first-match logic. This validates the abstraction before applying it to the complex parjob case. + ## Requirements -### Requirement 1: Unified picker loop (project → queue → job) +### Requirement 1: Pluggable job picker interface -**Objective:** As a maintainer of the scheduler, I want a single scheduling loop that composes project, queue, and job pickers in a fixed order, so that v1 (flat project space) and v2 (hierarchical projects with resource quotas) share the same code path. +**Objective:** As a maintainer of the scheduler, I want the core scheduling loop to call a single `next_job()` plugin interface with no knowledge of how jobs are selected, so that picking strategies (FCFS, priority, fair-share, deadline, etc.) are truly pluggable and the core loop never changes regardless of the picking algorithm. #### Acceptance Criteria -1. The Scheduler shall implement a scheduling loop that always follows the order: Level 0 (project picker) → Level 1 (inter-queue picker) → Level 2 (intra-queue picker) → allocation. -2. Each picker level shall conform to a common interface contract: `init(sched, parent_scope)`, `next(iter) → scope_or_NULL`, `fini(iter)`. -3. When a picker at any level returns NULL (exhausted), the loop shall advance the parent picker to its next scope and restart the child picker from that new scope. -4. In v1 (no project hierarchy), the default project picker `builtin_flat` shall yield exactly one scope: a "default" project containing all configured queues, so the loop behavior is identical to today's flat queue iteration. -5. Plugin resolution for Level 2 shall follow the precedence chain: `queue->job_pick_plugin` → `project->job_pick_plugin` → `sched->job_pick_plugin` → hardcoded default `"builtin_fcfs"`. -6. `sched_job_picker()` shall be retired and replaced by the unified loop; the hardcoded `"xce_sched_fcfs"` literal at `scheduler.c:43` shall be removed. +1. The Scheduler core loop shall call `picker->next_job(picker)` to get the next schedulable job (or NULL if none ready), and shall not contain any logic about project iteration, queue iteration, or job ordering — those concerns are entirely encapsulated inside the picker plugin. + +2. The job picker plugin interface (`xce_job_picker_iface_t`) shall define these operations, organized by category: + + **Lifecycle hooks** (MUST): + + | Operation | Signature | Semantics | + |-----------|-----------|-----------| + | `init` | `int (*init)(void* picker, xce_sched_core_t* core)` | One-time setup, receives core reference for state queries | + | `fini` | `void (*fini)(void* picker)` | Teardown and cleanup of all picker-owned resources | + + **State-change hooks** (MUST — called by core when events arrive from driver): + + | Operation | Signature | Semantics | Triggered by | + |-----------|-----------|-----------|-------------| + | `on_queue_update` | `int (*on_queue_update)(void* picker, xce_sched_queue_t* queue)` | Add/update queue in picker's internal ordering | `QUEUE_UPDATE` event | + | `on_job_push` | `int (*on_job_push)(void* picker, xce_sched_job_t* job)` | Add new pending job to picker's candidate set | `JOBS_PUSH` event | + | `on_job_state_change` | `int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t new_state)` | Remove job from candidates if no longer pending; re-add if returning to pending | `JOB_STATE_CHANGE` event | + | `on_job_update` | `int (*on_job_update)(void* picker, xce_sched_job_t* job)` | Reorder job in candidates if priority/queue changed | `JOB_UPDATE` event | + + **Session hooks** (MUST — called at round boundaries): + + | Operation | Signature | Semantics | + |-----------|-----------|-----------| + | `begin_session` | `int (*begin_session)(void* picker)` | Reset iteration cursor; called at start of each scheduling round | + | `next_job` | `xce_sched_job_t* (*next_job)(void* picker)` | Return next schedulable job per picker's ordering, or NULL when exhausted | + | `end_session` | `int (*end_session)(void* picker)` | End-of-round notification; called before core sleeps between rounds | + + **Post-allocation hooks** (MUST — called after allocation attempt): + + | Operation | Signature | Semantics | + |-----------|-----------|-----------| + | `on_dispatch` | `int (*on_dispatch)(void* picker, uint64_t jobid)` | Job was allocated; remove from candidates permanently | + | `on_pend` | `int (*on_pend)(void* picker, uint64_t jobid)` | Job was pended; keep for next round but may deprioritize | + + **Optional hooks** (MAY — NULL if not needed): + + | Operation | Signature | Semantics | When needed | + |-----------|-----------|-----------|------------| + | `on_config` | `int (*on_config)(void* picker, xce_conf4sched_t* config)` | Rebuild picker strategy from config | Config-driven pickers (deadline, fair-share weights) | + | `on_job_rusage` | `int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages)` | Track per-job resource usage | Resource-aware pickers (fair-share, backfill) | + + Hooks not defined on the picker interface (worker updates, resource updates, resval updates) are not forwarded to the picker — picking is about job ordering, not resource availability. + +3. The core shall call state-change hooks synchronously during `core_push_event()` processing, before the event returns. Session hooks and post-allocation hooks are called synchronously during the scheduling session (between `BEGIN_ROUND` and session exhaustion). The picker runs in the core's scheduling thread — no dedicated picker thread. + +4. The core scheduling loop shall: + + ```c + begin_session(); + while ((job = picker->next_job(picker)) != NULL) { + if (try_allocate(job) > 0) { + picker->on_dispatch(picker, job->jobid); + emit_decision(DISPATCH, job); + } else if (can_preempt(job)) { + emit_decision(PREEMPT, job, victims); + picker->on_pend(picker, job->jobid); // retry next round + } else { + emit_decision(PEND, job); + picker->on_pend(picker, job->jobid); + } + } + end_session(); + ``` + + The core loop shall contain zero knowledge of project/queue/job ordering — all selection logic lives behind `next_job()`. + +5. The picker plugin shall be cancel-safe: `next_job()` shall have defined `pthread_cancel` cancellation points. The picker shall be session-aware: `begin_session()` resets iteration state, `end_session()` is called before sleep-between-rounds. + +6. The hardcoded `"xce_sched_fcfs"` literal at `scheduler.c:43` (in `sched_job_picker()`) shall be replaced by loading the configured `sched->job_pick_plugin` via the existing `plugin_register`/`key_find` mechanism. If the configured plugin name is not registered, the Scheduler shall fail with a clear configuration error (same pattern as Req 5 AC 5). + +7. `sched_job_picker()`'s dedicated thread and `jbuf` cbuf communication shall be retired. The picker runs synchronously within the core's scheduling thread — `next_job()` is called directly, not via a separate thread + cbuf. This eliminates one thread and simplifies the session lifecycle for simulation mode. ### Requirement 2: Lazy allocation plan iterator @@ -90,20 +278,22 @@ These are fundamentally different concerns: 4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected. 5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents — the current stub behavior — for any job with `maxTasks > 1`. -### Requirement 5: Configurable picker plugin slots +### Requirement 5: Configurable picker plugin -**Objective:** As a cluster operator, I want to configure job-picking policies at the scheduler, queue, and project levels without recompiling the scheduler, so that FCFS, priority, or custom pickers can be swapped operationally and composed. +**Objective:** As a cluster operator, I want to configure which job-picking policy the scheduler uses without recompiling, and override it per-queue, so that FCFS, priority, or custom pickers can be swapped operationally. #### Acceptance Criteria -1. The Scheduler shall add a `project_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_flat"` when unset. -2. The Scheduler shall add a `queue_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_strict_priority"` when unset. -3. The Scheduler shall add a `job_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_fcfs"` when unset — serving as the global fallback. -4. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_queue_t`, defaulting to NULL (inherit from project or global). -5. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_project_t`, defaulting to NULL (inherit from global). -6. The Scheduler shall add a `project_name` field to `xce_sched_queue_t`, set from queue configuration, establishing the queue→project relationship for Level 1 scoping. -7. The Scheduler shall accept all picker plugin names from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild. -8. When a plugin slot names a plugin not registered via the existing `plugin_register`/`key_find` mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default. -9. The Scheduler shall document the picker plugin interface contract: each picker must be cancel-safe (responds to `pthread_cancel` at defined points) and session-aware (resets its state on `end_session`). +1. The Scheduler shall add a `job_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_fcfs"` when unset — this is the global picker plugin name. + +2. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_queue_t`, defaulting to NULL (inherit from global). When non-NULL, the configured global picker plugin shall use this per-queue override for jobs within that queue. How the picker plugin uses per-queue overrides is an implementation detail of the plugin, not the core loop. + +3. The Scheduler shall add a `project_name` field to `xce_sched_queue_t`, set from queue configuration, establishing the queue→project relationship. The picker plugin may use this field internally for project-scoped iteration. + +4. The Scheduler shall accept the picker plugin name from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild. + +5. When the configured plugin name is not registered via the existing `plugin_register`/`key_find` mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default. + +6. The picker plugin interface contract (`xce_job_picker_iface_t`) shall be documented: each picker must be cancel-safe (responds to `pthread_cancel` at defined points in `next_job()`) and session-aware (resets its iteration state on `begin_session()`). ### Requirement 6: Preemption policy and execution @@ -131,15 +321,58 @@ These are fundamentally different concerns: 4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption. 5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped. -### Requirement 8: Builtin default plugins +### Architectural Support for Future Object Sharing (Deferred) -**Objective:** As a maintainer, I want the new architecture to ship with builtin plugins that replicate today's exact scheduling behavior, so that operators who don't configure custom pickers see no behavioral change. +Full object sharing (resreq template registry, pend reason dedup pool) is deferred to a follow-up spec. However, the architecture designed in this spec must not block sharing from being added later. The following design invariants apply to all resreq, pend-reason, and related objects touched by this spec: + +1. **Resreq immutability**: `xce_primary_resreq_t`, `xce_compound_resreq_t`, and `xce_alter_resreq_t` objects shall be treated as immutable once created and validated. No code path shall mutate a resreq in place. Any modification (e.g., expanding a job-array template into per-element resreqs) shall create a new copy. This ensures that sharing via `const` pointer is safe to introduce later without auditing every consumer for mutation. + +2. **Pend reason factory**: All pend reason allocation shall go through `make_pendreason()` (already the single factory in `lib/preason.c`). Callers never `calloc(sizeof(xce_pendreason_t))` directly. This ensures a session-level dedup pool can be added inside the factory later without changing any caller. + +3. **Pointer-identity fast path**: Code that compares two resreqs or two pend reasons shall use pointer equality (`a == b`) as a fast path before falling back to deep comparison. When a resreq registry is introduced, identical resreqs will have the same pointer — the fast path avoids repeated deep comparison for no runtime cost today. + +4. **Select expression memoization**: The existing `select_eval_ctx_t` per-session memoization of parsed select-expression ASTs shall be preserved and extended to the core's internal resreq objects. The parsed/evaluated form of a select expression shall be attached to the resreq on first evaluation and reused for subsequent evaluations (same session, same resreq pointer). + +5. **Snapshot forward-compatibility**: The snapshot format shall not preclude object sharing. Shared objects in the snapshot shall be serializable by identity (an object table with IDs) rather than requiring inline embedding at every reference site. This is a design-phase concern but constrains the JSON schema choice. + +These invariants impose zero overhead in the current implementation — immutability is already the pattern, pointer-identity fast paths are already valid C, and the factory already exists. They simply rule out design choices that would make sharing expensive or impossible to retrofit. + +### Requirement 8: Builtin default picker + +**Objective:** As a maintainer, I want the new architecture to ship with a builtin picker plugin that replicates today's exact scheduling behavior, so that operators who don't configure a custom picker see no behavioral change. #### Acceptance Criteria -1. **`builtin_flat`** (Level 0, project picker): shall yield exactly one "default" project scope per scheduling round. All queues not explicitly assigned to a project belong to this default project. Always eligible; never exhausted during a session. -2. **`builtin_strict_priority`** (Level 1, inter-queue picker): shall iterate queues in descending priority order (`queue->priority`). Shall skip queues with no pending jobs (`sched_queue_is_ready` returns false). When scoped to a project, shall only iterate queues belonging to that project. Shall return NULL when all eligible queues are exhausted. -3. **`builtin_fcfs`** (Level 2, intra-queue picker): shall replicate today's FCFS behavior — iterate pending jobs within the queue by `priority desc, queue_time asc`. Shall skip jobs where `sched_job_is_ready` returns false. Shall communicate the picked jobid via the existing `jbuf` cbuf mechanism. Shall be cancel-safe and session-aware. -4. All three builtin plugins shall be registered automatically at scheduler startup without requiring explicit configuration. + +1. **Registration**: `builtin_fcfs` shall be registered automatically at scheduler startup as `"builtin_fcfs"` and set as the default `sched->job_pick_plugin`. + +2. **Hook implementation** — `builtin_fcfs` shall implement every MUST and SHOULD hook from `xce_job_picker_iface_t` (Req 1 AC 2) as follows: + + | Hook | FCFS Implementation | + |------|-------------------| + | `init` | Allocate `fcfs_plugin_data_t`. Initialize `ordq_t queues` (ordered by `queue->priority desc`). Set iteration cursor to NULL. | + | `fini` | Free all per-queue `ordq_t pendJobs`, the `ordq_t queues`, and plugin data. | + | `on_queue_update` | Upsert queue into `ordq_t queues`. If queue exists, update its priority and reorder. If new, allocate `fcfs_queue_t` with empty `ordq_t pendJobs`. | + | `on_job_push` | If `job->state & JOB_STAT_PEND`: find the job's queue in `ordq_t queues`, add `fcfs_job_t{job}` to that queue's `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). | + | `on_job_state_change` | If `new_state` is not pending (EXIT/FAIL/DONE): remove job from its queue's `pendJobs` by jobid. If `new_state` returns to PEND: re-add to `pendJobs`. | + | `on_job_update` | If job's `priority` or `queue` field changed: remove from old position in `pendJobs`, re-insert at correct position per ordering. | + | `begin_session` | Reset iteration cursor to head of `ordq_t queues`. | + | `next_job` | Walk `ordq_t queues` from cursor. For each queue, skip if `sched_queue_is_ready()` returns false. Within queue, walk `ordq_t pendJobs`. Return first job where `sched_job_is_ready()` returns true. Advance cursor. Return NULL when all queues and jobs exhausted. | + | `end_session` | No-op for FCFS (iteration state reset in `begin_session`). | + | `on_dispatch` | Remove job from its queue's `ordq_t pendJobs` by jobid. | + | `on_pend` | No-op for FCFS — job remains in `pendJobs`, will be retried next round in same position. | + | `on_config` | NULL — FCFS is config-agnostic. | + | `on_job_rusage` | NULL — FCFS is not resource-aware. | + +3. **Internal data structures** — `builtin_fcfs` shall maintain its own private state, not query the core's `sched->queues`/`sched->jobs` during `next_job()`. All candidate management happens in the state-change hooks. This ensures: + - `next_job()` is fast — no core lock acquisition, no hash table scans + - The picker is independently testable — feed hooks, verify `next_job()` order + - Simulation snapshots capture picker state naturally via the hooks already recorded in the event stream + +4. **Ordering equivalence** — `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`: inter-queue by descending `queue->priority`, intra-queue by `priority desc, queue_time asc`. Behavior with per-queue picker overrides is as specified in Req 5 AC 2. + +5. **No dedicated thread** — `builtin_fcfs` shall implement `next_job()` as a synchronous call, not a dedicated thread + `jbuf` cbuf. The picker runs in the core's scheduling thread. `pthread_cancel` cancellation points shall exist in the `next_job()` iteration loop. + +6. **Session state** — `end_session()` is a no-op. Iteration state is reset in `begin_session()`. Undispatched jobs remain in `pendJobs` across sessions (they carry over to the next round). ### Requirement 9: No regression to existing allocators, job picker, or HA soak @@ -147,27 +380,162 @@ These are fundamentally different concerns: #### Acceptance Criteria 1. `seqjob_allocator`'s externally observable allocation behavior (which host a `maxTasks == 1` job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the `xce_host_iter_t` interface. -2. `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`, given the same queue and job state. -3. `builtin_strict_priority` shall produce the same inter-queue order as today's `fcfs_elect_job` queue iteration loop. -4. `builtin_flat` shall produce the same effective behavior as today's lack of project scoping — all queues in one flat namespace. -5. The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today. -6. Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string or eager-one-shot behavior this feature replaces. -7. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. +2. `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`, given the same queue and job state — same inter-queue priority ordering and same intra-queue (priority desc, queue_time asc) ordering. +3. The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today. +4. Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string, dedicated picker thread, `jbuf` cbuf, or eager-one-shot behavior this feature replaces. +5. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from. + +### Requirement 10: Core/Driver Separation with Dual-Mode Operation + +**Objective:** As a scheduler developer, I want the scheduling core separated from all external I/O so that the same core logic runs deterministically in production (driven by a live controller) and in simulation (driven by recorded cluster events with a discrete event-driven virtual clock), enabling policy debugging, tuning, regression testing, and what-if analysis without a live cluster. + +#### Acceptance Criteria + +**Core Interface Contract** + +1. **Mode-agnostic core**: The Scheduler core shall have no concept of which mode it runs in. It processes events identically whether the driver is in normal mode (controller-connected, wall-clock time) or simulation mode (replay file, virtual clock). Specifically: + - There shall be no mode flag (`bool simulation_mode` or equivalent) anywhere in the core + - There shall be no `if (simulation) { ... } else { ... }` code path in the core + - `xce_sched_core_init()` shall take identical parameters regardless of mode (no mode-specific init arguments) + - The core's decision callback shall be the same function pointer type in both modes — the driver decides what to do with decisions (publish to controller vs. log to file) + - The core's `save_snapshot()` and `load_snapshot()` shall be identical in both modes — they serialize/deserialize core state, not driver state + +2. **Passive event-driven interface**: The core shall expose a pure event-driven interface — it receives typed events from the driver, updates internal scheduling state, and emits decisions through a driver-registered callback. The core shall have zero direct dependency on: controller communication, capnp serialization, network I/O, filesystem access, or wall-clock time sources. + +3. **Virtualized clock access**: The core shall access time exclusively through `xce_core_now()`. The core shall never call `time(NULL)`, `clock_gettime()`, `gettimeofday()`, or any other wall-clock time function directly. + + **Clock virtualization mechanism — two-base + speed formula:** + + The core stores three fields (atomically updated by the driver as a group): + + | Field | Type | Description | + |-------|------|-------------| + | `virtual_clock_base` | `_Atomic uint64_t` | Virtual-time anchor (nanoseconds) | + | `real_clock_base` | `_Atomic uint64_t` | Wall-clock anchor captured when `virtual_clock_base` was set (nanoseconds, `CLOCK_MONOTONIC`) | + | `speed_num` / `speed_den` | `_Atomic uint64_t` | Speed multiplier as rational fraction | + + `xce_core_now(core)` computes: + ```c + uint64_t wall_ns = clock_gettime(CLOCK_MONOTONIC); // only wall-clock call in the core + uint64_t elapsed = wall_ns - core->real_clock_base; + return core->virtual_clock_base + (elapsed * core->speed_num) / core->speed_den; + ``` + + | Mode | `virtual_clock_base` | `real_clock_base` | `speed` | Behavior | + |------|----------------------|-------------------|---------|----------| + | Normal | 0 | 0 | 1/1 | `xce_core_now()` = wall clock (degraded: `0 + (wall - 0) × 1`) | + | Simulation discrete | event `ts` | wall at load | 0/1 | `xce_core_now()` = frozen at event `ts` (time does not advance between events) | + | Simulation real-time | snapshot `ts` | wall at load | 1/1 | `xce_core_now()` tracks wall clock from virtual anchor | + | Simulation fast/slow | snapshot `ts` | wall at load | N/1 or 1/N | `xce_core_now()` runs N× faster or slower than wall clock | + + The driver atomically updates all three fields before each event dispatch. In simulation discrete mode (speed=0/1), the clock jumps directly from one event `ts` to the next — no continuous advancement. In normal mode (bases=0, speed=1/1), the formula degrades to returning wall-clock time directly, with zero overhead beyond the single `clock_gettime()` call. + + **Multi-thread safety**: `xce_core_now()` reads atomic fields lock-free. Both picker and allocator threads independently compute the same virtual time from the shared bases — no mutex needed. The driver updates all three fields as an atomic group (via a single mutex or sequenced atomic stores with a version counter). + + - All internal time uses go through `xce_core_now()`: scheduling logic (job queue-time ordering, session boundaries, pend-reason timestamps, job aging), **logging** (log message timestamps via tlog format function), **error messages**, **debug traces** — any code path that needs the current time calls `xce_core_now()` + - Job `sub_time` and `queue_time` are set by the driver (from `xce_core_now()` in normal mode, from event `ts` in simulation mode) before pushing the `JOBS_PUSH` event — the core never generates timestamps for externally-visible fields + - This virtualization is what enables deterministic replay: the same event file produces the same `xce_core_now()` return values (and therefore same log timestamps, same scheduling decisions) regardless of wall-clock speed + - **Third-party library constraint**: any library linked into the scheduler core that needs the current time must obtain it through `xce_core_now()` or a core-provided wrapper — never by calling `time()`, `clock_gettime()`, `gettimeofday()`, or similar directly. Specifically: + - **Logging (tlog)**: The `tlog` library internally captures `struct tlog_time` from wall-clock time before calling the registered format callback (`my_tlog_format`). The core shall register its own format function via `tlog_reg_format_func(fn, core_ptr)` where `core_ptr` is the `xce_sched_core_t*`. This format function shall ignore `info->time` entirely and instead read `xce_core_now(core_ptr)`, converting nanoseconds to `struct tm` fields via `gmtime_r()` (deterministic — pure computation from an integer, no system clock access). The `userptr` parameter of `tlog_reg_format_func()` carries the core pointer into the format callback. + - **libxce subset**: The core shall link only against the minimal data-structure subset of libxce (`ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti`) — not against `netio`, `delay`, `xceapi`, or `idgen` which call wall-clock time internally + - **pthread timed-blocking**: The core is multi-threaded (picker + allocator threads). In simulation mode, thread interleaving is made deterministic per AC 6 — `pthread_cond_timedwait()` and `pthread_mutex_timedlock()` are not needed in the deterministic execution path + - If a library cannot avoid calling wall-clock time directly (e.g., a third-party compression or serialization library that internally timestamps), it must not be linked into the core; such concerns stay in the driver layer + +4. **Core event types**: The core interface shall accept at minimum these input events: + + | Event Type | Payload | Description | + |------------|---------|-------------| + | `CONFIG_PUSH` | `xce_conf4sched_t*` | Full configuration push (startup or config change) | + | `JOBS_PUSH` | `int njobs, xce_job_t**` | Newly submitted jobs | + | `WORKERS_PUSH` | `int nworkers, xce_worker_t**` | Worker list update | + | `RESOURCES_PUSH` | `int n, xce_resource_t**` | Resource definitions | + | `RESVALS_FULL` | `int n, xce_resval_t**` | Full resource value snapshot | + | `RESVALS_DELTA` | `xce_resval_list_t*` | Incremental resource value update | + | `JOB_STATE_CHANGE` | `uint64_t jobid, uint32_t state` | Job started / exited / failed | + | `JOB_UPDATE` | `xce_job_t*` | In-place job field update (notify path) | + | `QUEUE_UPDATE` | `xce_queue_t*` | Queue definition update | + | `JOB_RUSAGE` | `xce_job_rusages_t*` | Job resource usage update | + | `BEGIN_ROUND` | `uint64_t timestamp_ns` | Trigger scheduling round at given logical time | + +5. **Decision output callback**: The core shall emit all scheduling decisions (`DISPATCH`, `PEND`, `PREEMPT`) through a `xce_core_decision_cb_t` callback function pointer registered by the driver at core init time, replacing the current internal `decisions` cbuf + `main.c` poll loop pattern. The callback signature shall be `int (*on_decision)(void* driver_ctx, xce_decision_t* decision)`. + +6. **Deterministic guarantee**: Given identical initial state and identical event sequences with identical timestamps, the core shall produce identical decision sequences. The core is multi-threaded (picker thread + allocator thread, matching today's architecture). To achieve deterministic replay: + - All hash table / key-value store iteration that affects scheduling output order shall be stabilized (sorted by key before iterating) + - Any required randomness (e.g., tie-breaking) shall use a PRNG seeded by the driver at init time + - In simulation mode, thread interleaving shall either be made deterministic (e.g., the simulation driver serializes event dispatch and decision collection through a single-threaded core path), or the core's threads shall synchronize on a simulation barrier that advances in lock-step with the virtual clock + +7. **Driver event queue**: The driver shall retain an internal pending-event queue (the existing `xce_sched_update_request_t` / `pend_updates` mechanism, or equivalent) to buffer events received from the controller between scheduling rounds. Events accumulate in the driver between `BEGIN_ROUND` invocations. When the core receives `BEGIN_ROUND`, it drains all buffered events, processes a scheduling session, and emits decisions. + +**Normal-Mode Driver** + +8. **Controller integration**: The normal-mode driver shall receive capnp push/notify messages from the controller (via the existing `scheduler_req_handler` callback), translate each to the corresponding core event type, and invoke the core's event push interface. The driver shall publish core-emitted decisions back to the controller via the existing `xcecli_ctx_t` API (`sched_publish_dispatch` / `sched_publish_pend_reason` semantics, generalized to include `PREEMPT`). + +9. **Runtime recording control via xceadmin**: The scheduler shall support two new xceadmin commands: + - `scheduler record start [FILE]` — take a full state snapshot, write it to the replay file, then begin recording all subsequent core input events and output decisions to the file + - `scheduler record stop` — stop recording, close the replay file, resume normal operation + + Recording shall have zero impact on scheduling correctness — the core processes identically whether recording is active or not. + +10. **Periodic snapshot during recording**: While recording is active, the driver shall periodically append fresh snapshots to the replay file (default: every `N` events, configurable via `--record-snapshot-interval N`, where N=0 disables periodic snapshots and records only the initial snapshot). This bounds replay time for any desired start point. + +**Simulation-Mode Driver** + +11. **Simulation mode startup**: The scheduler shall support a simulation mode activated by `--sim FILE` (replay file path) and `--sim-start-ts TIMESTAMP` (logical time to begin replay, default: start of file). In simulation mode, the driver loads the most recent snapshot with `ts <= --sim-start-ts`, then replays all delta events from that point forward. + +12. **Discrete event-driven virtual clock**: The simulation driver shall implement a virtual clock that advances by jumping directly to the timestamp of the next queued event — no fixed tick interval, no wall-clock waiting between events. The clock shall be strictly monotonically increasing. When multiple events share the same timestamp, they shall be delivered in file order. The driver shall support: + - `--sim-speed N` — rate limit: N=0 means "as fast as possible" (no delay), N=1 means real-time, N>1 means N× real-time + - `--sim-pause-at TIMESTAMP` — pause simulation at a specific logical time for interactive debugging + +13. **Deterministic replay verification**: In simulation mode with `--sim-verify`, the driver shall compare each core-emitted decision against the recorded decisions in the replay file. Any mismatch (missing, extra, or different decision) shall be reported as a regression with exact logical-time and decision-detail context. + +14. **What-if branching**: The simulation driver shall accept `--sim-alt-events FILE` (alternative delta event file). When both a base replay file (for snapshot loading) and an alt-events file are provided, the driver loads the snapshot from the base file, then replays delta events from the alt-events file — enabling "what-if" analysis (e.g., "what if job X had been submitted with higher priority?"). + +**Replay File Format** + +15. **JSONL format**: The replay file shall be newline-delimited JSON (JSONL), one event per line: + + ```jsonl + {"type":"HEADER","version":"x.y.z","config_hash":"sha256...","start_ts":1234567890000000000,"start_wall":"2026-07-27T12:00:00Z"} + {"type":"SNAPSHOT","ts":1234567890000000000,"data":{...full core state...}} + {"type":"CONFIG_PUSH","ts":1234568000000000000,"payload":{...}} + {"type":"JOBS_PUSH","ts":1234568100000000000,"payload":{"njobs":2,"jobs":[...]}} + {"type":"BEGIN_ROUND","ts":1234568200000000000,"payload":{}} + {"type":"DECISION","ts":1234568200000000000,"payload":{"op":"DISPATCH","jobid":"..."}} + ``` + + - `type`: event type name string + - `ts`: nanosecond logical timestamp (monotonic, unsigned 64-bit integer) + - `payload`: event-type-specific object + - `HEADER` line: first line of every file, carries scheduler version and configuration hash for compatibility checking + - `SNAPSHOT` lines: embed full core state in `data` (plain JSON, not base64-encoded — debuggable and directly extractable) + - `DECISION` lines: recorded decisions (output), present only in files produced by the normal-mode driver's recording feature — the simulation driver ignores them unless `--sim-verify` is active + - The event payload JSON schema is an open design question deferred to the design phase + +**State Snapshot** + +16. **Snapshot serialization**: The core shall support `save_snapshot()` producing a full state dump (plain JSON) containing all config, queues (with per-queue picker/preemption settings), jobs (with state, priority, queue-time, resreq), workers (with available/used resources), resources, resvals, users, projects, shared-resource tracking state, unsatisfiable-resreq cache, and session state. The core shall support `load_snapshot()` restoring to that exact state, producing a core that is indistinguishable from one that reached that state by replaying all events up to that timestamp. + +**Interaction with Existing Requirements** + +Reqs 1–9 describe logic **inside** the core — the picker loop, iterators, parjob allocator, preemption policy, resreq cache, and builtin plugins all operate within the core's event-driven boundary. Requirement 10 defines the boundary itself. Specifically: + +- Req 5 (picker slots from config): config arrives via `CONFIG_PUSH` event, not via capnp decode inside the core +- Req 6 (preemption → controller): `XCE_DECISION_PREEMPT` is emitted through the decision callback, not sent directly to controller +- Req 7 (resreq cache, session lifecycle): `BEGIN_ROUND` replaces the implicit `sleep(1)`-delimited session boundary in `main.c`; the cache still lives on `xce_sched_core_t` and is cleared at session start +- Req 9 (HA soak regression): the same core code can be validated in simulation mode against recorded HA failover traces, catching regressions without a live cluster ## Data Structure Changes Summary ### `xce_scheduler_t` additions ```c -char* project_pick_plugin; // default: "builtin_flat" -char* queue_pick_plugin; // default: "builtin_strict_priority" -char* job_pick_plugin; // default: "builtin_fcfs" (global fallback) +char* job_pick_plugin; // default: "builtin_fcfs" (Req 5) // unsatisfiable resreq cache (Req 7) ``` ### `xce_sched_queue_t` additions ```c -char* project_name; // owning project (for Level 1 scoping) -char* job_pick_plugin; // intra-queue picker (NULL = inherit) +char* project_name; // owning project (picker uses internally) +char* job_pick_plugin; // per-queue picker override (NULL = use global) char** preemptible_queues; // queues this queue CAN preempt from int n_preemptible_queues; bool preemptible; // CAN this queue's jobs BE preempted? @@ -176,11 +544,23 @@ char* preempt_policy; // victim selection plugin (NULL = default) ### `xce_sched_project_t` additions ```c -char* job_pick_plugin; // per-project picker (NULL = inherit, v1 field, v2 logic) int32_t priority; // inter-project ordering (future spec) // future: resource_quota, resource_used, parent_project (project-resource-ownership spec) ``` +### `xce_job_picker_iface_t` (new, Req 1) +```c +typedef struct { + int (*init)(void* picker, xce_sched_core_t* core); + void (*fini)(void* picker); + int (*begin_session)(void* picker); + xce_sched_job_t* (*next_job)(void* picker); + int (*end_session)(void* picker); + int (*on_dispatch)(void* picker, uint64_t jobid); + int (*on_pend)(void* picker, uint64_t jobid); +} xce_job_picker_iface_t; +``` + ### `decision.h` additions ```c typedef enum { @@ -199,43 +579,274 @@ typedef struct { } xce_preempt_decision_t; ``` +### Core/Driver Separation (Req 10) + +```c +// Core event (driver → core) +typedef enum { + XCE_CORE_EVENT_CONFIG_PUSH = 0, + XCE_CORE_EVENT_JOBS_PUSH, + XCE_CORE_EVENT_WORKERS_PUSH, + XCE_CORE_EVENT_RESOURCES_PUSH, + XCE_CORE_EVENT_RESVALS_FULL, + XCE_CORE_EVENT_RESVALS_DELTA, + XCE_CORE_EVENT_JOB_STATE_CHANGE, + XCE_CORE_EVENT_JOB_UPDATE, + XCE_CORE_EVENT_QUEUE_UPDATE, + XCE_CORE_EVENT_JOB_RUSAGE, + XCE_CORE_EVENT_BEGIN_ROUND, +} xce_core_event_type_t; + +typedef struct { + xce_core_event_type_t type; + uint64_t ts; // driver-supplied logical time (ns) + union { + xce_conf4sched_t* config; + struct { int njobs; xce_job_t** jobs; } job_push; + struct { int nworkers; xce_worker_t** workers; } worker_push; + struct { int n; xce_resource_t** resources; } resource_push; + struct { int n; xce_resval_t** vals; } resval_full; + xce_resval_list_t* resval_delta; + struct { uint64_t jobid; uint32_t state; } job_state; + xce_job_t* job_update; + xce_queue_t* queue_update; + xce_job_rusages_t* job_rusage; + // BEGIN_ROUND has no payload beyond ts + } payload; +} xce_core_event_t; + +// Decision callback (core → driver) +typedef int (*xce_core_decision_cb_t)(void* driver_ctx, xce_decision_t* decision); + +// Core struct (pure scheduling state, no I/O) +typedef struct { + // Scheduling state (moved from xce_scheduler_t) + xce_conf4sched_t* config; + uint64_t configver; + var plugins; + void* seqjob_allocator; + void* parjob_allocator; + void* affinity_allocator; + void* gpu_allocator; + var resmgr, ugmgr, qgmgr, pgmgr, hgmgr; + zarray_t sharedInst, sharedInUse; + var jobs, queues, workers, users, projects; + uint64_t nextid; + + // New fields for core/driver boundary + void* driver_ctx; // opaque driver context for callback + xce_core_decision_cb_t on_decision; // decision output callback + // Clock virtualization (Req 10 AC 3) — atomically updated by driver + _Atomic uint64_t virtual_clock_base; // virtual-time anchor (ns) + _Atomic uint64_t real_clock_base; // wall-clock anchor (ns, CLOCK_MONOTONIC) + _Atomic uint64_t speed_num; // speed numerator + _Atomic uint64_t speed_den; // speed denominator + uint64_t prng_seed; // seeded by driver at init + + // Picker plugin slot (Req 5) + char* job_pick_plugin; // default: "builtin_fcfs" + // per-queue overrides stored on xce_sched_queue_t.job_pick_plugin, + // read by the picker internally — not separate core slots + + // Unsatisfiable resreq cache (Req 7) + // ... (session-level fingerprint cache) + + // Session state + _Atomic int session_active; + _Atomic int eval_in_progress; + select_eval_ctx_t* sel_eval_ctx; +} xce_sched_core_t; + +// Normal-mode driver +typedef struct { + xce_sched_core_t core; + xcecli_ctx_t apictx; // controller API client + xce_cluster_config_t cluster_cfg; // HA candidate list + dataq_t* pend_updates[2]; // event buffer between rounds + // Recording state + FILE* record_file; + bool recording; + uint64_t record_snapshot_interval; + uint64_t events_since_snapshot; +} xce_sched_driver_t; + +// Simulation-mode driver +typedef struct { + xce_sched_core_t core; + FILE* replay_file; + uint64_t virtual_clock; // current logical time (ns) + uint64_t sim_speed; // 0=unlimited, 1=real-time, N=N× + uint64_t pause_at_ts; // 0=don't pause + bool verify; // compare decisions vs recorded + FILE* alt_events_file; // for what-if branching +} xce_sched_sim_driver_t; + +// Core interface +int xce_sched_core_init(xce_sched_core_t* core, xce_core_decision_cb_t cb, + void* driver_ctx, uint64_t prng_seed); +int xce_sched_core_push_event(xce_sched_core_t* core, xce_core_event_t* event); +uint64_t xce_core_now(xce_sched_core_t* core); // virtualized clock accessor +int xce_sched_core_save_snapshot(xce_sched_core_t* core, FILE* out); +int xce_sched_core_load_snapshot(xce_sched_core_t* core, FILE* in); +void xce_sched_core_fini(xce_sched_core_t* core); +``` + ## Architecture Diagram +### Core Scheduling Loop + ``` ┌──────────────────────────────────────────────────────────┐ -│ Level 0: Project Picker [sched->project_pick_plugin] │ +│ SCHEDULER CORE LOOP │ │ │ -│ v1 (builtin_flat): yields one "default" project │ -│ v2 (future): quota-aware project selection │ -└──────────────────────┬───────────────────────────────────┘ - │ project scope - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │ -│ │ -│ Iterates queues scoped to the selected project │ -│ Default: strict_priority (descending queue->priority) │ -└──────────────────────┬───────────────────────────────────┘ - │ queue - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Level 2: Intra-Queue Picker [queue→project→sched] │ -│ │ -│ Picks one job from the selected queue │ -│ Default: builtin_fcfs (priority desc, queue_time asc) │ -└──────────────────────┬───────────────────────────────────┘ - │ job - ▼ -┌──────────────────────────────────────────────────────────┐ -│ Allocation + Preemption │ -│ │ -│ plan_iter → host_iter → try_alloc │ -│ success → dispatch │ -│ fail → Level 3: preempt? → select victims → SIGSTP │ -│ → pend (no preemption possible) │ +│ begin_session() │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────┐ │ +│ │ picker->next_job() │ ←─ plugin decides HOW │ +│ │ returns job or NULL │ (FCFS, priority, fair-share, │ +│ └──────────┬───────────┘ deadline, ...) │ +│ │ │ +│ job │ NULL (exhausted) │ +│ ▼ ▼ │ +│ ┌──────────────────┐ end_session() → sleep → next round │ +│ │ try_allocate(job) │ │ +│ └────────┬─────────┘ │ +│ │ │ +│ success │ fail │ +│ ▼ ▼ │ +│ ┌────────────┐ ┌────────────────┐ │ +│ │ DISPATCH │ │ can_preempt? │ │ +│ │ on_dispatch│ │ yes → PREEMPT │ │ +│ └────────────┘ │ no → PEND │ │ +│ │ on_pend │ │ +│ └────────────────┘ │ +│ │ │ +│ ▼ │ +│ next_job() (loop continues) │ └──────────────────────────────────────────────────────────┘ ``` +### `builtin_fcfs` Internal Design + +The default picker plugin internally implements the equivalent of today's selection logic. This is **not** part of the core loop — it's the plugin's private implementation: + +``` +builtin_fcfs.next_job(): + ┌────────────────────────────────────────────┐ + │ Iterate queues by priority (descending) │ + │ Skip queues with no ready jobs │ + │ │ + │ For each queue: │ + │ If queue has per-queue picker override: │ + │ Delegate to that picker │ + │ Else: │ + │ Iterate jobs by priority desc, │ + │ queue_time asc (FCFS) │ + │ Skip jobs not ready │ + │ Return first ready job │ + │ │ + │ Return NULL if all queues exhausted │ + └────────────────────────────────────────────┘ +``` + +## Core/Driver Architecture (Req 10) + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ DRIVER LAYER │ +│ │ +│ ┌──────────────────────┐ ┌────────────────────────────┐ │ +│ │ NORMAL MODE │ │ SIMULATION MODE │ │ +│ │ │ │ │ │ +│ │ controller capnp ────┐│ │ replay.jsonl ──────────┐ │ │ +│ │ (scheduler_req_ ││ │ (event file) │ │ │ +│ │ handler) ││ │ │ │ │ +│ │ │ ││ │ │ │ │ │ +│ │ ▼ ││ │ ▼ │ │ │ +│ │ ┌──────────────┐ ││ │ ┌──────────────┐ │ │ │ +│ │ │ event queue │ ││ │ │ virtual clock│ │ │ │ +│ │ │ (pend_updates)│ ││ │ │ (discrete │ │ │ │ +│ │ └──────┬───────┘ ││ │ │ event-driven)│ │ │ │ +│ │ │ ││ │ └──────┬───────┘ │ │ │ +│ │ │ events ││ │ │ events │ │ │ +│ │ │ + BEGIN_ROUND││ │ │ + BEGIN_ROUND │ │ │ +│ │ ▼ ││ │ ▼ │ │ │ +│ │ ┌──────────────┐ ││ │ ┌──────────────┐ │ │ │ +│ │ │ publish │ ││ │ │ record/log │ │ │ │ +│ │ │ decisions to │ ││ │ │ decisions │ │ │ │ +│ │ │ controller │ ││ │ │ (--sim-verify)│ │ │ │ +│ │ └──────▲───────┘ ││ │ └──────▲───────┘ │ │ │ +│ │ │ decisions ││ │ │ decisions │ │ │ +│ └─────────┼─────────────┘│ └─────────┼───────────────┘ │ +│ │ │ │ │ +│ │ xce_core_decision_cb_t │ │ +│ │ (callback) │ │ +└────────────┼──────────────────────────────────┼───────────────────┘ + │ │ + ▼ ▼ +┌──────────────────────────────────────────────────────────────────┐ +│ SCHEDULER CORE │ +│ (xce_sched_core_t) │ +│ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ xce_sched_core_push_event(core, event) │ │ +│ │ │ │ +│ │ Event types: CONFIG_PUSH, JOBS_PUSH, WORKERS_PUSH, │ │ +│ │ RESOURCES_PUSH, RESVALS_FULL, RESVALS_DELTA, │ │ +│ │ JOB_STATE_CHANGE, JOB_UPDATE, QUEUE_UPDATE, │ │ +│ │ JOB_RUSAGE, BEGIN_ROUND │ │ +│ └──────────────────────────┬──────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ Internal Scheduling Logic (Reqs 1–9) │ │ +│ │ │ │ +│ │ picker->next_job() → try_allocate(job) │ │ +│ │ │ │ │ │ +│ │ │ xce_alloc_plan_iter_t │ │ │ +│ │ │ → xce_host_iter_t │ │ │ +│ │ │ → try alloc │ │ │ +│ │ │ │ │ │ +│ │ │ success → DISPATCH │ │ │ +│ │ │ fail → PREEMPT or PEND │ │ │ +│ │ │ → picker->on_dispatch() │ │ │ +│ │ │ → picker->on_pend() │ │ │ +│ │ │ │ │ │ +│ │ Preemption (if alloc fails): │ │ +│ │ select victims → SIGSTP → pend job → retry next round │ │ +│ │ Resreq fingerprint cache (per-session) │ │ +│ └──────────────────────────┬──────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────┐ │ +│ │ on_decision(driver_ctx, decision) │ │ +│ │ DISPATCH / PEND / PREEMPT │ │ +│ └─────────────────────────────────────────────────────────────┘ │ +│ │ +│ ZERO dependencies on: controller, capnp, network, wall-clock │ +└──────────────────────────────────────────────────────────────────┘ +``` + +### Simulation Replay Sequence + +``` + load snapshot replay delta events + │ │ + ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ HEADER │───▶│ SNAPSHOT │───▶│ EVENT @t1 │───▶│ EVENT @t2 │───▶ ... + └──────────┘ └──────────┘ └──────────┘ └──────────┘ + │ │ │ + ▼ ▼ ▼ + restore core push + BEGIN_ push + BEGIN_ + state via ROUND(t1) → ROUND(t2) → + load_snapshot emit decisions emit decisions + + Virtual clock: t_snap ────────▶ t1 ───────────▶ t2 ───────────▶ + (discrete jumps, no wall-clock delay, monotonic) +``` + ## Preemption Flow ``` @@ -261,15 +872,15 @@ XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt( J pended → next scheduling round → allocation retries on freed resources ``` -## Future: Project Resource Ownership (Level 0 full) +## Future: Project Resource Ownership Deferred to a separate `project-resource-ownership` spec. When implemented: - Projects form a tree with resource quota at each node -- Level 0 project picker selects which project gets resources (fair-share, priority, proportional) -- Level 1 queue picker is scoped to the selected project's queues -- Level 2 job picker sees only jobs from the selected project within each queue -- Project quota enforcement: projects over quota are skipped by Level 0 -- The unified picker loop (Requirement 1) requires zero code changes — only a new project picker plugin replaces `builtin_flat` +- A project-aware picker plugin selects which project gets resources (fair-share, priority, proportional), then iterates queues scoped to that project, then jobs within each queue +- Project quota enforcement: projects over quota are skipped by the picker +- The core scheduling loop (Requirement 1) requires zero code changes — only a new picker plugin replaces `builtin_fcfs`. The core still calls `picker->next_job()` with no knowledge of projects. +- `queue->project_name` (added here) provides the queue→project mapping the v2 picker consumes +- `xce_sched_project_t` scaffolding (added here) provides the struct the v2 picker extends with quota/usage fields -The architecture is designed so that v1 (flat) and v2 (hierarchical) share the identical scheduling loop. The only difference is which project picker plugin is configured. +The architecture is designed so that v1 (flat, `builtin_fcfs` ignores projects) and v2 (hierarchical, project-aware picker) share the identical core scheduling loop. The only difference is which picker plugin is configured. diff --git a/.kiro/specs/scheduler-framework/spec.json b/.kiro/specs/scheduler-framework/spec.json index 4e263138..8c5192d2 100644 --- a/.kiro/specs/scheduler-framework/spec.json +++ b/.kiro/specs/scheduler-framework/spec.json @@ -1,9 +1,10 @@ { "feature_name": "scheduler-framework", "created_at": "2026-07-09T21:20:00Z", - "updated_at": "2026-07-10T10:00:00Z", + "updated_at": "2026-07-27T00:00:00Z", "language": "en", - "phase": "requirements-generated", + "phase": "decomposed", + "description": "DECOMPOSED into scheduler-core, scheduler-allocation, scheduler-preemption. This spec is retained for discovery context; see child specs for active requirements.", "approvals": { "requirements": { "generated": true, @@ -19,5 +20,6 @@ } }, "ready_for_implementation": false, - "blocked_on": [] + "blocked_on": [], + "child_specs": ["scheduler-core", "scheduler-allocation", "scheduler-preemption"] } diff --git a/.kiro/specs/scheduler-preemption/brief.md b/.kiro/specs/scheduler-preemption/brief.md new file mode 100644 index 00000000..25025ed1 --- /dev/null +++ b/.kiro/specs/scheduler-preemption/brief.md @@ -0,0 +1,183 @@ +# Brief: scheduler-preemption + +## Problem + +The XCE scheduler has no preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the scheduler has no way to free those resources. The job is simply pended, and it waits indefinitely — regardless of its priority — until running jobs naturally complete. + +Three specific gaps exist: + +1. **No preemption decision type** — the `xce_decision_op_t` enum has `DISPATCH` and `PEND` but no `PREEMPT`. The scheduler cannot communicate a preemption intent to the controller. +2. **No victim selection mechanism** — even if a preemption decision type existed, there is no logic to select which running jobs should be preempted. Cross-queue victim selection (a high-priority job in queue A preempting a low-priority job in queue B) is entirely absent. +3. **No SIGSTP pipeline** — the controller and worker have no path to deliver a stop signal to a running job and transition its state to `JOB_SUSPENDED`, freeing its resources for the preempting job. + +Operators with mixed-priority workloads need preemption to ensure critical work proceeds without manual intervention — the operator should not have to manually kill low-priority jobs to make room. + +## Current State + +### What exists +- **Decision framework** (`decision.h`): `xce_decision_op_t` with `DISPATCH`, `PEND`, and `NONE` ops. Core/decision callback flows to driver/controller. +- **Queue structure** (`xce_sched_queue_t`): has `priority` field for inter-queue ordering within the picker, but no preemption-related fields. +- **HOOK_DETAIL_JOB_STOP** (enum value 65 in `cluster_def.h`): existing hook detail enum, reused for preemption stop notifications. +- **Controller→worker messaging**: existing message pipeline for dispatch/pend decisions; the same path can carry preemption actions to workers. + +### What's missing +- `XCE_DECISION_PREEMPT` opcode and `xce_preempt_decision_t` payload in `decision.h` +- Victim selection policy: which running jobs get preempted for a given high-priority job +- Per-queue preemption configuration: which queues can preempt from which other queues, which queues are immune to preemption +- SIGSTP delivery pipeline: scheduler → controller → worker → process group +- Job state transition to `JOB_SUSPENDED` triggered by preemption +- Integration of preemption into the scheduling loop: when allocation fails, attempt preemption before pended + +## Desired Outcome + +A job preemption system with three layers: + +**1. Decision type** — `XCE_DECISION_PREEMPT` carries victim jobids, signal type, and preempting job id through the core's decision callback → normal-mode driver → controller → worker pipeline, reusing the same callback infrastructure (`xce_core_decision_cb_t`) used by `DISPATCH` and `PEND`. + +**2. Victim selection** — a `preempt_policy` plugin on each queue selects which running jobs to preempt. The default policy `"lowest_priority_first"` implements LSF-style cross-queue victim selection: +- Collect running jobs from the preempting job's `preemptible_queues` list +- Filter: victim priority < preempting job priority AND victim's queue has `preemptible == true` +- Rank by priority ascending (lowest priority first) +- Select the minimum subset whose freed resources satisfy the preempting job's resreq + +**3. SIGSTP pipeline** — the controller receives `XCE_DECISION_PREEMPT`, delivers `HOOK_DETAIL_JOB_STOP` to each victim's worker, the worker sends SIGSTP to the job's process group, the controller transitions the job to `JOB_SUSPENDED`, and resources are released. The preempting job is pended and retried in the next scheduling round (async preemption). + +### Queue Preemption Configuration + +Each queue carries three new fields: + +| Field | Type | Purpose | +|-------|------|---------| +| `preemptible_queues` | `char**` | List of queue names this queue CAN preempt from | +| `preemptible` | `bool` | Whether this queue's jobs CAN BE preempted by other queues | +| `preempt_policy` | `char*` | Victim-selection plugin name (NULL = use default) | + +## Alternatives Considered + +### Approach A: Kill-and-requeue only (rejected) + +| Aspect | Detail | +|--------|--------| +| **Mechanism** | SIGKILL victims, requeue from scratch | +| **Pros** | Simple, no suspended state, no resume logic | +| **Cons** | Destroys progress, lose all intermediate computation. Requeued jobs restart from zero. Not viable for long-running HPC workloads where checkpointing is explicit. | + +**Rejected because:** HPC jobs can run for hours or days. Killing a job that is 90% complete to make room for a higher-priority job wastes all accumulated work. SIGSTP + resume preserves progress when the preempted job is later resumed. + +### Approach B: SIGSTP with suspend/resume (chosen) + +| Aspect | Detail | +|--------|--------| +| **Mechanism** | SIGSTP to process group → job → `JOB_SUSPENDED` → resources freed → preempting job runs → preempted job optionally resumed later | +| **Victim selection** | Policy plugin per queue, default `"lowest_priority_first"` | +| **Cross-queue** | Per-queue `preemptible_queues` list enables cross-queue preemption | +| **Scope** | v1 is SIGSTP only; kill-and-requeue escalation deferred | + +**Chosen because:** Preserves victim job progress. Enables later resume (outside this spec's scope). Aligns with LSF-style preemption semantics familiar to HPC operators. The async pattern (pend preempting job, retry next round after SIGSTP) keeps the scheduling loop simple. + +## Approach + +### Preemption Flow + +``` +scheduler controller worker +─────────── ────────── ────── +alloc fails for high-prio job J + +preempt_policy.select_victims(J): + collect running jobs from + Q_aggressor.preemptible_queues + filter: victim.prio < J.prio + filter: victim.queue.preemptible + rank: lowest priority first + select minimum set satisfying J.resreq + + → [victim1, victim2] + +XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 + .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED + .signal = SIGSTP victim's worker release resources + .preempting_jobid = J + +J pended → next round → retry alloc +``` + +### Integration into Scheduling Loop + +Preemption fires at Level 3 in the core scheduling loop — after allocation failure, before pending: + +```c +while ((job = picker->next_job(picker)) != NULL) { + if (try_allocate(job) > 0) { + emit_decision(DISPATCH, job); + picker->on_dispatch(picker, job->jobid); + } else if (can_preempt(job)) { + victims = select_victims(job); + emit_decision(PREEMPT, job, victims); + picker->on_pend(picker, job->jobid); // retry next round + } else { + emit_decision(PEND, job); + picker->on_pend(picker, job->jobid); + } +} +``` + +The core loop itself contains no victim selection logic — that lives in the `preempt_policy` plugin. The core only checks `can_preempt(job)` (does the job's queue have a non-NULL `preemptible_queues` list?) and delegates to the policy plugin for victim selection. + +### What stays unchanged +- Core scheduling loop — preemption is an additional branch in the alloc-fail path (Level 3), not a restructure +- Decision callback (`xce_core_decision_cb_t`) — `PREEMPT` flows through the same callback as `DISPATCH` and `PEND` +- Existing plugin framework — `preempt_policy` is registered, loaded, and invoked through the existing `plugin_register`/`key_find` mechanism +- `HOOK_DETAIL_JOB_STOP` — existing enum value, reused without change + +### What's new +- `XCE_DECISION_PREEMPT` in `xce_decision_op_t` (`decision.h`) +- `xce_preempt_decision_t` payload carrying `victim_jobids[]`, `n_victims`, `signal`, `preempting_jobid` +- `preemptible_queues`, `preemptible`, `preempt_policy` fields on `xce_sched_queue_t` +- Default `"lowest_priority_first"` preemption policy plugin +- Controller handler for `XCE_DECISION_PREEMPT` → SIGSTP delivery to workers +- Worker-side SIGSTP handler → process group signal + `JOB_SUSPENDED` state transition + +## Scope + +### In +- `XCE_DECISION_PREEMPT` decision type and `xce_preempt_decision_t` payload +- `preempt_policy` plugin interface (victim selection) +- Default `"lowest_priority_first"` preemption policy +- LSF-style cross-queue victim selection via `preemptible_queues` list +- SIGSTP pipeline: scheduler → controller → worker → process group +- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy` +- Victim safety guards: never preempt same-or-higher priority, never preempt non-`preemptible` queues +- `JOB_SUSPENDED` state transition on victim jobs +- Reuse of `HOOK_DETAIL_JOB_STOP` for preemption stop notifications +- Unit tests: victim selection policy, safety guard enforcement, edge cases (empty target queues, no eligible victims, insufficient resources from any victim set) +- Integration test: end-to-end preemption flow — high-priority job submits, preemption fires, victims get SIGSTP, resources freed, preempting job dispatched in next round + +### Out +- Kill-and-requeue escalation (v1 is SIGSTP only) +- Deadline-based preemption (requires `deadline` field on jobs — deferred) +- Preempted job resume logic (outside this spec's scope — the `JOB_SUSPENDED` state is defined, but the resume mechanism is a separate feature) +- Preemption priority inversion guards beyond the priority comparison already specified +- Fair-share or proportional victim selection beyond the default `"lowest_priority_first"` policy (the plugin interface enables custom policies, but none beyond the default ship here) + +## Upstream / Downstream + +### Upstream (dependencies) +- **scheduler-core** (sibling spec): provides the decision callback (`xce_core_decision_cb_t`) through which `XCE_DECISION_PREEMPT` flows. The core scheduling loop and `xce_sched_core_t` are defined there. Preemption hooks into the core loop at Level 3 (alloc-fail branch). +- **scheduler-allocation** (sibling spec): provides `try_allocate()` — preemption fires only when allocation fails. The resreq fingerprint cache interacts with preemption (a cached-unsatisfiable resreq skips allocation AND preemption). +- **Controller message handling**: existing handler dispatch path for decisions — extended to handle `XCE_DECISION_PREEMPT`. +- **Worker job control**: existing job lifecycle management — extended to handle SIGSTP delivery and `JOB_SUSPENDED`. + +### Downstream (future consumers) +- **Resume mechanism** — a future spec for resuming `JOB_SUSPENDED` jobs, potentially with priority-based ordering +- **Kill-and-requeue escalation** — a future enhancement where repeated preemption failure escalates to SIGKILL +- **Custom preemption policies** — deadline-aware, fair-share, or resource-specific victim selection implemented as new `preempt_policy` plugins + +## Constraints +- **C (gnu17)** — no C++, no external libraries beyond existing (pthreads) +- **Async preemption** — the preempting job is pended and retried in the next scheduling round; preemption is not synchronous +- **Priority guard** — victim priority must be strictly less than preempting job priority (never preempt equal or higher) +- **Preemptible guard** — victim queue must have `preemptible == true` (opt-in — queues are not preemptible by default) +- **Minimal victim set** — select the minimum subset of victims whose freed resources satisfy the preempting job's resreq (avoid over-preemption) +- **No self-preemption** — the preempting job's own queue is implicitly excluded from victim selection +- **Cancel-safe** — `preempt_policy.select_victims()` must have `pthread_cancel` cancellation points diff --git a/.kiro/specs/scheduler-preemption/requirements.md b/.kiro/specs/scheduler-preemption/requirements.md new file mode 100644 index 00000000..72e8b22d --- /dev/null +++ b/.kiro/specs/scheduler-preemption/requirements.md @@ -0,0 +1,109 @@ +# Requirements Document + +## Introduction + +The XCE scheduler currently has no job preemption mechanism. When a high-priority job cannot be scheduled because all resources are consumed by lower-priority running jobs, the only available action is to pend the job — it waits indefinitely until running jobs complete naturally, regardless of priority. + +This feature closes the preemption gap with three additions: + +1. A `XCE_DECISION_PREEMPT` decision type flowing through the same callback as `DISPATCH` and `PEND`, carrying victim job ids and signal type to the controller→worker pipeline. +2. A configurable `preempt_policy` plugin mechanism for victim selection, with a built-in `"lowest_priority_first"` default policy implementing LSF-style cross-queue victim selection. +3. A SIGSTP delivery pipeline: scheduler emits `XCE_DECISION_PREEMPT` → controller delivers `HOOK_DETAIL_JOB_STOP` to each victim's worker → worker sends SIGSTP to the job's process group → controller transitions the job to `JOB_SUSPENDED` and releases resources. + +The design uses async preemption: the preempting job is pended and retried in the next scheduling round after resources are freed, rather than blocking the scheduling loop waiting for victims to stop. + +> Full discovery context (problem, current state, alternatives considered, approach, boundary) is in `.kiro/specs/scheduler-preemption/brief.md`. +> +> This spec is a child of `scheduler-framework`. It depends on **scheduler-core** for the decision callback (`xce_core_decision_cb_t`) through which `XCE_DECISION_PREEMPT` flows. The preemption branch (Level 3) fires in the core scheduling loop after allocation failure, before pending — this integration point is defined in scheduler-core's requirements. + +## Boundary Context + +- **In scope**: `XCE_DECISION_PREEMPT` opcode and `xce_preempt_decision_t` payload in `decision.h`; `preempt_policy` plugin interface for victim selection; default `"lowest_priority_first"` preemption policy implementing LSF-style cross-queue victim selection; per-queue preemption configuration (`preemptible_queues`, `preemptible`, `preempt_policy` fields on `xce_sched_queue_t`); SIGSTP delivery pipeline through controller→worker; `JOB_SUSPENDED` state transition on victim jobs; reuse of `HOOK_DETAIL_JOB_STOP` for preemption stop notifications; victim safety guards (never preempt same-or-higher priority, never preempt non-`preemptible` queues, never over-preempt); integration with the core scheduling loop at Level 3 (alloc-fail branch). + +- **Out of scope**: Kill-and-requeue escalation beyond SIGSTP (deferred — v1 is SIGSTP only); deadline-based preemption (requires `deadline` field on `xce_sched_job_t`, deferred); preempted job resume logic (the `JOB_SUSPENDED` state is defined but resume is a separate feature); fair-share or proportional victim selection policies beyond the default (the plugin interface enables them, but no additional policies ship here); changes to the picker interface, iterator abstractions, core/driver separation, or the scheduling loop structure itself (those belong to sibling specs `scheduler-core` and `scheduler-allocation`). + +- **Adjacent expectations**: This spec depends on `scheduler-core` for the decision callback (`xce_core_decision_cb_t`) and the core scheduling loop into which the preemption branch integrates. The existing controller→worker message pipeline for dispatch/pend is extended to carry `XCE_DECISION_PREEMPT`. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum (value 65 in `cluster_def.h`) is reused for preemption stop actions. The existing `plugin_register`/`key_find` mechanism loads the `preempt_policy` plugin. + +## Data Structure Changes + +### `decision.h` additions +```c +typedef enum { + XCE_DECISION_NONE = 0, + XCE_DECISION_DISPATCH, + XCE_DECISION_PEND, + XCE_DECISION_PREEMPT, // NEW — preemption decision + XCE_DECISION_NEW_SESSION +} xce_decision_op_t; + +typedef struct { + uint64_t* victim_jobids; // jobs to preempt + int n_victims; // count of victims + int signal; // signal to send (SIGSTP in v1, SIGKILL in future) + uint64_t preempting_jobid; // job that triggered preemption +} xce_preempt_decision_t; +``` + +### `xce_sched_queue_t` additions +```c +char** preemptible_queues; // queue names this queue CAN preempt from (NULL = none) +int n_preemptible_queues; // count of preemptible_queues entries +bool preemptible; // CAN this queue's jobs BE preempted by other queues? +char* preempt_policy; // victim-selection plugin name (NULL = use default) +``` + +## Preemption Flow + +``` +scheduler controller worker +─────────── ────────── ────── +alloc fails for high-prio job J + +preempt_policy.select_victims(J): + collect running jobs from + Q_aggressor.preemptible_queues + filter: victim.prio < J.prio + filter: victim.queue.preemptible + rank: lowest priority first + select minimum set satisfying J.resreq + + → [victim1, victim2] + +XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1 + .victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED + .signal = SIGSTP victim's worker release resources + .preempting_jobid = J + +J pended → next scheduling round → allocation retries on freed resources +``` + +## Requirements + +### Requirement: Preemption policy and execution + +**Objective:** As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention. + +#### Acceptance Criteria + +1. The Scheduler shall add a `XCE_DECISION_PREEMPT` opcode to `xce_decision_op_t` in `decision.h`, with a payload type `xce_preempt_decision_t` carrying `victim_jobids[]`, `n_victims`, `signal` (SIGSTP in v1), and `preempting_jobid`. The `XCE_DECISION_PREEMPT` decision shall flow through the same `xce_core_decision_cb_t` callback as `DISPATCH` and `PEND`; the core scheduling loop shall contain zero knowledge of how the driver publishes preemption decisions. + +2. When allocation fails for a job J whose queue has a non-NULL `preemptible_queues` list, the Scheduler shall invoke the queue's `preempt_policy` plugin to select victim jobs from the target queues. When `preempt_policy` is NULL, the Scheduler shall use the default `"lowest_priority_first"` policy. + +3. The default preemption policy `"lowest_priority_first"` shall: WHILE the preempting job's resource request is not yet satisfied DO collect running jobs from target queues; filter to those with `priority < J.priority` AND `queue->preemptible == true`; rank by priority ascending (lowest priority first); select the next victim candidate and add to the victim set. The policy shall return the minimum subset of victims whose freed resources satisfy J's resource request. When multiple victim subsets of equal minimum size exist, the policy shall prefer the subset whose victims have the lowest summed priority. + +4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via the decision callback, and shall pend job J for the next scheduling round (async preemption — the job is retried after victim resources are freed). The Scheduler shall not wait synchronously for victim jobs to transition to `JOB_SUSPENDED`. + +5. When no sufficient victim set exists after exhausting all eligible running jobs from the target queues, the Scheduler shall pend job J with a resource-shortage pend reason without invoking preemption. The decision callback shall not be called for the preemption path in this case. + +6. The worker receiving a preemption action shall send the specified signal (SIGSTP in v1) to the victim job's process group. The controller shall transition the victim job's state to `JOB_SUSPENDED` after confirming the signal was delivered. Resources held by the victim job shall be released upon suspension. + +7. The Scheduler shall add `preemptible_queues` (list of queue names this queue CAN preempt from, NULL = none), `preemptible` (bool — CAN this queue's jobs BE preempted by other queues, default false), and `preempt_policy` (victim-selection plugin name, NULL = use default `"lowest_priority_first"`) fields to `xce_sched_queue_t`. These fields shall be populated from queue configuration at `QUEUE_UPDATE` time. + +8. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum (value 65 in `cluster_def.h`) shall be reused for preemption stop notifications, with no changes to the enum definition. The controller shall deliver this hook detail to each victim's worker as part of the preemption action. + +9. Preemption victim selection shall never select: + - A job whose priority is greater than or equal to the preempting job's priority. + - A job whose queue has `preemptible == false`. + - A job that is not in running state. + - A job from a queue not listed in the preempting job's `preemptible_queues` list. + - More victims than necessary to satisfy the preempting job's resource request (minimum sufficient victim set). diff --git a/.kiro/specs/scheduler-preemption/spec.json b/.kiro/specs/scheduler-preemption/spec.json new file mode 100644 index 00000000..3ffc24e2 --- /dev/null +++ b/.kiro/specs/scheduler-preemption/spec.json @@ -0,0 +1,26 @@ +{ + "feature_name": "scheduler-preemption", + "created_at": "2026-07-27T00:00:00Z", + "updated_at": "2026-07-27T00:00:00Z", + "language": "en", + "phase": "requirements-generated", + "parent_spec": "scheduler-framework", + "description": "Job preemption policy and execution for the XCE scheduler — SIGSTP-based victim selection and controller→worker pipeline", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "blocked_on": ["scheduler-core"], + "sibling_specs": ["scheduler-core", "scheduler-allocation"] +}