# Technical Design: scheduler-core

## Overview

**Purpose**: This feature replaces the scheduler's hardcoded FCFS job-selection logic and its hard-wired coupling to the live controller with a pluggable picker interface and a passive, mode-agnostic scheduling core. The core is driven by either a normal-mode driver (live controller) or a simulation-mode driver (recorded `.replay` file), and both modes are required to produce byte-identical scheduling decisions given identical input — this is the spec's central, testable invariant.

**Users**: Cluster operators configure job-picking policy without a rebuild (Req 2) and gain a `xceadmin scheduler record`/`dump` workflow. Scheduler developers gain a `--sim`/`--interactive` simulation driver to debug and tune scheduling policy against recorded production traces without a live cluster (Req 5).

**Impact**: `xce_scheduler_t` (today: scheduling state + `apictx` + `cluster_cfg` in one struct, `scheduler.c`) splits into `xce_sched_core_t` (pure scheduling state, zero I/O) plus `xce_sched_driver_t` (normal, controller-attached) and `xce_sched_sim_driver_t` (simulation, replay-attached). The hardcoded `"xce_sched_fcfs"` string at `scheduler.c:43` is replaced by a configurable, plugin-registered picker. A new controller opcode (`OPCode_schedAdmin`) and a new xceadmin command group (`scheduler record`/`scheduler dump`) are added as pass-through admin plumbing.

### Goals
- Job-picking policy configurable per-scheduler and per-queue without recompilation, with `builtin_fcfs` shipping as the zero-behavior-change default (Req 1-4).
- Scheduling core with zero controller I/O, wall-clock, or filesystem dependencies, driven identically by a normal or simulation driver (Req 5 AC 1-8).
- Runtime recording of live scheduling sessions to a self-contained `.replay` file, and a simulation driver + interactive REPL that replays it deterministically (Req 5 AC 9-22).
- Normal↔sim equivalence as a verifiable regression test: record a live session, replay it, diff the decisions — zero divergence (Req 5 AC 7.1).

### Non-Goals
- New job-picking policies beyond `builtin_fcfs` (interface only; sibling work).
- Allocation iterators / `try_allocate()` internals (`scheduler-allocation`).
- Preemption policy / victim selection (`scheduler-preemption`) — this design defines `PreemptDecision`'s **wire shape** only (needed for `Req 5 AC 5`'s decision table to compile), not preemption logic.
- Hierarchical project resource quotas (`project-resource-ownership`).
- REPL state mutation, reverse execution, or scripting (explicitly deferred in Req 5 AC 22).

## Boundary Commitments

### This Spec Owns
- `xce_job_picker_iface_t` and the `builtin_fcfs` implementation. As-built: a new `src/scheduler/src/plugins/builtin_fcfs.c` implements the interface; `plugins/fcfs.c` was deliberately left unchanged (still `job_elect_fn`-only) as the live exercise target for `plugin.c`'s legacy compat wrapper (see Components — BuiltinFcfsPicker).
- `xce_sched_core_t`: all scheduling state, the picker/allocator thread pair, the driver↔core protocol (`core_push_events`, `core_go`, `on_decision`), and `xce_core_now()` clock virtualization.
- `xce_sched_driver_t` (normal-mode): controller event translation, decision publishing, and recording control (start/stop/status, periodic snapshot).
- `xce_sched_sim_driver_t` (simulation-mode): `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`, and the interactive REPL.
- The `.replay` capnp wire format (`ReplayHeader`, `SnapshotEntry`, `ReplayEvent`, `ReplaySession`, `ReplayDecision`, and their payload sub-structs) in `src/schema/controller.capnp`.
- The `OPCode_schedAdmin` request/reply pair and its **pass-through** dispatch case in the controller's `myreq_handler` (`src/controller/cluster.c`), plus its accompanying `src/controller/testsuite/` coverage — this spec's only footprint outside `src/scheduler/`.
- `xceadmin scheduler record start|stop|status` and `xceadmin scheduler dump <FILE>` (new `src/xceadmin/src/xceadmin/scheduler.py`).
- `job_pick_plugin` (global + per-queue) and `project_name` (per-queue) fields; `xce_sched_project_t` scaffolding (name + priority only — no quota logic).

### Out of Boundary
- `try_allocate()` and the allocation iterator (`scheduler-allocation` owns the implementation; this spec only calls it from the allocator thread, unchanged signature).
- `can_preempt()` and victim-selection semantics (`scheduler-preemption` owns the algorithm; this spec owns only `PreemptDecision`'s wire shape, see Data Models).
- Hierarchical project quota, `resource_quota`/`resource_used`/`parent_project` on `xce_sched_project_t` (`project-resource-ownership`).
- Any controller state or persistence for recording (`OPCode_schedAdmin` is explicitly a stateless pass-through — the controller commits nothing, owns no WAL entry for it).
- REPL write-side mutation, semantic snapshot diff in xceadmin (both explicitly deferred, Req 5 AC 22).

### Allowed Dependencies
- Existing plugin framework (`plugin.h`/`plugin.c`) — `xce_job_picker_iface_t` is a new plugin category registered through the same `plugin_register`/`key_find` mechanism, not a parallel system.
- Existing select-expression evaluator (`resreq.h`, `select_eval_ctx_t`) — used as-is by the allocator thread; the core does not re-implement or wrap it.
- libxce data-structure subset only inside the core: `ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti` (Req 5 AC 3). The core must **not** link `netio`, `delay`, `xceapi`, or `idgen`.
- `xcecli_ctx_t` API (`xcecli_jobdispatch`, `xcecli_set_pend_reason`, `xcecli_preempt`) — normal-mode driver only, never referenced from `xce_sched_core_t` or `xce_sched_sim_driver_t`.
- `xcecommon.wire`/`xcecommon.schema`/`xceadmin.discovery` (Python) — the new `xceadmin/scheduler.py` module reuses these exactly as `push.py` does; it does not open its own socket or reload the schema.

### Revalidation Triggers
- Any change to `xce_job_picker_iface_t`'s hook signatures — every picker plugin (present and future) must be re-verified.
- Any change to `jbuf` cbuf capacity away from 1 — invalidates the determinism argument in Req 5 AC 7; `scheduler-allocation` and `scheduler-preemption` must both re-verify ordering assumptions.
- Any change to the `.replay` capnp schema's message set or `ReplayDecision.body` union — `scheduler-preemption`'s `PreemptDecision` consumer and any external tooling reading `.replay` files must re-check compatibility.
- Any change to `OPCode_schedAdmin`'s request/reply shape — `xceadmin.scheduler` module must be updated in lockstep (same repo, same PR).

## Architecture

### Existing Architecture Analysis

Today, `xce_scheduler_t` (`src/scheduler/include/scheduler.h`) owns scheduling state (`jobs`, `queues`, `workers`, `plugins` kvmgrs), controller I/O (`apictx: xcecli_ctx_t`, `cluster_cfg: xce_cluster_config_t`), and three threads (`jobpicker`, `allocator`, main) wired through three `cbuf`s (`wbuf` cap 1, `jbuf` cap `njobs`≡1 today, `decisions` cap `ndecisions`≡5). `main.c`'s loop calls `sched_update` → `sched_begin_session` → `sched_go` → drains `decisions` cbuf with a hardcoded `switch (decision->op)` → `sched_end_session`. The picker plugin (`fcfs.c`) is selected by a literal string (`scheduler.c:43`) and implements the old `xce_sched_plugin_t->job_elect_fn` hook, tightly coupled to `sched->queues`/`sched->jobs` kvmgrs directly.

**Preserved without change**: the three-thread/three-cbuf pattern (`wbuf`→picker, `jbuf`→allocator, `decisions`→driver); `plugin_register`/`key_find` plugin lookup; `select_eval_ctx_t` per-session memoization; `sched_queue_is_ready()`/`sched_job_is_ready()` (currently stub `return 1;` in `scheduler.c`, called as-is by both today's `fcfs_elect_job` and the new `builtin_fcfs`).

**What changes**: `xce_scheduler_t` splits along the controller-I/O seam into `xce_sched_core_t` (no `apictx`/`cluster_cfg`) and `xce_sched_driver_t` (holds them). The picker plugin category becomes `xce_job_picker_iface_t`, replacing `job_elect_fn` (deprecated with a compatibility wrapper, Req 5 AC "Migration from old job_elect_fn").

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph DriverLayer[Driver Layer]
        NormalDriver[Normal Driver]
        SimDriver[Simulation Driver]
        Controller[Live Controller]
        ReplayFile[Replay File]
        Controller --> NormalDriver
        ReplayFile --> SimDriver
    end

    subgraph CoreLayer[Scheduler Core]
        Core[Scheduler Core State]
        Picker[Job Picker Plugin]
        Allocator[Allocator Thread]
        Core --> Picker
        Picker --> Allocator
    end

    NormalDriver -->|core_push_events core_go| Core
    SimDriver -->|core_push_events core_go| Core
    Core -->|on_decision| NormalDriver
    Core -->|on_decision| SimDriver

    NormalDriver -->|xcecli| Controller
    SimDriver -->|write| RecordFile[Sim Output Replay File]

    subgraph AdminPath[Admin Control Path]
        XceAdmin[xceadmin scheduler CLI]
        SchedAdminOp[OPCode_schedAdmin dispatch]
        XceAdmin --> SchedAdminOp
        SchedAdminOp --> Controller
    end
    Controller -->|forward request-reply| NormalDriver
```

**Architecture Integration**:
- **Selected pattern**: Event-sourced core with a swappable driver (ports-and-adapters, applied narrowly — the core is the hexagon, the two drivers are the only adapters). Selected in requirements.md as the only pattern that makes normal↔sim equivalence a structural property rather than a discipline (see `research.md`, Architecture Pattern Evaluation).
- **Domain boundaries**: Core owns scheduling *decisions*; drivers own I/O translation *only* — no scheduling logic in either driver. `OPCode_schedAdmin` dispatch is a third, thin adapter (controller-side) that never touches core or driver state directly, only forwards bytes.
- **Existing patterns preserved**: three-thread/cbuf pipeline, plugin registration, select-expression memoization.
- **New components rationale**: `SimDriver` and the `.replay` format are net-new (no existing analog); `SchedAdminOpcode` is new but structurally mirrors the existing `OPCode_upgradeConfig` case (see `research.md`).
- **Steering compliance**: no new wall-clock/network dependency inside `src/scheduler/`'s core files (`ha-async-io.md`'s async-I/O discipline extended to a synchronous core by design, not violated — the core has no I/O at all); `.capnp` used for the new wire format per `structure.md`'s "Cap'n Proto for wire formats" convention.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Scheduler core / drivers | C (gnu17) | `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t` | No new language/runtime; existing `src/scheduler/` toolchain |
| Wire format | Cap'n Proto (existing `controller.capnp`) | `.replay` stream, `OPCode_schedAdmin` request/reply | Extends the existing schema file, not a new one |
| Admin CLI | Python / `click` (existing xceadmin stack) | `scheduler record`/`scheduler dump` commands | New module (`xceadmin/scheduler.py`) in the existing package, same `pycapnp` schema loader |
| Build | Meson ≥ 1.5 (existing) | New `.c` files added to `src/scheduler/meson.build`; testsuite entries | No new build system |

### Dependency Direction

`Picker Plugin Interface → Scheduler Core → {Normal Driver, Simulation Driver} → {Controller (xcecli), Replay File I/O}`. Controller-side `SchedAdminOpcode` dispatch depends only on the capnp schema (shared, not owned by the core) and forwards to the driver without depending on `xce_sched_core_t` internals. xceadmin's `scheduler.py` depends on `xcecommon`/`xceadmin.discovery` only — never on any C header. Each layer imports only from layers to its left; the core never imports driver headers (enforced by the "Allowed Dependencies" list above — no `xcecli_ctx_t`/`netio` symbol may appear in `core.c`).

## File Structure Plan

### Directory Structure

**As-built** (updated 2026-08-07 after implementation; see tasks.md's `## Implementation Notes` for the full rationale behind each split — all were 800-line-file-cap splits made during implementation, not scope changes):

```
src/scheduler/
├── include/
│   ├── scheduler.h        # unchanged Phase 1; Phase 2 shrinks to xce_scheduler_t compat shim
│   ├── plugin.h            # + xce_job_picker_iface_t (Req 1 AC 2), deprecation of job_elect_fn
│   ├── core.h              # xce_sched_core_t, xce_core_event_t, decision cb type
│   ├── driver.h            # xce_sched_driver_t (normal-mode)
│   ├── sim.h                # xce_sched_sim_driver_t -- built as sim.h, not sim_driver.h (task 11.1, documented deviation)
│   ├── recording.h          # record start/stop/status types, ReplayEvent/ReplaySession writer contract
│   ├── repl.h                # REPL hook wrappers consumed by sim_loop.c -- no-op unless sim->interactive
│   ├── decision.h           # pre-existing; extended with XCE_DECISION_PREEMPT (task 4.4)
│   └── sched/types.h       # + job_pick_plugin, project_name on xce_sched_queue_t (Req 2)
├── src/
│   ├── scheduler.c         # sched_job_picker(): job_elect_fn != NULL delegates to the legacy loop, NULL runs the generic xce_job_picker_iface_t loop
│   ├── core.c              # xce_sched_core_init/fini, jbuf(cap=1)/decisions cbuf plumbing
│   ├── core_events.c        # xce_core_push_events() -- split out of core.c (800-line cap; task 4.3)
│   ├── core_session.c       # xce_core_begin_session/xce_core_end_session -- split out of core.c (task 4.4)
│   ├── core_clock.c         # xce_core_now(), tlog format integration -- the ONE file allowed to call clock_gettime() (task 4.5)
│   ├── driver.c             # normal-mode driver: capnp<->core event translation, decision publishing, deferred admin-request handling
│   ├── recording.c          # record start/stop/status, header/snapshot writes
│   ├── recording_events.c   # ReplayEvent writes -- split out of recording.c (task 9.2/9.3)
│   ├── recording_session.c  # ReplaySession begin/end writes -- split out of recording.c (task 9.2/9.3)
│   ├── repl.c               # REPL command loop core (execution control, Req 5 AC 22)
│   ├── repl_break.c          # REPL breakpoints + event/decision hook wrappers -- split out of repl.c (task 12.2)
│   ├── repl_info.c           # REPL state-inspection commands -- split out of repl.c (task 12.2)
│   ├── repl_params.c         # REPL snapshot/simulation-parameter commands -- split out of repl.c (task 12.3)
│   ├── repl_internal.h       # shared repl_*.c internal declarations (task 12.2)
│   ├── snapshot.c           # save_snapshot()/load_snapshot() orchestration, picker rebuild
│   ├── snapshot_wire.c       # SnapshotData capnp encode/decode -- split out of snapshot.c (task 8.1)
│   ├── snapshot_load.c       # snapshot restore application -- split out of snapshot.c (task 8.1)
│   ├── sim.c                 # simulation startup (Req 5 AC 13): replay header/snapshot load, virtual clock seed
│   ├── sim_loop.c            # simulation event loop + REPL hook call sites (Req 5 AC 14/18)
│   ├── sim_record.c          # --sim-output ReplayEvent/ReplaySession writes -- split out of sim_loop.c (task 11.5)
│   ├── sim_verify.c          # --sim-verify decision comparator -- split out of sim_loop.c (task 11.4)
│   ├── handler.c            # unchanged — capnp push/notify enqueue into pend_updates; scheduler_req_handler dispatches SchedAdminRequest
│   ├── plugin.c             # + picker plugin registration category, + job_elect_fn compat
                              #   wrapper (adapts old-style job_elect_fn-only plugins to
                              #   xce_job_picker_iface_t, logs a deprecation warning at register time)
│   └── plugins/
│       ├── builtin_fcfs.c   # the native xce_job_picker_iface_t default picker (Req 3) -- NEW file, see note below
│       └── fcfs.c           # UNCHANGED -- still registers job_elect_fn only; kept as the live exercise target for plugin.c's compat wrapper (task 1.2). NOT rewritten as originally planned here; see BuiltinFcfsPicker component notes.
├── testsuite/
│   ├── test_picker_iface.c, test_picker_fcfs.c, test_picker_fcfs_session.c, test_picker_wiring.c,
│   │   test_picker_config_fields.c, test_picker_core_loop_boundary.c, test_picker_pendreason_factory_audit.c,
│   │   test_plugin_picker_registration.c        # picker interface + builtin_fcfs + registration/compat-wrapper coverage
│   ├── test_core_events.c, test_core_session.c, test_core_full_cycle.c, test_core_isolation.c,
│   │   test_core_jbuf_capacity_pin.c            # core contract + determinism-pin coverage
│   ├── test_clock.c                             # xce_core_now() two-base+speed formula
│   ├── test_driver_translate.c, test_driver_decision_cb.c, test_driver_session_ordering.c
│   ├── test_recording_path_validation.c, test_record_lifecycle.c, test_admin_request.c
│   ├── test_snapshot_save_load.c
│   ├── test_sim_startup.c, test_sim_event_loop.c, test_sim_isolation.c, test_sim_alt_events.c,
│   │   test_sim_verify.c, test_sim_record_output.c, test_sim_equivalence.c
│   └── test_repl_execution_control.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c,
│       test_repl_equivalence.c
src/controller/
├── cluster.c                 # + case OPCode_schedAdmin (Req 5 AC 10)
└── testsuite/
    └── test_sched_admin_dispatch.c  # standby rejection, disconnected-scheduler, timeout, success round-trip
                                      # (name differs from this plan's original test_sched_admin.c; same coverage,
                                      #  registered in controller_test.c same as existing opcode tests)
src/schema/
└── controller.capnp          # + SchedAdminRequest/Reply, ReplayHeader/.../ReplayDecision + payload structs (see Data Models);
                               #   + WalCatchupEntry.payload/batchPayloads switched Data->List(UInt8) (bug-1505, follow-up
                               #   outside this spec's boundary, same codec-crash class as SnapshotData.internalState/bug-1504)
src/xceadmin/src/xceadmin/
├── cli.py                   # + @cli.group() scheduler; nested @scheduler.group() record with
                              #   start/stop/status commands, plus @scheduler.command() dump (thin layer only)
└── scheduler.py              # record_start/stop/status(), dump() — mirrors push.py's pattern
src/xceadmin/tests/
├── test_scheduler.py         # mock-TLS-server tests for record start/stop/status + dump parsing
└── test_cli_scheduler_mock.py # additional CLI-level mock-server coverage (task 10.6, beyond this plan's original file list)
```

### Modified Files
- `src/scheduler/src/scheduler.c` — Phase 1: hardcoded picker string removed; Phase 2: `sched_job_picker()` branches `job_elect_fn != NULL` (legacy plugins, delegates to their own self-contained loop) vs `NULL` (native `xce_job_picker_iface_t` pickers, generic per-session loop) — `scheduler.c` was not retired; it remains the picker-dispatch and allocator-loop home.
- `src/scheduler/src/plugins/fcfs.c` — **as-built: left unchanged**, not rewritten. `plugins/builtin_fcfs.c` (new file) implements `xce_job_picker_iface_t` and is the registered default (`schedobj.c`); `fcfs.c` keeps registering `job_elect_fn` only, deliberately, as the live exercise target for `plugin.c`'s legacy compat wrapper (task 1.2) — see BuiltinFcfsPicker component notes for the full rationale.
- `src/scheduler/src/main.c` — decision-drain `switch` replaced by `on_decision` callback registration (Req 5 AC 6); also gained the `--sim`/`--interactive`/`--record-dir`/`--sim-*` CLI surface (tasks 9.1, 11.1).
- `src/controller/cluster.c` — one new `case OPCode_schedAdmin:` block, structurally mirroring `OPCode_upgradeConfig` (primary-gate, delegate, explicit reply every path).
- `src/schema/controller.capnp` — additive only: new `OPCode_schedAdmin` enum value, new structs, two new union members on existing `Request.body`/`RequestReply.data` unions (ordinals `@31`/`@19` respectively — next available, verified against current schema in `research.md`).
- `src/xceadmin/src/xceadmin/cli.py` — one new `@cli.group() scheduler`, containing a nested `@scheduler.group() record` (with `start`/`stop`/`status` commands) plus a sibling `@scheduler.command() dump` — matching the literal CLI syntax `xceadmin scheduler record start`/`xceadmin scheduler dump`, not 4 flat commands.

## System Flows

### Normal-Mode Scheduling Round (driver ↔ core)

```mermaid
sequenceDiagram
    participant Ctrl as Controller
    participant Drv as Normal Driver
    participant Core as Scheduler Core
    participant Pick as Picker Thread
    participant Alloc as Allocator Thread

    Ctrl->>Drv: capnp push or notify
    Drv->>Drv: buffer into pend_updates
    Drv->>Core: core_push_events
    Core->>Pick: state change hooks, synchronous, picker idle
    Drv->>Core: core_go, non blocking
    Core->>Pick: signal via wbuf
    Pick->>Pick: begin_session then next_job loop
    Pick->>Alloc: send jobid via jbuf per job
    Alloc->>Alloc: try_allocate then can_preempt
    Alloc->>Drv: send decision via decisions cbuf
    Drv->>Ctrl: on_decision publishes dispatch, pend reason, or preempt
    Alloc->>Drv: end of session marker via decisions cbuf
    Drv->>Core: core_end_session
```

Key decision not obvious from the diagram: decisions publish to the controller **immediately per-decision**, not batched at session end (Req 5, "Decision publishing") — the drain loop in the driver thread interleaves `cbuf_get`/`on_decision` one at a time.

### `OPCode_schedAdmin` Recording Control (admin path)

```mermaid
sequenceDiagram
    participant CLI as XceAdmin SchedulerModule
    participant Ctrl as Controller ClusterDispatch
    participant Sched as Scheduler NormalDriver

    CLI->>Ctrl: schedAdmin request, action plus recordFile
    Ctrl->>Ctrl: check ha_is_primary, reject standby
    Ctrl->>Sched: forward via scheduler endpoint, five second timeout
    Sched->>Sched: validate recordFile, reject or resolve under record_dir
    Sched->>Sched: save_snapshot then write header and snapshot entry
    Sched-->>Ctrl: schedAdmin reply, ok plus recording state and counters
    Ctrl-->>CLI: relay reply to caller
```

Key decision not obvious from the diagram: the controller is a pure pass-through — it does not interpret `SchedAdminRequest` fields beyond the opcode dispatch, and owns no recording state (Boundary Commitments, "This Spec Owns" — recording logic lives entirely in the scheduler driver).

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|---|---|---|---|---|
| 1.1 | Core loop calls `next_job()` only, no ordering logic | SchedulerCore, JobPicker | `xce_job_picker_iface_t.next_job` | Normal-Mode Round |
| 1.2 | Picker hook categories (lifecycle/state/session/post-alloc/optional) | JobPicker | `xce_job_picker_iface_t` | — |
| 1.3 | Three-thread cbuf model | SchedulerCore | `wbuf`/`jbuf`/`decisions` cbufs | Normal-Mode Round |
| 1.4 | Picker/allocator/main-loop pseudocode contract | SchedulerCore, JobPicker | `core_push_events`/`core_go` | Normal-Mode Round |
| 1.5 | Cancel-safe, session-aware picker | JobPicker | `begin_session`/`end_session`/`next_job` | — |
| 1.6 | Replace hardcoded picker string | SchedulerCore | `plugin_register`/`key_find` | — |
| 1.7 | Preserve thread architecture, decision publish via callback | SchedulerCore | `on_decision` cb | Normal-Mode Round |
| 2.1 | `job_pick_plugin` on `xce_scheduler_t`, default `builtin_fcfs` | SchedulerCore | — | — |
| 2.2 | `job_pick_plugin` per-queue override | JobPicker | `xce_sched_queue_t.job_pick_plugin` | — |
| 2.3 | `project_name` on queue | SchedulerCore | `xce_sched_queue_t.project_name` | — |
| 2.4 | Config-driven picker name, no rebuild | SchedulerCore | scheduler config | — |
| 2.5 | Unregistered plugin fails loud | SchedulerCore | `plugin_register`/`key_find` | — |
| 2.6 | Picker contract documented (cancel-safe, session-aware) | JobPicker | `xce_job_picker_iface_t` | — |
| 3.1 | `builtin_fcfs` auto-registered as default | BuiltinFcfsPicker | plugin registration | — |
| 3.2 | Hook-by-hook FCFS implementation | BuiltinFcfsPicker | `xce_job_picker_iface_t` | — |
| 3.3 | Private `ordq_t` state, no core queries in `next_job` | BuiltinFcfsPicker | — | — |
| 3.4 | Ordering equivalence with today's `fcfs_elect_job` | BuiltinFcfsPicker | — | — |
| 3.5 | Dedicated picker thread retained, cancel points | BuiltinFcfsPicker, SchedulerCore | `next_job`/`cbuf_get(wbuf)` | Normal-Mode Round |
| 3.6 | `end_session` no-op, `pendJobs` carries over | BuiltinFcfsPicker | — | — |
| 4.1 | Job-picking order regression check | BuiltinFcfsPicker | — | test_picker_fcfs.c |
| 4.2 | Existing scheduler unit tests pass unmodified | SchedulerCore | — | testsuite/ |
| 4.3 | HA soak — no job loss/divergence | SchedulerCore | — | `/xce-test-ha` |
| 5.1 | Mode-agnostic core, no mode flag | SchedulerCore | `xce_sched_core_init()` | — |
| 5.2 | Passive event-driven interface, zero I/O deps | SchedulerCore | `core_push_events`/`on_decision` | — |
| 5.3 | Virtualized clock, two-base+speed formula | SchedulerCore | `xce_core_now()` | — |
| 5.4 | Core event types (typed payloads) | SchedulerCore | `xce_core_event_t` | — |
| 5.5 | Decision output types (DISPATCH/PEND/PREEMPT) | SchedulerCore | `xce_decision_t` | — |
| 5.6 | Decision output callback | SchedulerCore, NormalDriver | `xce_core_decision_cb_t` | Normal-Mode Round |
| 5.7 | Deterministic guarantee + normal↔sim equivalence | SchedulerCore | `jbuf` cap=1, hash-iteration audit, PRNG seed | test_sim_equivalence.c |
| 5.8 | Driver pending-event queue | NormalDriver | `pend_updates` dataq | Normal-Mode Round |
| 5.9 | Controller integration (capnp translate, publish decisions) | NormalDriver | `xcecli_ctx_t` | Normal-Mode Round |
| 5.10 | Runtime recording control (`OPCode_schedAdmin`) | NormalDriver, SchedAdminOpcode, XceAdminSchedulerCLI | `SchedAdminRequest`/`Reply` | Admin Path |
| 5.11 | Periodic snapshot during recording | NormalDriver | `record_snapshot_interval` | — |
| 5.12 | Session response time metrics | SchedulerCore | `SessionSummary` | — |
| 5.13 | Simulation mode startup (load header+snapshot) | SimDriver | `.replay` load sequence | — |
| 5.14 | Discrete event-driven virtual clock | SimDriver | `--sim-speed` | — |
| 5.15 | `--sim-verify` deterministic replay verification | SimDriver | `decision_eq` | — |
| 5.16 | `--sim-alt-events` what-if branching | SimDriver | alt-events file | — |
| 5.17 | Simulation isolation (no controller deps) | SimDriver | `xce_sched_sim_driver_t` | — |
| 5.18 | `ReplaySession` as session boundary | SimDriver | event loop | — |
| 5.19 | `--sim-output` simulation record file | SimDriver | `.replay` writer | — |
| 5.20 | `.replay` capnp stream format | ReplayFormat | schema structs | — |
| 5.21 | Snapshot serialization / picker rebuild | SchedulerCore, ReplayFormat | `save_snapshot`/`load_snapshot` | — |
| 5.22 | Interactive REPL | SimDriver | command loop | — |
| 6.1 | Resreq immutability | SchedulerCore | — | — |
| 6.2 | Pend reason factory (`make_pendreason()`) | SchedulerCore | `lib/preason.c` | — |
| 6.3 | Pointer-identity fast path | SchedulerCore | — | — |
| 6.4 | Select expression memoization preserved | SchedulerCore | `select_eval_ctx_t` | — |
| 6.5 | Snapshot forward-compatible with object sharing | ReplayFormat | `SnapshotData` | — |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|---------------------------|-----------|
| JobPicker (interface) | Scheduler Core | Defines the pluggable job-selection contract | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 | plugin.h (P0) | Service |
| BuiltinFcfsPicker | Scheduler Core | Default picker replicating today's FCFS order | 3.1-3.6, 4.1 | JobPicker (P0) | Service |
| SchedulerCore | Scheduler Core | Passive, mode-agnostic scheduling engine | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 | JobPicker (P0), allocation/preemption (P1, sibling specs) | Service, State |
| NormalDriver | Scheduler Driver | Controller-attached driver, recording control | 5.6, 5.8-5.11 | SchedulerCore (P0), `xcecli_ctx_t` (P0) | Service, Event |
| SimDriver | Scheduler Driver | Replay-attached driver, virtual clock, REPL | 5.13-5.19, 5.22 | SchedulerCore (P0), ReplayFormat (P0) | Service, State |
| ReplayFormat | Wire Protocol | `.replay` capnp schema, `SnapshotData` | 5.20, 5.21, 6.5 | `controller.capnp` (P0) | Batch |
| SchedAdminOpcode | Controller | Pass-through dispatch for `OPCode_schedAdmin` | 5.10 | `cluster.c` `myreq_handler` (P0) | API |
| XceAdminSchedulerCLI | Tooling | `scheduler record`/`scheduler dump` commands | 5.10 | `xcecommon.wire`/`discovery` (P0) | API |

### Scheduler Core

#### JobPicker

| Field | Detail |
|-------|--------|
| Intent | Contract every job-selection strategy implements; hides all ordering logic from the core loop |
| Requirements | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 |

**Responsibilities & Constraints**
- Owns iteration order over pending jobs; the core never inspects queue/job internals for ordering.
- Cancel-safe (`pthread_cancel` points in `next_job()`) and session-scoped (`begin_session` resets iteration state).
- State-change hooks run synchronously in the driver thread during `core_push_events()`; session hooks run in the picker thread; post-allocation hooks run in the allocator thread (Req 1 AC 3).

**Dependencies**
- Inbound: SchedulerCore — invokes every hook (P0)
- Outbound: none — picker implementations must not call back into core internals outside the hook parameters they're given

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
typedef struct {
    int  (*init)(void* picker, xce_sched_core_t* core);
    void (*fini)(void* picker);
    int  (*begin_session)(void* picker);
    xce_sched_job_t* (*next_job)(void* picker);
    int  (*end_session)(void* picker);
    int  (*on_dispatch)(void* picker, uint64_t jobid);
    int  (*on_pend)(void* picker, uint64_t jobid);
    int  (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
    int  (*on_job_push)(void* picker, xce_sched_job_t* job);
    int  (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
    int  (*on_job_update)(void* picker, xce_sched_job_t* job);
    int  (*on_config)(void* picker, xce_conf4sched_t* config);       /* MAY be NULL */
    int  (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); /* MAY be NULL */
} xce_job_picker_iface_t;
```
- Preconditions: `init()` called exactly once before any other hook; `begin_session()` precedes every `next_job()` sequence.
- Postconditions: `next_job()` returns NULL exactly when no ready job remains for the current session.
- Invariants: hook implementations never block indefinitely (picker thread holds no lock the allocator needs); `next_job()` never allocates a job the same session has already returned.

**Implementation Notes**
- Integration: registered via `plugin_register`/`key_find` under the picker plugin category (new `plugin.c` registration table entry).
- Validation: `test_picker_fcfs.c` feeds hooks directly and asserts `next_job()` order — no live core needed (Req 3 AC 3's testability rationale).
- Risks: a picker implementation that queries core state directly during `next_job()` (bypassing the hook-fed private state pattern) would still compile but silently reintroduce lock contention — not enforceable at the interface level, called out for code review.

#### BuiltinFcfsPicker

| Field | Detail |
|-------|--------|
| Intent | Default `xce_job_picker_iface_t` implementation; replicates today's `fcfs_elect_job` ordering exactly through the new interface |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1 |

**Responsibilities & Constraints**
- Maintains private state only — `ordq_t queues` (ordered by `queue->priority desc`) and, per queue, an `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). Never queries `sched->queues`/`sched->jobs` during `next_job()` (Req 3 AC 3).
- Every hook's behavior is fully specified in requirements.md Req 3 AC 2's hook-by-hook table — not repeated here; this block covers integration and testability, not the algorithm.
- Auto-registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` at scheduler startup (Req 3 AC 1).
- `end_session()` is a no-op; undispatched jobs carry over in `pendJobs` to the next round unchanged (Req 3 AC 6).

**Dependencies**
- Inbound: SchedulerCore — invokes every `xce_job_picker_iface_t` hook, same as any picker (P0)
- Outbound: none — implements the interface, depends on nothing beyond it

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
Implements `xce_job_picker_iface_t` (struct shown under JobPicker, above) with the `fcfs_plugin_data_t` private state described in Responsibilities. No additional public surface beyond the interface itself.
- Preconditions/Postconditions/Invariants: inherits JobPicker's contract in full; adds no relaxations or extensions.

**Implementation Notes**
- Integration, as-built: implemented in a **new** file, `plugins/builtin_fcfs.c`, registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` (`schedobj.c`) — NOT a rewrite of `plugins/fcfs.c` in place, as originally planned above. `fcfs.c` was deliberately left untouched (still `job_elect_fn`-only, unregistered as default) because task 1.2's `job_elect_fn` compat wrapper needs a real legacy-style plugin to exercise; `fcfs.c` fills that role, and `scheduler.c`'s `sched_job_picker()` dispatch (`job_elect_fn != NULL` vs `NULL`) depends on both variants existing side by side. Functionally equivalent to the original plan (one default picker implementing the new interface, ordering-equivalent to legacy `fcfs_elect_job`); only the file-layout mechanics differ.
- Validation: `test_picker_fcfs.c` is this component's primary test — feeds `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly and asserts `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order, matching today's `fcfs_elect_job` (Req 4.1's regression check).
- Risks: ordering-equivalence regression is the main risk (any drift from today's `fcfs_elect_job` output is a behavior change operators didn't opt into) — mitigated by `test_picker_fcfs.c` asserting against the documented ordering rule directly, plus the HA soak in Req 4.3.

#### SchedulerCore

| Field | Detail |
|-------|--------|
| Intent | Owns all scheduling state; drives the picker/allocator threads; exposes the driver↔core protocol |
| Requirements | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 |

**Responsibilities & Constraints**
- Zero direct dependency on controller communication, capnp, network I/O, filesystem, or wall-clock time (Req 5 AC 2) — enforced by the "Allowed Dependencies" link list, not just convention.
- `jbuf` cbuf capacity is hardcoded to 1 inside `xce_sched_core_init()` — never a parameter (Req 5 AC 7, pinned against the existing `njobs` FIXME in `scheduler.c:384`).
- All internal time via `xce_core_now()`; no direct `time()`/`clock_gettime()`/`gettimeofday()` calls anywhere in `core.c`.

**Dependencies**
- Inbound: NormalDriver, SimDriver — both call `core_push_events`/`core_go`, register `on_decision` (P0)
- Outbound: JobPicker (P0); `try_allocate()`/`can_preempt()` from sibling specs (P1 — signature-stable dependency, no behavior assumption)
- External: none

**Contracts**: Service [x] / State [x]

##### Service Interface
```c
int      xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed);
int      xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents);
int      xce_core_go(xce_sched_core_t*);
int      xce_core_end_session(xce_sched_core_t*);
uint64_t xce_core_now(xce_sched_core_t*);
int      xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out);
int      xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in);
void     xce_sched_core_fini(xce_sched_core_t*);
```
- Preconditions: `core_push_events()` only valid when no session is active (`session_active == 0`); `core_go()` returns `-EINVAL` if a session is already active. `xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()` are likewise only valid when `session_active == 0` — both return `-EBUSY` if called mid-session, rather than silently capturing or restoring torn state. Every caller in this design (record start, periodic checkpoint, simulation startup) already invokes them between sessions by construction; the `-EBUSY` return is a defensive contract check, not a path any correct caller should hit.
- Postconditions: after `on_decision` delivers the end-of-session `NULL` marker, `session_active` is guaranteed to transition to 0 on the subsequent `core_end_session()` call.
- Invariants: given identical `(config, event sequence)`, `core_push_events`+`core_go` produce an identical decision sequence (Req 5 AC 7) — the whole point of the design.

##### State Management
- State model: scheduling state (jobs/queues/workers/resources/resvals/users/projects) held in `kvmgr`s inside `xce_sched_core_t`, mirroring today's `xce_scheduler_t` layout minus `apictx`/`cluster_cfg`.
- Persistence & consistency: no persistence inside the core itself — `save_snapshot()`/`load_snapshot()` are the only serialization boundary, and only the driver decides when to invoke them.
- Concurrency strategy: three-thread cbuf pipeline unchanged from today; clock atoms (`virtual_clock_base`, `real_clock_base`, `speed_num`, `speed_den`) are lock-free `_Atomic uint64_t`, updated by the driver as a group before each event dispatch.

**Implementation Notes**
- Integration: `core_push_events`/`core_go`/`on_decision` are the only surface either driver touches — no driver reaches into `xce_sched_core_t` internals.
- Validation: `test_core_events.c` (contract-level), `test_sim_equivalence.c` (record→replay→diff, Req 5 AC 7.1 — the spec's central regression test).
- Risks: the `kvmgr` iteration-order audit (Req 5 AC 7, "Stabilized hash table iteration") is a Phase 2 deliverable with real but bounded scope — jobs/queues/users/projects only (per `research.md` and requirements.md's explicit Phase 2 task).

### Scheduler Driver

#### NormalDriver

| Field | Detail |
|-------|--------|
| Intent | Connects the core to a live controller; owns recording control |
| Requirements | 5.6, 5.8, 5.9, 5.10, 5.11 |

**Responsibilities & Constraints**
- Translates capnp push/notify into `xce_core_event_t`, buffers between sessions in `pend_updates` (existing dataq swap mechanism).
- Publishes each decision immediately via `xcecli_jobdispatch`/`xcecli_set_pend_reason`/`xcecli_preempt` as it drains from `decisions` cbuf — not batched at session end.
- Recording is boolean-gated: zero file I/O, zero extra allocation when `recording == false` (Req 5 AC 10, Performance).
- Validates `recordFile` before ever calling `fopen()`: reject `/` or `..`, resolve under `record_dir`, confirm containment via `realpath()` (Req 5 AC 10, "Record file path validation" — this design inherits, not re-derives, that requirement).
- `record_dir` is provisioned exactly one way: a `--record-dir DIR` scheduler startup CLI flag (default `/var/log/xce/`), parsed alongside the existing normal-mode arguments in `main.c` — never a capnp field on any request path (`SchedAdminRequest` included), so nothing reachable over the wire can redirect where recordings are written. Matches the existing `--sim-*` flag family's provisioning style (CLI-only, not config-pushed).

**Dependencies**
- Inbound: `main.c` (process entry point) (P0)
- Outbound: SchedulerCore (P0), `xcecli_ctx_t` (P0)
- External: `SchedAdminOpcode` dispatch (P1 — receives forwarded requests, not a caller relationship)

**Contracts**: Service [x] / Event [x]

##### Service Interface
```c
typedef struct {
    xce_sched_core_t     core;
    xcecli_ctx_t         apictx;
    xce_cluster_config_t cluster_cfg;
    dataq_t*             pend_updates[2];
    char*    record_dir;              /* from --record-dir CLI flag at startup; default "/var/log/xce/";
                                          never settable via any request-path field */
    char*    record_file_path;
    FILE*    record_file;
    bool     recording;
    uint64_t record_event_count;
    uint64_t record_snapshot_interval;
    uint64_t record_start_ts;
} xce_sched_driver_t;

int sched_driver_run(xce_sched_driver_t*, xce_conf4sched_t*, xce_cluster_config_t*, void* handler);
int sched_driver_record_start(xce_sched_driver_t*, const char* record_file /* bare filename or NULL */);
int sched_driver_record_stop(xce_sched_driver_t*);
int sched_driver_record_status(xce_sched_driver_t*, SchedAdminReply* out);
```
- Preconditions: `record_start` fails with `-EALREADY`-equivalent if already recording; `record_stop` fails if not recording.
- Postconditions: on `record_start` success, `ReplayHeader` + initial `SnapshotEntry` are durably on disk before the reply is sent.
- Invariants: `recording == true` iff `record_file != NULL`.

##### Event Contract
- Subscribed events: capnp push/notify from the controller (existing handler.c path, unchanged).
- Published events: decisions to the controller via `xcecli_ctx_t`; `SchedAdminReply` back through the forwarding path on recording-control requests.
- Ordering / delivery guarantees: decisions publish in allocator output order; recording writes events/decisions in the same order they cross the driver thread.

**Implementation Notes**
- Integration: `main.c`'s mode-selection branch calls `sched_driver_run()` when `--sim` is absent.
- Validation: recording lifecycle tested against a real filesystem (temp dir) — start/stop/status round trip, path-validation rejection cases, disk-full simulated via a read-only mount point.
- Risks: `OPCode_schedAdmin` ships with the same primary/standby gate as `OPCode_upgradeConfig` and no additional RBAC/auth check — a deliberate scope decision (project owner, 2026-08-02), not an oversight. Strengthening authentication for this opcode (and `config push`, if it also lacks one) is deferred to a follow-up spec. See Open Questions/Risks.

#### SimDriver

| Field | Detail |
|-------|--------|
| Intent | Connects the core to a `.replay` file; owns virtual clock, verification, and the REPL |
| Requirements | 5.13-5.19, 5.22 |

**Responsibilities & Constraints**
- Contains no `xcecli_ctx_t`, `xce_cluster_config_t`, capnp socket, or controller RPC of any kind (Req 5 AC 17) — enforced by the struct definition itself having no such fields.
- Discrete virtual clock: jumps directly to the next event's timestamp, never a fixed tick (Req 5 AC 14).
- REPL is a single `fgets` loop in the same thread as the simulation loop — no REPL thread (Req 5 AC 22, Implementation note).

**Dependencies**
- Inbound: `main.c` (P0, when `--sim` is present)
- Outbound: SchedulerCore (P0), ReplayFormat (P0)
- External: none

**Contracts**: Service [x] / State [x]

##### Service Interface
```c
typedef struct {
    xce_sched_core_t  core;
    FILE *replay_file, *alt_events_file, *record_file;
    uint64_t sim_speed_num, sim_speed_den;
    bool     verify;
    bool     interactive;
} xce_sched_sim_driver_t;

int sched_sim_driver_run(xce_sched_sim_driver_t*, /* CLI arguments struct */ const void* args);
```
- Preconditions: replay file's second message must be a `SnapshotEntry` (rejected otherwise, Req 5 AC 20).
- Postconditions: with `--sim-output`, every session cycle appends exactly one `ReplaySession` to the output file before the next `ReplayEvent` batch begins.
- Invariants: REPL `step`/`continue` to completion produces byte-identical output to batch `--sim` (Req 5 AC 22, "Deterministic replay is preserved").

##### State Management
- State model: same `xce_sched_core_t` as normal mode, loaded via `load_snapshot()` instead of live events.
- Persistence & consistency: `--sim-output` writes a new independent `.replay` file; never mutates the input file.
- Concurrency strategy: single-threaded driver loop around the same core thread pair; REPL blocks the driver loop intentionally (no wall-clock deadline in simulation).

**Implementation Notes**
- Integration: `main.c`'s mode-selection branch calls `sched_sim_driver_run()` when `--sim` is present.
- Validation: `test_sim_equivalence.c` drives this component directly (record via NormalDriver's test harness, replay via this one, diff).
- Risks: none beyond what's already captured under SchedulerCore (the determinism guarantee is a core property; SimDriver only needs to not violate it, e.g. by never bypassing `core_push_events`/`core_go`).

### Wire Protocol

#### ReplayFormat

| Field | Detail |
|-------|--------|
| Intent | `.replay` capnp stream schema and `SnapshotData` |
| Requirements | 5.20, 5.21, 6.5 |

**Responsibilities & Constraints**
- Additive-only changes to `controller.capnp` — reuses existing structs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) wherever they already exist (verified in `research.md`); only defines new structs where none exist.
- `SnapshotData` field shapes track the *actual current* C struct field sets (e.g. `xce_sched_user_t` is `{name}` only today, `xce_sched_project_t` is `{name}` only today) — the schema does not invent fields the C structs don't have yet.

**Dependencies**
- Inbound: SchedulerCore (`save_snapshot`/`load_snapshot`), NormalDriver (recording writer), SimDriver (reader/writer) (all P0)
- Outbound: none
- External: none

**Contracts**: Batch [x]

##### Batch / Job Contract
- Trigger: `record start` (normal mode) or `--sim-output` (simulation mode) write a `.replay` stream incrementally; `--sim`/`xceadmin scheduler dump` read one.
- Input / validation: first message must be `ReplayHeader`; second must be `SnapshotEntry` (Req 5 AC 20) — both enforced at read time, not just documented.
- Output / destination: append-only file; `record stop` / clean simulation completion writes `ReplayEvent{endSession}`.
- Idempotency & recovery: a write failure mid-message truncates cleanly to the last complete message (Req 5 AC 10, error handling) — no partial-message corruption possible because each message is a full capnp frame written atomically at the `fwrite` level.

**New capnp structs** (additive to `controller.capnp`; `ReplayHeader`/`ReplayEvent`/`ReplaySession`/`ReplayDecision`/`SessionSummary` wrapper structs already fully specified in requirements.md Req 5 AC 20 — not repeated here):

```capnp
# New event payloads (JOBS_PUSH/WORKERS_PUSH/etc reuse existing Conf4Sched,
# SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages —
# verified present in controller.capnp, research.md).

struct JobStateChangeEvent $C.mapname("job_state_change_event_t") {
  jobid    @0: UInt64;
  oldState @1: UInt32;
  newState @2: UInt32;
  exitCode @3: Int32;
}

struct JobUpdateEvent $C.mapname("job_update_event_t") {
  jobid         @0: UInt64;
  changedFields @1: UInt32;   # bitmask: JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ
  job           @2: Job;      # full updated record; reuses existing Job struct
}

# New decision payloads for ReplayDecision.body (Req 5 AC 5).
# DISPATCH/PEND reuse existing JobAlloc/JobPendReason; PREEMPT is new
# (wire shape owned here per Boundary Commitments; victim-selection
# semantics owned by scheduler-preemption — see research.md).
#
# AC 5's field list for DISPATCH is "jobid, workerid, allocations[]"
# (singular workerid). JobAlloc already generalizes this correctly for
# multi-host jobs: JobAlloc.jobid covers jobid, JobAlloc.allocs is
# List(HostAlloc) with each HostAlloc.host as that allocation's worker
# id and HostAlloc.allocs: List(ResAlloc) as its resource/amount pairs
# -- verified against controller.capnp, no new fields needed. jobid
# lives ONLY in JobAlloc.jobid (not duplicated at DispatchDecision
# level) to avoid two unsynchronized sources of truth for the same job.

struct DispatchDecision $C.mapname("sched_dispatch_decision_t") {
  alloc  @0: JobAlloc;
}

struct PendDecision $C.mapname("sched_pend_decision_t") {
  jobid  @0: UInt64;
  reason @1: JobPendReason;
}

struct PreemptVictim $C.mapname("preempt_victim_t") {
  jobid    @0: UInt64;
  workerid @1: Text;
}

struct PreemptDecision $C.mapname("sched_preempt_decision_t") {
  jobid   @0: UInt64;
  victims @1: List(PreemptVictim);
  reason  @2: JobPendReason;
}

# SnapshotData — full core-state checklist from requirements.md Req 5 AC 21,
# given a concrete wire shape. Reuses existing List(...) payload structs
# for domain objects already carried elsewhere on the wire; the two
# scheduler-internal-only caches (shared-resource tracking, unsatisfiable-
# resreq cache) and session/PRNG state have no existing external contract,
# so they are versioned opaque blobs the core alone reads/writes -- no
# consumer outside save_snapshot()/load_snapshot() ever inspects their
# internal shape (WHAT not HOW: the boundary that matters is "round-trips
# through the core, opaque to everyone else").

struct SnapshotData $C.mapname("sched_snapshot_data_t") {
  config          @0: Conf4Sched;
  jobs            @1: SchedJobData;
  queues          @2: List(QueueConf);
  workers         @3: WorkerData;
  resources       @4: ResourceData;
  resvals         @5: ResValList;
  users           @6: List(SchedUser);
  projects        @7: List(SchedProject);
  prngSeed        @8: UInt64;
  internalState   @9: Data;   # shared-resource tracking + unsatisfiable-resreq
                              # cache + session/iterator state, core-opaque
  internalVersion @10: UInt32; # format tag for internalState, so a future
                                # incompatible internal layout fails loud
                                # instead of misinterpreting old bytes
}

struct SchedUser $C.mapname("xce_sched_user_t") {
  name @0: Text;   # matches xce_sched_user_t today (name only) --
                   # extend in lockstep if xce_sched_user_t gains fields
}

struct SchedProject $C.mapname("xce_sched_project_t") {
  name     @0: Text;
  priority @1: Int32;
}
```

**Implementation Notes**
- Integration: `save_snapshot()`/`load_snapshot()` are the only functions that construct/consume `SnapshotData`; no other component touches `internalState`'s bytes.
- Validation: as-built, round-trip coverage for every new struct lives in `src/controller/testsuite/test_e2e_capnp.c` (task 7.5), not a dedicated `test_replay_format.c` — this repo's existing convention is one shared capnp round-trip test file per schema, and this spec's new structs were added to it rather than starting a second one; `internalVersion` mismatch is a dedicated negative test within it.
- Risks: `internalState`'s opaque-blob design trades wire-level inspectability for implementation flexibility — acceptable because AC 21 only requires the *core* to round-trip correctly, never that `xceadmin scheduler dump` render its internals meaningfully (dump can print `internalState`'s byte length and `internalVersion` only).

### Controller

#### SchedAdminOpcode

| Field | Detail |
|-------|--------|
| Intent | Pass-through dispatch for `OPCode_schedAdmin` in the controller's request handler |
| Requirements | 5.10 |

**Responsibilities & Constraints**
- Structurally mirrors the existing `OPCode_upgradeConfig` case (`cluster.c:670-705`) for the primary-gate/delegate/explicit-reply shape: primary-only gate via `ha_is_primary()`, explicit reply on every path, 5s timeout.
- Owns zero recording state — every field of `SchedAdminRequest`/`Reply` passes through untouched.
- **Blocking-wait admission (hardened 2026-08-19, design review follow-up)**: like `OPCode_upgradeConfig` (in-process commit-barrier wait, `COMMIT_WAIT_RESERVED`-admitted), this case now also admission-gates its blocking wait. Originally (accepted, project owner, 2026-08-02) this case blocked its `vtask` handler thread for up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock`, with no reservation — a boundary case reviewed against `ha-async-io.md`'s vtaskmgr admission rule and accepted on the grounds that `xceadmin scheduler` commands are cluster-administrator-triggered, low-frequency, human-paced traffic, not expected to burst concurrently the way heartbeat/replication traffic does. A retrospective design review flagged this as a real (if low-probability) starvation risk under a burst of concurrent/scripted requests, since each blocked vtask thread occupies a worker for up to 5s and `c->sched.lock` already serializes them one at a time — N concurrent requests could occupy N threads for up to N×5s. Fix: `vtaskmgr_blocking_acquire(&c->vtaskmgr)`/`vtaskmgr_blocking_release(&c->vtaskmgr)` now wrap the ENTIRE forward+wait region (acquired before `mylock_lock(&c->sched.lock)`, released on every exit path) — the same shared budget `commit.c`'s `wait_admit_locked()`/`wait_release_locked()` already gate on (wired via `commit_set_pool(&c->commit, &c->vtaskmgr)`). Admission-exhausted requests fast-reject with `{ok=false, error="server busy, try again"}` before ever touching `c->sched.lock` — never queued. See requirements.md Req 5 AC 10's own Amendment for the full rationale.

**Dependencies**
- Inbound: XceAdminSchedulerCLI, via TLS+capnp (P0)
- Outbound: NormalDriver (P0, forwards the request)
- External: none

**Contracts**: API [x]

##### API Contract
| Method | Endpoint (opcode) | Request | Response | Errors |
|--------|--------------------|---------|----------|--------|
| Request/Reply | `OPCode_schedAdmin` | `SchedAdminRequest` | `SchedAdminReply` | `-EAGAIN` (not primary), `"scheduler not connected"`, `"request timeout"`, `"invalid record file name"`, `"already recording"`, `"not recording"`, `"server busy, try again"` (admission exhausted) (full table: requirements.md Req 5 AC 10) |

**Implementation Notes**
- Integration: one new `case` in `myreq_handler`'s switch (`cluster.c`); no new file.
- Validation: `test_sched_admin_dispatch.c` (`src/controller/testsuite/`; named differently from this plan's original `test_sched_admin.c`, same coverage) asserting the reply-on-every-path property (standby rejection, disconnected-scheduler rejection, timeout) — mirrors existing `OPCode_upgradeConfig` test coverage style.
- Risks: no RBAC/auth check is visible for the comparable `OPCode_upgradeConfig` case in `cluster.c` (see `research.md`), and `OPCode_schedAdmin` inherits that same gap by deliberate decision — auth hardening for this opcode is deferred to a follow-up spec, not silently accepted as permanent.

### Tooling

#### XceAdminSchedulerCLI

| Field | Detail |
|-------|--------|
| Intent | `xceadmin scheduler record start/stop/status` and `xceadmin scheduler dump <FILE>` |
| Requirements | 5.10 |

**Responsibilities & Constraints**
- `cli.py` stays a thin `click` layer; all logic lives in the new `scheduler.py` module, mirroring `push.py`'s split.
- `record_start/stop/status()` go through `xceadmin.discovery.follow_redirects` + `decode_reply` (request/reply opcode, needs redirect-following, unlike `probe_config()`).
- `dump()` has no network dependency — decodes a local `.replay` file only.

**Dependencies**
- Inbound: operator CLI invocation (P0)
- Outbound: `xcecommon.wire`/`schema`, `xceadmin.discovery` (P0); SchedAdminOpcode via the wire (P0, for record/status only)
- External: none

**Contracts**: API [x]

##### API Contract
| Method | Endpoint | Request | Response | Errors |
|--------|----------|---------|----------|--------|
| CLI | `scheduler record start [FILE]` | `SchedAdminRequest{recordStart}` | `SchedAdminReply` | `XceAdminError` subclasses (connection, redirect, decode) |
| CLI | `scheduler record stop` | `SchedAdminRequest{recordStop}` | `SchedAdminReply` | same |
| CLI | `scheduler record status` | `SchedAdminRequest{recordStatus}` | `SchedAdminReply` | same |
| CLI | `scheduler dump <FILE>` | (local file) | formatted text to stdout | `SchemaNotFoundError`, file-not-found |

**Implementation Notes**
- Integration: `cli.py` imports `record_start`/`record_stop`/`record_status`/`dump` from `scheduler.py`, same import shape as today's `build_cluster_config`/`push_config`.
- Validation: `test_scheduler.py` — mock-TLS-server tests for the three record commands (mirrors `test_config_push_mock.py`), plus golden-file tests for `dump()` against a small fixture `.replay` file.
- Risks: none beyond ReplayFormat's (this component only formats/transmits, never interprets scheduling semantics).

## Data Models

Covered in full under **Components and Interfaces → ReplayFormat** above (new capnp structs) — not duplicated here per the dedup guidance in design-principles.md. `SnapshotData`'s reuse of existing `Conf4Sched`/`SchedJobData`/`WorkerData`/`ResourceData`/`ResValList`/`QueueConf` is the only cross-cutting data-model decision; see `research.md`, "Reusable capnp payload types," for the verification trail.

## Error Handling

### Error Strategy
Fail-loud, no silent fallback — consistent with the controller's existing "fail loud, stay alive" design invariant (`src/controller/CLAUDE.md`), applied to the scheduler process at the scope this feature touches.

### Error Categories and Responses
- **Configuration errors** (unregistered picker plugin, Req 2 AC 5): scheduler startup fails with a clear error — no silent fallback to `builtin_fcfs`.
- **Recording errors** (disk full, permission error, invalid `recordFile`): logged, recording state cleanly torn down (`recording=false`, file closed, `record_file_path` freed), scheduling continues unaffected (Req 5 AC 10, error handling) — recording failure is never fatal to the scheduler process.
- **Simulation errors** (missing/corrupt replay file, missing initial `SnapshotEntry`): simulation driver fails at startup with a clear message — no partial/best-effort simulation.
- **Admin-path errors** (`OPCode_schedAdmin`): every failure mode gets an explicit `{ok=false, error=...}` reply — never a dropped request (mirrors the "no silent drop" precedent already established for replication-class messages, `src/controller/CLAUDE.md`'s protocol checklist).

### Monitoring
No new metrics infrastructure — `SessionSummary` (Req 5 AC 12) provides session-duration diagnostics through the existing `.replay` recording path; no separate telemetry system is introduced.

## Testing Strategy

- **Unit — Picker contract** (`test_picker_fcfs.c`): feed `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly, assert `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order — verifies Req 3.2-3.4 without a live core.
- **Unit — Core event contract** (`test_core_events.c`): `core_push_events()` followed by `core_go()` with a stub picker asserts state-change hooks fire exactly once per event, synchronously, before `core_go()` returns — verifies Req 5.1-5.8.
- **Unit — Clock virtualization** (`test_clock.c`): `xce_core_now()` against the four mode rows in Req 5 AC 3's table (normal, discrete, real-time, fast/slow) — asserts the two-base+speed formula directly, no wall-clock flakiness (fixed `real_clock_base`/mocked `clock_gettime` delta).
- **Unit — Replay format round-trip** (as-built: `src/controller/testsuite/test_e2e_capnp.c`, not a dedicated `test_replay_format.c` — added to this repo's existing shared capnp round-trip test file instead): encode/decode every new capnp struct (`JobStateChangeEvent`, `JobUpdateEvent`, `DispatchDecision`, `PendDecision`, `PreemptDecision`, `SnapshotData`) — field-for-field equality after round-trip.
- **Integration — Deterministic replay** (`test_sim_equivalence.c`, Req 5.7/5.7.1): record a synthetic scheduling session through `NormalDriver`'s test harness → replay through `SimDriver` → assert `ReplayDecision` sequences match exactly. This is the design's highest-value test — it directly exercises the central invariant.
- **Integration — Recording lifecycle** (driver-level, Req 5.10-5.11): `record start` → submit synthetic events → `record stop` → verify file structure (`ReplayHeader`, `SnapshotEntry`, alternating `ReplayEvent`/`ReplaySession`, periodic checkpoints at the configured interval, final `endSession`).
- **Integration — `OPCode_schedAdmin` dispatch** (`test_sched_admin_dispatch.c`, controller-level; named differently from this plan's original `test_sched_admin.c`): standby rejection, disconnected-scheduler rejection, timeout, and successful forward/reply round trip — mirrors existing `OPCode_upgradeConfig` test coverage.
- **Integration — xceadmin scheduler CLI** (`test_scheduler.py`): mock-TLS-server tests for `record start/stop/status` (redirect-following, reply decoding) plus golden-file tests for `dump()` against a fixture `.replay` file.
- **E2E — HA soak regression** (Req 4.3, Req 5 per-phase verification): `/xce-test-ha` sequential and chaotic rounds after each phase, asserting zero new job loss/divergence attributable to this feature — required before each phase merges, not a one-time gate at the end.
- **E2E — REPL equivalence** (Req 5.22): `--interactive` with `step`/`continue` to completion produces the same decision sequence as batch `--sim` on the same replay file.

## Security Considerations

_Feature-specific decisions only; baseline RBAC/auth practices are in `.kiro/steering/security.md`._

- **`recordFile` path validation** is already fully specified at the requirements level (Req 5 AC 10, "Record file path validation" — reject `/`/`..`, resolve under configured `record_dir`, `realpath()` containment check before `fopen()`). This design's `NormalDriver` and `SchedAdminOpcode` components inherit that requirement directly. `record_dir` itself is provisioned only via a `--record-dir` scheduler startup CLI flag — never from `SchedAdminRequest` or any other request-path field — so the validation boundary can't be redirected by a caller.
- **`OPCode_schedAdmin` ships with no RBAC/auth check beyond the primary/standby gate — accepted scope decision, deferred hardening.** The comparable `OPCode_upgradeConfig` case has no visible RBAC/auth check in `cluster.c` either (see `research.md`). Project owner decision (2026-08-02): authentication for this opcode is enhanced in a follow-up spec, not blocking this one. Phase 3 ships `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current (unauthenticated-beyond-primary-gate) posture — this is a known, accepted gap, not a silent one.
- **`.replay` files may contain sensitive cluster state** (full job/user/resource snapshots) — file permissions on `record_dir` are an operational concern (steering-level, not new to this feature) but worth calling out since this is the first feature to write full scheduler snapshots to disk in a human-decodable format (`xceadmin scheduler dump`).

## Performance & Scalability

_Feature-specific targets only; general practices are in steering docs._

- **Recording overhead**: zero file I/O, zero extra allocation when `recording == false` — a single boolean branch per session (Req 5 AC 10, explicit requirement, inherited here as a design constraint on `NormalDriver`).
- **Determinism vs throughput**: `jbuf` cbuf capacity is permanently 1 (rendezvous, not a buffer) — this bounds picker→allocator throughput to strictly sequential handoff. This is an intentional trade-off (determinism over pipelining) already made at the requirements level (Req 5 AC 7) and is not revisited here; any future throughput work must find a different mechanism that preserves the ordering guarantee (e.g. deterministic batching with an explicit order tag), not simply raise the cbuf capacity.
- **Simulation fast mode**: `speed_num=0` means the virtual clock never advances between events at wall-clock cost — `--sim` on a large `.replay` file runs at CPU-bound speed, not wall-clock speed. No specific throughput target is set here (none given in requirements); implementation should avoid introducing any per-event syscall beyond what decoding the capnp stream requires.
- **`OPCode_schedAdmin` vtask occupancy (hardened 2026-08-19)**: each request blocks its vtask handler thread up to 5s (cross-process wait on the scheduler's reply) while holding `c->sched.lock` — now gated by the same `vtaskmgr_blocking_acquire`/`_release` admission reservation `commit.c`'s commit-wait barrier shares (see `SchedAdminOpcode` component above). Admission-exhausted bursts fast-reject instead of occupying vtask threads; not separately load-tested against a real concurrent burst (unit-tested only, `test_schedadmin_fast_rejects_when_admission_exhausted`), but the mechanism itself is the same one already proven under `test_vtask_admission.c` and production commit-wait traffic.

## Open Questions / Risks

1. **`OPCode_schedAdmin` authorization mechanism** (P2 — accepted gap, deferred). Ships at parity with `OPCode_upgradeConfig`'s current posture (primary/standby gate only, no RBAC/auth check). Project owner decision (2026-08-02): auth hardening for this opcode is a follow-up spec, not a Phase 3 blocker. Tracked in `.kiro/steering/roadmap.md`'s existing `peer-authentication-hardening` backlog item (updated 2026-08-02 to cover `OPCode_schedAdmin` alongside `OP_upgradeConfig` — same `validate_host()` gate closes both). See Security Considerations and `research.md`.
2. **`PreemptDecision`'s wire shape is pre-committed ahead of `scheduler-preemption`'s own design** (P1). Flagged as a revalidation trigger in Boundary Commitments; `scheduler-preemption`'s design should explicitly confirm or extend this shape early, not discover it mid-implementation.
3. **`SnapshotData.internalState` opaque-blob scope** (P2). If a future spec needs `xceadmin scheduler dump` to render shared-resource-tracking or unsatisfiable-resreq-cache contents meaningfully (not just byte length), `internalState` will need to become a structured capnp field instead of opaque `Data` — deferred here because no current requirement asks for it (AC 20's dump description only requires human-readable output for what dump already needs to show, and internals were never asked to be inspectable).
4. **`xce_sched_core_t`'s `config`/`plugins`/`jobs`/`queues`/`workers`/`users`/`projects` field block is left as `var ...` in requirements.md's data-structure summary** (not fully typed) — Phase 2 implementation must finalize these as concrete `kvmgr`/pointer types when `core.c` is written; this design does not re-type them since requirements.md already commits to "moved from `xce_scheduler_t`" as the authoritative source of truth for their shapes.
5. ~~**`OPCode_schedAdmin` blocks a vtask handler thread up to 5s per request with no admission reservation**~~ **FIXED 2026-08-19** (design-review follow-up). Now gated by `vtaskmgr_blocking_acquire`/`_release` against the same `c->vtaskmgr` reservation `commit.c`'s commit-wait barrier shares; admission-exhausted requests fast-reject with `{ok=false, error="server busy, try again"}` instead of blocking. See `SchedAdminOpcode` component above and `research.md`'s "SUPERSEDES" note.
6. ~~**Req 5 AC 3's tlog virtual-clock logging integration is built but never activated**~~ **FIXED 2026-08-19** (design-review follow-up). `sched_sim_driver_start()` (`sim.c`) now calls `xce_core_register_tlog_format()`/`xce_core_set_active_for_logging(&sim->core)` right after `sim_set_virtual_clock()`, exactly at the site this entry identified; `sched_sim_driver_run()` (`sim_loop.c`) deactivates (`xce_core_set_active_for_logging(NULL)`) on all 7 of its return paths to avoid the ASAN stack-use-after-scope hazard this entry flagged — confirmed by fixing a real instance of exactly that hazard, caught live when the new test was added (bug-1598, `test_sim_event_loop.c`'s fixtures switched to `static` storage). No sim-driver fini function was added; the per-return-path deactivation is the shipped mitigation instead.
7. **`plugins/fcfs.c` vs `plugins/builtin_fcfs.c` split was undocumented in this design until 2026-08-07** (P3 — documentation gap only, now fixed here). Found by `/kiro-validate-impl`: the file-structure and BuiltinFcfsPicker sections above originally said `fcfs.c` would be rewritten; the actual implementation added a new `builtin_fcfs.c` and deliberately left `fcfs.c` untouched (task 1.2's compat-wrapper exercise target). Functionally correct and confirmed live (`builtin_fcfs` is the registered default), just wasn't reflected here until this update. Tasks 2.1-2.6 in tasks.md still carry no `_Verified:` annotations (unlike every other task group) — worth backfilling for traceability.