lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(scheduler-core): commit design.md/research.md, sync design.md with as-built code

Browse source 
Both files existed only on disk since the design phase (2026-08-03) and
were never committed despite ~40 implementation commits landing since.
Also corrects design.md drift found by /kiro-validate-impl: the File
Structure Plan and BuiltinFcfsPicker sections claimed plugins/fcfs.c
would be rewritten in place, but the actual implementation added a new
plugins/builtin_fcfs.c and deliberately kept fcfs.c untouched as the
job_elect_fn compat-wrapper's exercise target. Updates the directory
tree to the real file layout (core_events/session/clock.c, sim.c/
sim_loop.c/sim_record.c/sim_verify.c, snapshot/recording/repl splits,
test_e2e_capnp.c vs the planned test_replay_format.c, test_sched_admin_dispatch.c
vs test_sched_admin.c), and adds two Open Questions/Risks entries for
the AC 3 tlog-logging-integration gap and this fcfs.c/builtin_fcfs.c
documentation gap, both found during validation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-08 03:19:01 +00:00
parent b5205ab043
commit 7a5a864fb6
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
2 changed files with 891 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

794
.kiro/specs/scheduler-core/design.md Normal file
View file

@ -0,0 +1,794 @@
# Technical Design: scheduler-core
## Overview
**Purpose**: This feature replaces the scheduler's hardcoded FCFS job-selection logic and its hard-wired coupling to the live controller with a pluggable picker interface and a passive, mode-agnostic scheduling core. The core is driven by either a normal-mode driver (live controller) or a simulation-mode driver (recorded `.replay` file), and both modes are required to produce byte-identical scheduling decisions given identical input — this is the spec's central, testable invariant.
**Users**: Cluster operators configure job-picking policy without a rebuild (Req 2) and gain a `xceadmin scheduler record`/`dump` workflow. Scheduler developers gain a `--sim`/`--interactive` simulation driver to debug and tune scheduling policy against recorded production traces without a live cluster (Req 5).
**Impact**: `xce_scheduler_t` (today: scheduling state + `apictx` + `cluster_cfg` in one struct, `scheduler.c`) splits into `xce_sched_core_t` (pure scheduling state, zero I/O) plus `xce_sched_driver_t` (normal, controller-attached) and `xce_sched_sim_driver_t` (simulation, replay-attached). The hardcoded `"xce_sched_fcfs"` string at `scheduler.c:43` is replaced by a configurable, plugin-registered picker. A new controller opcode (`OPCode_schedAdmin`) and a new xceadmin command group (`scheduler record`/`scheduler dump`) are added as pass-through admin plumbing.
### Goals
- Job-picking policy configurable per-scheduler and per-queue without recompilation, with `builtin_fcfs` shipping as the zero-behavior-change default (Req 1-4).
- Scheduling core with zero controller I/O, wall-clock, or filesystem dependencies, driven identically by a normal or simulation driver (Req 5 AC 1-8).
- Runtime recording of live scheduling sessions to a self-contained `.replay` file, and a simulation driver + interactive REPL that replays it deterministically (Req 5 AC 9-22).
- Normal↔sim equivalence as a verifiable regression test: record a live session, replay it, diff the decisions — zero divergence (Req 5 AC 7.1).
### Non-Goals
- New job-picking policies beyond `builtin_fcfs` (interface only; sibling work).
- Allocation iterators / `try_allocate()` internals (`scheduler-allocation`).
- Preemption policy / victim selection (`scheduler-preemption`) — this design defines `PreemptDecision`'s **wire shape** only (needed for `Req 5 AC 5`'s decision table to compile), not preemption logic.
- Hierarchical project resource quotas (`project-resource-ownership`).
- REPL state mutation, reverse execution, or scripting (explicitly deferred in Req 5 AC 22).
## Boundary Commitments
### This Spec Owns
- `xce_job_picker_iface_t` and the `builtin_fcfs` implementation. As-built: a new `src/scheduler/src/plugins/builtin_fcfs.c` implements the interface; `plugins/fcfs.c` was deliberately left unchanged (still `job_elect_fn`-only) as the live exercise target for `plugin.c`'s legacy compat wrapper (see Components — BuiltinFcfsPicker).
- `xce_sched_core_t`: all scheduling state, the picker/allocator thread pair, the driver↔core protocol (`core_push_events`, `core_go`, `on_decision`), and `xce_core_now()` clock virtualization.
- `xce_sched_driver_t` (normal-mode): controller event translation, decision publishing, and recording control (start/stop/status, periodic snapshot).
- `xce_sched_sim_driver_t` (simulation-mode): `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`, and the interactive REPL.
- The `.replay` capnp wire format (`ReplayHeader`, `SnapshotEntry`, `ReplayEvent`, `ReplaySession`, `ReplayDecision`, and their payload sub-structs) in `src/schema/controller.capnp`.
- The `OPCode_schedAdmin` request/reply pair and its **pass-through** dispatch case in the controller's `myreq_handler` (`src/controller/cluster.c`), plus its accompanying `src/controller/testsuite/` coverage — this spec's only footprint outside `src/scheduler/`.
- `xceadmin scheduler record start|stop|status` and `xceadmin scheduler dump <FILE>` (new `src/xceadmin/src/xceadmin/scheduler.py`).
- `job_pick_plugin` (global + per-queue) and `project_name` (per-queue) fields; `xce_sched_project_t` scaffolding (name + priority only — no quota logic).
### Out of Boundary
- `try_allocate()` and the allocation iterator (`scheduler-allocation` owns the implementation; this spec only calls it from the allocator thread, unchanged signature).
- `can_preempt()` and victim-selection semantics (`scheduler-preemption` owns the algorithm; this spec owns only `PreemptDecision`'s wire shape, see Data Models).
- Hierarchical project quota, `resource_quota`/`resource_used`/`parent_project` on `xce_sched_project_t` (`project-resource-ownership`).
- Any controller state or persistence for recording (`OPCode_schedAdmin` is explicitly a stateless pass-through — the controller commits nothing, owns no WAL entry for it).
- REPL write-side mutation, semantic snapshot diff in xceadmin (both explicitly deferred, Req 5 AC 22).
### Allowed Dependencies
- Existing plugin framework (`plugin.h`/`plugin.c`) — `xce_job_picker_iface_t` is a new plugin category registered through the same `plugin_register`/`key_find` mechanism, not a parallel system.
- Existing select-expression evaluator (`resreq.h`, `select_eval_ctx_t`) — used as-is by the allocator thread; the core does not re-implement or wrap it.
- libxce data-structure subset only inside the core: `ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti` (Req 5 AC 3). The core must **not** link `netio`, `delay`, `xceapi`, or `idgen`.
- `xcecli_ctx_t` API (`xcecli_jobdispatch`, `xcecli_set_pend_reason`, `xcecli_preempt`) — normal-mode driver only, never referenced from `xce_sched_core_t` or `xce_sched_sim_driver_t`.
- `xcecommon.wire`/`xcecommon.schema`/`xceadmin.discovery` (Python) — the new `xceadmin/scheduler.py` module reuses these exactly as `push.py` does; it does not open its own socket or reload the schema.
### Revalidation Triggers
- Any change to `xce_job_picker_iface_t`'s hook signatures — every picker plugin (present and future) must be re-verified.
- Any change to `jbuf` cbuf capacity away from 1 — invalidates the determinism argument in Req 5 AC 7; `scheduler-allocation` and `scheduler-preemption` must both re-verify ordering assumptions.
- Any change to the `.replay` capnp schema's message set or `ReplayDecision.body` union — `scheduler-preemption`'s `PreemptDecision` consumer and any external tooling reading `.replay` files must re-check compatibility.
- Any change to `OPCode_schedAdmin`'s request/reply shape — `xceadmin.scheduler` module must be updated in lockstep (same repo, same PR).
## Architecture
### Existing Architecture Analysis
Today, `xce_scheduler_t` (`src/scheduler/include/scheduler.h`) owns scheduling state (`jobs`, `queues`, `workers`, `plugins` kvmgrs), controller I/O (`apictx: xcecli_ctx_t`, `cluster_cfg: xce_cluster_config_t`), and three threads (`jobpicker`, `allocator`, main) wired through three `cbuf`s (`wbuf` cap 1, `jbuf` cap `njobs`≡1 today, `decisions` cap `ndecisions`≡5). `main.c`'s loop calls `sched_update` → `sched_begin_session` → `sched_go` → drains `decisions` cbuf with a hardcoded `switch (decision->op)` → `sched_end_session`. The picker plugin (`fcfs.c`) is selected by a literal string (`scheduler.c:43`) and implements the old `xce_sched_plugin_t->job_elect_fn` hook, tightly coupled to `sched->queues`/`sched->jobs` kvmgrs directly.
**Preserved without change**: the three-thread/three-cbuf pattern (`wbuf`→picker, `jbuf`→allocator, `decisions`→driver); `plugin_register`/`key_find` plugin lookup; `select_eval_ctx_t` per-session memoization; `sched_queue_is_ready()`/`sched_job_is_ready()` (currently stub `return 1;` in `scheduler.c`, called as-is by both today's `fcfs_elect_job` and the new `builtin_fcfs`).
**What changes**: `xce_scheduler_t` splits along the controller-I/O seam into `xce_sched_core_t` (no `apictx`/`cluster_cfg`) and `xce_sched_driver_t` (holds them). The picker plugin category becomes `xce_job_picker_iface_t`, replacing `job_elect_fn` (deprecated with a compatibility wrapper, Req 5 AC "Migration from old job_elect_fn").
### Architecture Pattern & Boundary Map
```mermaid
graph TB
subgraph DriverLayer[Driver Layer]
NormalDriver[Normal Driver]
SimDriver[Simulation Driver]
Controller[Live Controller]
ReplayFile[Replay File]
Controller --> NormalDriver
ReplayFile --> SimDriver
end
subgraph CoreLayer[Scheduler Core]
Core[Scheduler Core State]
Picker[Job Picker Plugin]
Allocator[Allocator Thread]
Core --> Picker
Picker --> Allocator
end
NormalDriver -->|core_push_events core_go| Core
SimDriver -->|core_push_events core_go| Core
Core -->|on_decision| NormalDriver
Core -->|on_decision| SimDriver
NormalDriver -->|xcecli| Controller
SimDriver -->|write| RecordFile[Sim Output Replay File]
subgraph AdminPath[Admin Control Path]
XceAdmin[xceadmin scheduler CLI]
SchedAdminOp[OPCode_schedAdmin dispatch]
XceAdmin --> SchedAdminOp
SchedAdminOp --> Controller
end
Controller -->|forward request-reply| NormalDriver
```
**Architecture Integration**:
- **Selected pattern**: Event-sourced core with a swappable driver (ports-and-adapters, applied narrowly — the core is the hexagon, the two drivers are the only adapters). Selected in requirements.md as the only pattern that makes normal↔sim equivalence a structural property rather than a discipline (see `research.md`, Architecture Pattern Evaluation).
- **Domain boundaries**: Core owns scheduling *decisions*; drivers own I/O translation *only* — no scheduling logic in either driver. `OPCode_schedAdmin` dispatch is a third, thin adapter (controller-side) that never touches core or driver state directly, only forwards bytes.
- **Existing patterns preserved**: three-thread/cbuf pipeline, plugin registration, select-expression memoization.
- **New components rationale**: `SimDriver` and the `.replay` format are net-new (no existing analog); `SchedAdminOpcode` is new but structurally mirrors the existing `OPCode_upgradeConfig` case (see `research.md`).
- **Steering compliance**: no new wall-clock/network dependency inside `src/scheduler/`'s core files (`ha-async-io.md`'s async-I/O discipline extended to a synchronous core by design, not violated — the core has no I/O at all); `.capnp` used for the new wire format per `structure.md`'s "Cap'n Proto for wire formats" convention.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Scheduler core / drivers | C (gnu17) | `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t` | No new language/runtime; existing `src/scheduler/` toolchain |
| Wire format | Cap'n Proto (existing `controller.capnp`) | `.replay` stream, `OPCode_schedAdmin` request/reply | Extends the existing schema file, not a new one |
| Admin CLI | Python / `click` (existing xceadmin stack) | `scheduler record`/`scheduler dump` commands | New module (`xceadmin/scheduler.py`) in the existing package, same `pycapnp` schema loader |
| Build | Meson ≥ 1.5 (existing) | New `.c` files added to `src/scheduler/meson.build`; testsuite entries | No new build system |
### Dependency Direction
`Picker Plugin Interface → Scheduler Core → {Normal Driver, Simulation Driver} → {Controller (xcecli), Replay File I/O}`. Controller-side `SchedAdminOpcode` dispatch depends only on the capnp schema (shared, not owned by the core) and forwards to the driver without depending on `xce_sched_core_t` internals. xceadmin's `scheduler.py` depends on `xcecommon`/`xceadmin.discovery` only — never on any C header. Each layer imports only from layers to its left; the core never imports driver headers (enforced by the "Allowed Dependencies" list above — no `xcecli_ctx_t`/`netio` symbol may appear in `core.c`).
## File Structure Plan
### Directory Structure
**As-built** (updated 2026-08-07 after implementation; see tasks.md's `## Implementation Notes` for the full rationale behind each split — all were 800-line-file-cap splits made during implementation, not scope changes):
```
src/scheduler/
├── include/
│ ├── scheduler.h # unchanged Phase 1; Phase 2 shrinks to xce_scheduler_t compat shim
│ ├── plugin.h # + xce_job_picker_iface_t (Req 1 AC 2), deprecation of job_elect_fn
│ ├── core.h # xce_sched_core_t, xce_core_event_t, decision cb type
│ ├── driver.h # xce_sched_driver_t (normal-mode)
│ ├── sim.h # xce_sched_sim_driver_t -- built as sim.h, not sim_driver.h (task 11.1, documented deviation)
│ ├── recording.h # record start/stop/status types, ReplayEvent/ReplaySession writer contract
│ ├── repl.h # REPL hook wrappers consumed by sim_loop.c -- no-op unless sim->interactive
│ ├── decision.h # pre-existing; extended with XCE_DECISION_PREEMPT (task 4.4)
│ └── sched/types.h # + job_pick_plugin, project_name on xce_sched_queue_t (Req 2)
├── src/
│ ├── scheduler.c # sched_job_picker(): job_elect_fn != NULL delegates to the legacy loop, NULL runs the generic xce_job_picker_iface_t loop
│ ├── core.c # xce_sched_core_init/fini, jbuf(cap=1)/decisions cbuf plumbing
│ ├── core_events.c # xce_core_push_events() -- split out of core.c (800-line cap; task 4.3)
│ ├── core_session.c # xce_core_begin_session/xce_core_end_session -- split out of core.c (task 4.4)
│ ├── core_clock.c # xce_core_now(), tlog format integration -- the ONE file allowed to call clock_gettime() (task 4.5)
│ ├── driver.c # normal-mode driver: capnp<->core event translation, decision publishing, deferred admin-request handling
│ ├── recording.c # record start/stop/status, header/snapshot writes
│ ├── recording_events.c # ReplayEvent writes -- split out of recording.c (task 9.2/9.3)
│ ├── recording_session.c # ReplaySession begin/end writes -- split out of recording.c (task 9.2/9.3)
│ ├── repl.c # REPL command loop core (execution control, Req 5 AC 22)
│ ├── repl_break.c # REPL breakpoints + event/decision hook wrappers -- split out of repl.c (task 12.2)
│ ├── repl_info.c # REPL state-inspection commands -- split out of repl.c (task 12.2)
│ ├── repl_params.c # REPL snapshot/simulation-parameter commands -- split out of repl.c (task 12.3)
│ ├── repl_internal.h # shared repl_*.c internal declarations (task 12.2)
│ ├── snapshot.c # save_snapshot()/load_snapshot() orchestration, picker rebuild
│ ├── snapshot_wire.c # SnapshotData capnp encode/decode -- split out of snapshot.c (task 8.1)
│ ├── snapshot_load.c # snapshot restore application -- split out of snapshot.c (task 8.1)
│ ├── sim.c # simulation startup (Req 5 AC 13): replay header/snapshot load, virtual clock seed
│ ├── sim_loop.c # simulation event loop + REPL hook call sites (Req 5 AC 14/18)
│ ├── sim_record.c # --sim-output ReplayEvent/ReplaySession writes -- split out of sim_loop.c (task 11.5)
│ ├── sim_verify.c # --sim-verify decision comparator -- split out of sim_loop.c (task 11.4)
│ ├── handler.c # unchanged — capnp push/notify enqueue into pend_updates; scheduler_req_handler dispatches SchedAdminRequest
│ ├── plugin.c # + picker plugin registration category, + job_elect_fn compat
# wrapper (adapts old-style job_elect_fn-only plugins to
# xce_job_picker_iface_t, logs a deprecation warning at register time)
│ └── plugins/
│ ├── builtin_fcfs.c # the native xce_job_picker_iface_t default picker (Req 3) -- NEW file, see note below
│ └── fcfs.c # UNCHANGED -- still registers job_elect_fn only; kept as the live exercise target for plugin.c's compat wrapper (task 1.2). NOT rewritten as originally planned here; see BuiltinFcfsPicker component notes.
├── testsuite/
│ ├── test_picker_iface.c, test_picker_fcfs.c, test_picker_fcfs_session.c, test_picker_wiring.c,
│ │ test_picker_config_fields.c, test_picker_core_loop_boundary.c, test_picker_pendreason_factory_audit.c,
│ │ test_plugin_picker_registration.c # picker interface + builtin_fcfs + registration/compat-wrapper coverage
│ ├── test_core_events.c, test_core_session.c, test_core_full_cycle.c, test_core_isolation.c,
│ │ test_core_jbuf_capacity_pin.c # core contract + determinism-pin coverage
│ ├── test_clock.c # xce_core_now() two-base+speed formula
│ ├── test_driver_translate.c, test_driver_decision_cb.c, test_driver_session_ordering.c
│ ├── test_recording_path_validation.c, test_record_lifecycle.c, test_admin_request.c
│ ├── test_snapshot_save_load.c
│ ├── test_sim_startup.c, test_sim_event_loop.c, test_sim_isolation.c, test_sim_alt_events.c,
│ │ test_sim_verify.c, test_sim_record_output.c, test_sim_equivalence.c
│ └── test_repl_execution_control.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c,
│ test_repl_equivalence.c
src/controller/
├── cluster.c # + case OPCode_schedAdmin (Req 5 AC 10)
└── testsuite/
└── test_sched_admin_dispatch.c # standby rejection, disconnected-scheduler, timeout, success round-trip
# (name differs from this plan's original test_sched_admin.c; same coverage,
# registered in controller_test.c same as existing opcode tests)
src/schema/
└── controller.capnp # + SchedAdminRequest/Reply, ReplayHeader/.../ReplayDecision + payload structs (see Data Models);
# + WalCatchupEntry.payload/batchPayloads switched Data->List(UInt8) (bug-1505, follow-up
# outside this spec's boundary, same codec-crash class as SnapshotData.internalState/bug-1504)
src/xceadmin/src/xceadmin/
├── cli.py # + @cli.group() scheduler; nested @scheduler.group() record with
# start/stop/status commands, plus @scheduler.command() dump (thin layer only)
└── scheduler.py # record_start/stop/status(), dump() — mirrors push.py's pattern
src/xceadmin/tests/
├── test_scheduler.py # mock-TLS-server tests for record start/stop/status + dump parsing
└── test_cli_scheduler_mock.py # additional CLI-level mock-server coverage (task 10.6, beyond this plan's original file list)
```
### Modified Files
- `src/scheduler/src/scheduler.c` — Phase 1: hardcoded picker string removed; Phase 2: `sched_job_picker()` branches `job_elect_fn != NULL` (legacy plugins, delegates to their own self-contained loop) vs `NULL` (native `xce_job_picker_iface_t` pickers, generic per-session loop) — `scheduler.c` was not retired; it remains the picker-dispatch and allocator-loop home.
- `src/scheduler/src/plugins/fcfs.c` — **as-built: left unchanged**, not rewritten. `plugins/builtin_fcfs.c` (new file) implements `xce_job_picker_iface_t` and is the registered default (`schedobj.c`); `fcfs.c` keeps registering `job_elect_fn` only, deliberately, as the live exercise target for `plugin.c`'s legacy compat wrapper (task 1.2) — see BuiltinFcfsPicker component notes for the full rationale.
- `src/scheduler/src/main.c` — decision-drain `switch` replaced by `on_decision` callback registration (Req 5 AC 6); also gained the `--sim`/`--interactive`/`--record-dir`/`--sim-*` CLI surface (tasks 9.1, 11.1).
- `src/controller/cluster.c` — one new `case OPCode_schedAdmin:` block, structurally mirroring `OPCode_upgradeConfig` (primary-gate, delegate, explicit reply every path).
- `src/schema/controller.capnp` — additive only: new `OPCode_schedAdmin` enum value, new structs, two new union members on existing `Request.body`/`RequestReply.data` unions (ordinals `@31`/`@19` respectively — next available, verified against current schema in `research.md`).
- `src/xceadmin/src/xceadmin/cli.py` — one new `@cli.group() scheduler`, containing a nested `@scheduler.group() record` (with `start`/`stop`/`status` commands) plus a sibling `@scheduler.command() dump` — matching the literal CLI syntax `xceadmin scheduler record start`/`xceadmin scheduler dump`, not 4 flat commands.
## System Flows
### Normal-Mode Scheduling Round (driver ↔ core)
```mermaid
sequenceDiagram
participant Ctrl as Controller
participant Drv as Normal Driver
participant Core as Scheduler Core
participant Pick as Picker Thread
participant Alloc as Allocator Thread
Ctrl->>Drv: capnp push or notify
Drv->>Drv: buffer into pend_updates
Drv->>Core: core_push_events
Core->>Pick: state change hooks, synchronous, picker idle
Drv->>Core: core_go, non blocking
Core->>Pick: signal via wbuf
Pick->>Pick: begin_session then next_job loop
Pick->>Alloc: send jobid via jbuf per job
Alloc->>Alloc: try_allocate then can_preempt
Alloc->>Drv: send decision via decisions cbuf
Drv->>Ctrl: on_decision publishes dispatch, pend reason, or preempt
Alloc->>Drv: end of session marker via decisions cbuf
Drv->>Core: core_end_session
```
Key decision not obvious from the diagram: decisions publish to the controller **immediately per-decision**, not batched at session end (Req 5, "Decision publishing") — the drain loop in the driver thread interleaves `cbuf_get`/`on_decision` one at a time.
### `OPCode_schedAdmin` Recording Control (admin path)
```mermaid
sequenceDiagram
participant CLI as XceAdmin SchedulerModule
participant Ctrl as Controller ClusterDispatch
participant Sched as Scheduler NormalDriver
CLI->>Ctrl: schedAdmin request, action plus recordFile
Ctrl->>Ctrl: check ha_is_primary, reject standby
Ctrl->>Sched: forward via scheduler endpoint, five second timeout
Sched->>Sched: validate recordFile, reject or resolve under record_dir
Sched->>Sched: save_snapshot then write header and snapshot entry
Sched-->>Ctrl: schedAdmin reply, ok plus recording state and counters
Ctrl-->>CLI: relay reply to caller
```
Key decision not obvious from the diagram: the controller is a pure pass-through — it does not interpret `SchedAdminRequest` fields beyond the opcode dispatch, and owns no recording state (Boundary Commitments, "This Spec Owns" — recording logic lives entirely in the scheduler driver).
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|---|---|---|---|---|
| 1.1 | Core loop calls `next_job()` only, no ordering logic | SchedulerCore, JobPicker | `xce_job_picker_iface_t.next_job` | Normal-Mode Round |
| 1.2 | Picker hook categories (lifecycle/state/session/post-alloc/optional) | JobPicker | `xce_job_picker_iface_t` | — |
| 1.3 | Three-thread cbuf model | SchedulerCore | `wbuf`/`jbuf`/`decisions` cbufs | Normal-Mode Round |
| 1.4 | Picker/allocator/main-loop pseudocode contract | SchedulerCore, JobPicker | `core_push_events`/`core_go` | Normal-Mode Round |
| 1.5 | Cancel-safe, session-aware picker | JobPicker | `begin_session`/`end_session`/`next_job` | — |
| 1.6 | Replace hardcoded picker string | SchedulerCore | `plugin_register`/`key_find` | — |
| 1.7 | Preserve thread architecture, decision publish via callback | SchedulerCore | `on_decision` cb | Normal-Mode Round |
| 2.1 | `job_pick_plugin` on `xce_scheduler_t`, default `builtin_fcfs` | SchedulerCore | — | — |
| 2.2 | `job_pick_plugin` per-queue override | JobPicker | `xce_sched_queue_t.job_pick_plugin` | — |
| 2.3 | `project_name` on queue | SchedulerCore | `xce_sched_queue_t.project_name` | — |
| 2.4 | Config-driven picker name, no rebuild | SchedulerCore | scheduler config | — |
| 2.5 | Unregistered plugin fails loud | SchedulerCore | `plugin_register`/`key_find` | — |
| 2.6 | Picker contract documented (cancel-safe, session-aware) | JobPicker | `xce_job_picker_iface_t` | — |
| 3.1 | `builtin_fcfs` auto-registered as default | BuiltinFcfsPicker | plugin registration | — |
| 3.2 | Hook-by-hook FCFS implementation | BuiltinFcfsPicker | `xce_job_picker_iface_t` | — |
| 3.3 | Private `ordq_t` state, no core queries in `next_job` | BuiltinFcfsPicker | — | — |
| 3.4 | Ordering equivalence with today's `fcfs_elect_job` | BuiltinFcfsPicker | — | — |
| 3.5 | Dedicated picker thread retained, cancel points | BuiltinFcfsPicker, SchedulerCore | `next_job`/`cbuf_get(wbuf)` | Normal-Mode Round |
| 3.6 | `end_session` no-op, `pendJobs` carries over | BuiltinFcfsPicker | — | — |
| 4.1 | Job-picking order regression check | BuiltinFcfsPicker | — | test_picker_fcfs.c |
| 4.2 | Existing scheduler unit tests pass unmodified | SchedulerCore | — | testsuite/ |
| 4.3 | HA soak — no job loss/divergence | SchedulerCore | — | `/xce-test-ha` |
| 5.1 | Mode-agnostic core, no mode flag | SchedulerCore | `xce_sched_core_init()` | — |
| 5.2 | Passive event-driven interface, zero I/O deps | SchedulerCore | `core_push_events`/`on_decision` | — |
| 5.3 | Virtualized clock, two-base+speed formula | SchedulerCore | `xce_core_now()` | — |
| 5.4 | Core event types (typed payloads) | SchedulerCore | `xce_core_event_t` | — |
| 5.5 | Decision output types (DISPATCH/PEND/PREEMPT) | SchedulerCore | `xce_decision_t` | — |
| 5.6 | Decision output callback | SchedulerCore, NormalDriver | `xce_core_decision_cb_t` | Normal-Mode Round |
| 5.7 | Deterministic guarantee + normal↔sim equivalence | SchedulerCore | `jbuf` cap=1, hash-iteration audit, PRNG seed | test_sim_equivalence.c |
| 5.8 | Driver pending-event queue | NormalDriver | `pend_updates` dataq | Normal-Mode Round |
| 5.9 | Controller integration (capnp translate, publish decisions) | NormalDriver | `xcecli_ctx_t` | Normal-Mode Round |
| 5.10 | Runtime recording control (`OPCode_schedAdmin`) | NormalDriver, SchedAdminOpcode, XceAdminSchedulerCLI | `SchedAdminRequest`/`Reply` | Admin Path |
| 5.11 | Periodic snapshot during recording | NormalDriver | `record_snapshot_interval` | — |
| 5.12 | Session response time metrics | SchedulerCore | `SessionSummary` | — |
| 5.13 | Simulation mode startup (load header+snapshot) | SimDriver | `.replay` load sequence | — |
| 5.14 | Discrete event-driven virtual clock | SimDriver | `--sim-speed` | — |
| 5.15 | `--sim-verify` deterministic replay verification | SimDriver | `decision_eq` | — |
| 5.16 | `--sim-alt-events` what-if branching | SimDriver | alt-events file | — |
| 5.17 | Simulation isolation (no controller deps) | SimDriver | `xce_sched_sim_driver_t` | — |
| 5.18 | `ReplaySession` as session boundary | SimDriver | event loop | — |
| 5.19 | `--sim-output` simulation record file | SimDriver | `.replay` writer | — |
| 5.20 | `.replay` capnp stream format | ReplayFormat | schema structs | — |
| 5.21 | Snapshot serialization / picker rebuild | SchedulerCore, ReplayFormat | `save_snapshot`/`load_snapshot` | — |
| 5.22 | Interactive REPL | SimDriver | command loop | — |
| 6.1 | Resreq immutability | SchedulerCore | — | — |
| 6.2 | Pend reason factory (`make_pendreason()`) | SchedulerCore | `lib/preason.c` | — |
| 6.3 | Pointer-identity fast path | SchedulerCore | — | — |
| 6.4 | Select expression memoization preserved | SchedulerCore | `select_eval_ctx_t` | — |
| 6.5 | Snapshot forward-compatible with object sharing | ReplayFormat | `SnapshotData` | — |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|---------------------------|-----------|
| JobPicker (interface) | Scheduler Core | Defines the pluggable job-selection contract | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 | plugin.h (P0) | Service |
| BuiltinFcfsPicker | Scheduler Core | Default picker replicating today's FCFS order | 3.1-3.6, 4.1 | JobPicker (P0) | Service |
| SchedulerCore | Scheduler Core | Passive, mode-agnostic scheduling engine | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 | JobPicker (P0), allocation/preemption (P1, sibling specs) | Service, State |
| NormalDriver | Scheduler Driver | Controller-attached driver, recording control | 5.6, 5.8-5.11 | SchedulerCore (P0), `xcecli_ctx_t` (P0) | Service, Event |
| SimDriver | Scheduler Driver | Replay-attached driver, virtual clock, REPL | 5.13-5.19, 5.22 | SchedulerCore (P0), ReplayFormat (P0) | Service, State |
| ReplayFormat | Wire Protocol | `.replay` capnp schema, `SnapshotData` | 5.20, 5.21, 6.5 | `controller.capnp` (P0) | Batch |
| SchedAdminOpcode | Controller | Pass-through dispatch for `OPCode_schedAdmin` | 5.10 | `cluster.c` `myreq_handler` (P0) | API |
| XceAdminSchedulerCLI | Tooling | `scheduler record`/`scheduler dump` commands | 5.10 | `xcecommon.wire`/`discovery` (P0) | API |
### Scheduler Core
#### JobPicker
| Field | Detail |
|-------|--------|
| Intent | Contract every job-selection strategy implements; hides all ordering logic from the core loop |
| Requirements | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 |
**Responsibilities & Constraints**
- Owns iteration order over pending jobs; the core never inspects queue/job internals for ordering.
- Cancel-safe (`pthread_cancel` points in `next_job()`) and session-scoped (`begin_session` resets iteration state).
- State-change hooks run synchronously in the driver thread during `core_push_events()`; session hooks run in the picker thread; post-allocation hooks run in the allocator thread (Req 1 AC 3).
**Dependencies**
- Inbound: SchedulerCore — invokes every hook (P0)
- Outbound: none — picker implementations must not call back into core internals outside the hook parameters they're given
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
```c
typedef struct {
int (*init)(void* picker, xce_sched_core_t* core);
void (*fini)(void* picker);
int (*begin_session)(void* picker);
xce_sched_job_t* (*next_job)(void* picker);
int (*end_session)(void* picker);
int (*on_dispatch)(void* picker, uint64_t jobid);
int (*on_pend)(void* picker, uint64_t jobid);
int (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
int (*on_job_push)(void* picker, xce_sched_job_t* job);
int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
int (*on_job_update)(void* picker, xce_sched_job_t* job);
int (*on_config)(void* picker, xce_conf4sched_t* config); /* MAY be NULL */
int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); /* MAY be NULL */
} xce_job_picker_iface_t;
```
- Preconditions: `init()` called exactly once before any other hook; `begin_session()` precedes every `next_job()` sequence.
- Postconditions: `next_job()` returns NULL exactly when no ready job remains for the current session.
- Invariants: hook implementations never block indefinitely (picker thread holds no lock the allocator needs); `next_job()` never allocates a job the same session has already returned.
**Implementation Notes**
- Integration: registered via `plugin_register`/`key_find` under the picker plugin category (new `plugin.c` registration table entry).
- Validation: `test_picker_fcfs.c` feeds hooks directly and asserts `next_job()` order — no live core needed (Req 3 AC 3's testability rationale).
- Risks: a picker implementation that queries core state directly during `next_job()` (bypassing the hook-fed private state pattern) would still compile but silently reintroduce lock contention — not enforceable at the interface level, called out for code review.
#### BuiltinFcfsPicker
| Field | Detail |
|-------|--------|
| Intent | Default `xce_job_picker_iface_t` implementation; replicates today's `fcfs_elect_job` ordering exactly through the new interface |
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1 |
**Responsibilities & Constraints**
- Maintains private state only — `ordq_t queues` (ordered by `queue->priority desc`) and, per queue, an `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). Never queries `sched->queues`/`sched->jobs` during `next_job()` (Req 3 AC 3).
- Every hook's behavior is fully specified in requirements.md Req 3 AC 2's hook-by-hook table — not repeated here; this block covers integration and testability, not the algorithm.
- Auto-registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` at scheduler startup (Req 3 AC 1).
- `end_session()` is a no-op; undispatched jobs carry over in `pendJobs` to the next round unchanged (Req 3 AC 6).
**Dependencies**
- Inbound: SchedulerCore — invokes every `xce_job_picker_iface_t` hook, same as any picker (P0)
- Outbound: none — implements the interface, depends on nothing beyond it
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
Implements `xce_job_picker_iface_t` (struct shown under JobPicker, above) with the `fcfs_plugin_data_t` private state described in Responsibilities. No additional public surface beyond the interface itself.
- Preconditions/Postconditions/Invariants: inherits JobPicker's contract in full; adds no relaxations or extensions.
**Implementation Notes**
- Integration, as-built: implemented in a **new** file, `plugins/builtin_fcfs.c`, registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` (`schedobj.c`) — NOT a rewrite of `plugins/fcfs.c` in place, as originally planned above. `fcfs.c` was deliberately left untouched (still `job_elect_fn`-only, unregistered as default) because task 1.2's `job_elect_fn` compat wrapper needs a real legacy-style plugin to exercise; `fcfs.c` fills that role, and `scheduler.c`'s `sched_job_picker()` dispatch (`job_elect_fn != NULL` vs `NULL`) depends on both variants existing side by side. Functionally equivalent to the original plan (one default picker implementing the new interface, ordering-equivalent to legacy `fcfs_elect_job`); only the file-layout mechanics differ.
- Validation: `test_picker_fcfs.c` is this component's primary test — feeds `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly and asserts `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order, matching today's `fcfs_elect_job` (Req 4.1's regression check).
- Risks: ordering-equivalence regression is the main risk (any drift from today's `fcfs_elect_job` output is a behavior change operators didn't opt into) — mitigated by `test_picker_fcfs.c` asserting against the documented ordering rule directly, plus the HA soak in Req 4.3.
#### SchedulerCore
| Field | Detail |
|-------|--------|
| Intent | Owns all scheduling state; drives the picker/allocator threads; exposes the driver↔core protocol |
| Requirements | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 |
**Responsibilities & Constraints**
- Zero direct dependency on controller communication, capnp, network I/O, filesystem, or wall-clock time (Req 5 AC 2) — enforced by the "Allowed Dependencies" link list, not just convention.
- `jbuf` cbuf capacity is hardcoded to 1 inside `xce_sched_core_init()` — never a parameter (Req 5 AC 7, pinned against the existing `njobs` FIXME in `scheduler.c:384`).
- All internal time via `xce_core_now()`; no direct `time()`/`clock_gettime()`/`gettimeofday()` calls anywhere in `core.c`.
**Dependencies**
- Inbound: NormalDriver, SimDriver — both call `core_push_events`/`core_go`, register `on_decision` (P0)
- Outbound: JobPicker (P0); `try_allocate()`/`can_preempt()` from sibling specs (P1 — signature-stable dependency, no behavior assumption)
- External: none
**Contracts**: Service [x] / State [x]
##### Service Interface
```c
int xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed);
int xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents);
int xce_core_go(xce_sched_core_t*);
int xce_core_end_session(xce_sched_core_t*);
uint64_t xce_core_now(xce_sched_core_t*);
int xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out);
int xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in);
void xce_sched_core_fini(xce_sched_core_t*);
```
- Preconditions: `core_push_events()` only valid when no session is active (`session_active == 0`); `core_go()` returns `-EINVAL` if a session is already active. `xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()` are likewise only valid when `session_active == 0` — both return `-EBUSY` if called mid-session, rather than silently capturing or restoring torn state. Every caller in this design (record start, periodic checkpoint, simulation startup) already invokes them between sessions by construction; the `-EBUSY` return is a defensive contract check, not a path any correct caller should hit.
- Postconditions: after `on_decision` delivers the end-of-session `NULL` marker, `session_active` is guaranteed to transition to 0 on the subsequent `core_end_session()` call.
- Invariants: given identical `(config, event sequence)`, `core_push_events`+`core_go` produce an identical decision sequence (Req 5 AC 7) — the whole point of the design.
##### State Management
- State model: scheduling state (jobs/queues/workers/resources/resvals/users/projects) held in `kvmgr`s inside `xce_sched_core_t`, mirroring today's `xce_scheduler_t` layout minus `apictx`/`cluster_cfg`.
- Persistence & consistency: no persistence inside the core itself — `save_snapshot()`/`load_snapshot()` are the only serialization boundary, and only the driver decides when to invoke them.
- Concurrency strategy: three-thread cbuf pipeline unchanged from today; clock atoms (`virtual_clock_base`, `real_clock_base`, `speed_num`, `speed_den`) are lock-free `_Atomic uint64_t`, updated by the driver as a group before each event dispatch.
**Implementation Notes**
- Integration: `core_push_events`/`core_go`/`on_decision` are the only surface either driver touches — no driver reaches into `xce_sched_core_t` internals.
- Validation: `test_core_events.c` (contract-level), `test_sim_equivalence.c` (record→replay→diff, Req 5 AC 7.1 — the spec's central regression test).
- Risks: the `kvmgr` iteration-order audit (Req 5 AC 7, "Stabilized hash table iteration") is a Phase 2 deliverable with real but bounded scope — jobs/queues/users/projects only (per `research.md` and requirements.md's explicit Phase 2 task).
### Scheduler Driver
#### NormalDriver
| Field | Detail |
|-------|--------|
| Intent | Connects the core to a live controller; owns recording control |
| Requirements | 5.6, 5.8, 5.9, 5.10, 5.11 |
**Responsibilities & Constraints**
- Translates capnp push/notify into `xce_core_event_t`, buffers between sessions in `pend_updates` (existing dataq swap mechanism).
- Publishes each decision immediately via `xcecli_jobdispatch`/`xcecli_set_pend_reason`/`xcecli_preempt` as it drains from `decisions` cbuf — not batched at session end.
- Recording is boolean-gated: zero file I/O, zero extra allocation when `recording == false` (Req 5 AC 10, Performance).
- Validates `recordFile` before ever calling `fopen()`: reject `/` or `..`, resolve under `record_dir`, confirm containment via `realpath()` (Req 5 AC 10, "Record file path validation" — this design inherits, not re-derives, that requirement).
- `record_dir` is provisioned exactly one way: a `--record-dir DIR` scheduler startup CLI flag (default `/var/log/xce/`), parsed alongside the existing normal-mode arguments in `main.c` — never a capnp field on any request path (`SchedAdminRequest` included), so nothing reachable over the wire can redirect where recordings are written. Matches the existing `--sim-*` flag family's provisioning style (CLI-only, not config-pushed).
**Dependencies**
- Inbound: `main.c` (process entry point) (P0)
- Outbound: SchedulerCore (P0), `xcecli_ctx_t` (P0)
- External: `SchedAdminOpcode` dispatch (P1 — receives forwarded requests, not a caller relationship)
**Contracts**: Service [x] / Event [x]
##### Service Interface
```c
typedef struct {
xce_sched_core_t core;
xcecli_ctx_t apictx;
xce_cluster_config_t cluster_cfg;
dataq_t* pend_updates[2];
char* record_dir; /* from --record-dir CLI flag at startup; default "/var/log/xce/";
never settable via any request-path field */
char* record_file_path;
FILE* record_file;
bool recording;
uint64_t record_event_count;
uint64_t record_snapshot_interval;
uint64_t record_start_ts;
} xce_sched_driver_t;
int sched_driver_run(xce_sched_driver_t*, xce_conf4sched_t*, xce_cluster_config_t*, void* handler);
int sched_driver_record_start(xce_sched_driver_t*, const char* record_file /* bare filename or NULL */);
int sched_driver_record_stop(xce_sched_driver_t*);
int sched_driver_record_status(xce_sched_driver_t*, SchedAdminReply* out);
```
- Preconditions: `record_start` fails with `-EALREADY`-equivalent if already recording; `record_stop` fails if not recording.
- Postconditions: on `record_start` success, `ReplayHeader` + initial `SnapshotEntry` are durably on disk before the reply is sent.
- Invariants: `recording == true` iff `record_file != NULL`.
##### Event Contract
- Subscribed events: capnp push/notify from the controller (existing handler.c path, unchanged).
- Published events: decisions to the controller via `xcecli_ctx_t`; `SchedAdminReply` back through the forwarding path on recording-control requests.
- Ordering / delivery guarantees: decisions publish in allocator output order; recording writes events/decisions in the same order they cross the driver thread.
**Implementation Notes**
- Integration: `main.c`'s mode-selection branch calls `sched_driver_run()` when `--sim` is absent.
- Validation: recording lifecycle tested against a real filesystem (temp dir) — start/stop/status round trip, path-validation rejection cases, disk-full simulated via a read-only mount point.
- Risks: `OPCode_schedAdmin` ships with the same primary/standby gate as `OPCode_upgradeConfig` and no additional RBAC/auth check — a deliberate scope decision (project owner, 2026-08-02), not an oversight. Strengthening authentication for this opcode (and `config push`, if it also lacks one) is deferred to a follow-up spec. See Open Questions/Risks.
#### SimDriver
| Field | Detail |
|-------|--------|
| Intent | Connects the core to a `.replay` file; owns virtual clock, verification, and the REPL |
| Requirements | 5.13-5.19, 5.22 |
**Responsibilities & Constraints**
- Contains no `xcecli_ctx_t`, `xce_cluster_config_t`, capnp socket, or controller RPC of any kind (Req 5 AC 17) — enforced by the struct definition itself having no such fields.
- Discrete virtual clock: jumps directly to the next event's timestamp, never a fixed tick (Req 5 AC 14).
- REPL is a single `fgets` loop in the same thread as the simulation loop — no REPL thread (Req 5 AC 22, Implementation note).
**Dependencies**
- Inbound: `main.c` (P0, when `--sim` is present)
- Outbound: SchedulerCore (P0), ReplayFormat (P0)
- External: none
**Contracts**: Service [x] / State [x]
##### Service Interface
```c
typedef struct {
xce_sched_core_t core;
FILE *replay_file, *alt_events_file, *record_file;
uint64_t sim_speed_num, sim_speed_den;
bool verify;
bool interactive;
} xce_sched_sim_driver_t;
int sched_sim_driver_run(xce_sched_sim_driver_t*, /* CLI arguments struct */ const void* args);
```
- Preconditions: replay file's second message must be a `SnapshotEntry` (rejected otherwise, Req 5 AC 20).
- Postconditions: with `--sim-output`, every session cycle appends exactly one `ReplaySession` to the output file before the next `ReplayEvent` batch begins.
- Invariants: REPL `step`/`continue` to completion produces byte-identical output to batch `--sim` (Req 5 AC 22, "Deterministic replay is preserved").
##### State Management
- State model: same `xce_sched_core_t` as normal mode, loaded via `load_snapshot()` instead of live events.
- Persistence & consistency: `--sim-output` writes a new independent `.replay` file; never mutates the input file.
- Concurrency strategy: single-threaded driver loop around the same core thread pair; REPL blocks the driver loop intentionally (no wall-clock deadline in simulation).
**Implementation Notes**
- Integration: `main.c`'s mode-selection branch calls `sched_sim_driver_run()` when `--sim` is present.
- Validation: `test_sim_equivalence.c` drives this component directly (record via NormalDriver's test harness, replay via this one, diff).
- Risks: none beyond what's already captured under SchedulerCore (the determinism guarantee is a core property; SimDriver only needs to not violate it, e.g. by never bypassing `core_push_events`/`core_go`).
### Wire Protocol
#### ReplayFormat
| Field | Detail |
|-------|--------|
| Intent | `.replay` capnp stream schema and `SnapshotData` |
| Requirements | 5.20, 5.21, 6.5 |
**Responsibilities & Constraints**
- Additive-only changes to `controller.capnp` — reuses existing structs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) wherever they already exist (verified in `research.md`); only defines new structs where none exist.
- `SnapshotData` field shapes track the *actual current* C struct field sets (e.g. `xce_sched_user_t` is `{name}` only today, `xce_sched_project_t` is `{name}` only today) — the schema does not invent fields the C structs don't have yet.
**Dependencies**
- Inbound: SchedulerCore (`save_snapshot`/`load_snapshot`), NormalDriver (recording writer), SimDriver (reader/writer) (all P0)
- Outbound: none
- External: none
**Contracts**: Batch [x]
##### Batch / Job Contract
- Trigger: `record start` (normal mode) or `--sim-output` (simulation mode) write a `.replay` stream incrementally; `--sim`/`xceadmin scheduler dump` read one.
- Input / validation: first message must be `ReplayHeader`; second must be `SnapshotEntry` (Req 5 AC 20) — both enforced at read time, not just documented.
- Output / destination: append-only file; `record stop` / clean simulation completion writes `ReplayEvent{endSession}`.
- Idempotency & recovery: a write failure mid-message truncates cleanly to the last complete message (Req 5 AC 10, error handling) — no partial-message corruption possible because each message is a full capnp frame written atomically at the `fwrite` level.
**New capnp structs** (additive to `controller.capnp`; `ReplayHeader`/`ReplayEvent`/`ReplaySession`/`ReplayDecision`/`SessionSummary` wrapper structs already fully specified in requirements.md Req 5 AC 20 — not repeated here):
```capnp
# New event payloads (JOBS_PUSH/WORKERS_PUSH/etc reuse existing Conf4Sched,
# SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages —
# verified present in controller.capnp, research.md).
struct JobStateChangeEvent $C.mapname("job_state_change_event_t") {
jobid @0: UInt64;
oldState @1: UInt32;
newState @2: UInt32;
exitCode @3: Int32;
}
struct JobUpdateEvent $C.mapname("job_update_event_t") {
jobid @0: UInt64;
changedFields @1: UInt32; # bitmask: JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ
job @2: Job; # full updated record; reuses existing Job struct
}
# New decision payloads for ReplayDecision.body (Req 5 AC 5).
# DISPATCH/PEND reuse existing JobAlloc/JobPendReason; PREEMPT is new
# (wire shape owned here per Boundary Commitments; victim-selection
# semantics owned by scheduler-preemption — see research.md).
#
# AC 5's field list for DISPATCH is "jobid, workerid, allocations[]"
# (singular workerid). JobAlloc already generalizes this correctly for
# multi-host jobs: JobAlloc.jobid covers jobid, JobAlloc.allocs is
# List(HostAlloc) with each HostAlloc.host as that allocation's worker
# id and HostAlloc.allocs: List(ResAlloc) as its resource/amount pairs
# -- verified against controller.capnp, no new fields needed. jobid
# lives ONLY in JobAlloc.jobid (not duplicated at DispatchDecision
# level) to avoid two unsynchronized sources of truth for the same job.
struct DispatchDecision $C.mapname("sched_dispatch_decision_t") {
alloc @0: JobAlloc;
}
struct PendDecision $C.mapname("sched_pend_decision_t") {
jobid @0: UInt64;
reason @1: JobPendReason;
}
struct PreemptVictim $C.mapname("preempt_victim_t") {
jobid @0: UInt64;
workerid @1: Text;
}
struct PreemptDecision $C.mapname("sched_preempt_decision_t") {
jobid @0: UInt64;
victims @1: List(PreemptVictim);
reason @2: JobPendReason;
}
# SnapshotData — full core-state checklist from requirements.md Req 5 AC 21,
# given a concrete wire shape. Reuses existing List(...) payload structs
# for domain objects already carried elsewhere on the wire; the two
# scheduler-internal-only caches (shared-resource tracking, unsatisfiable-
# resreq cache) and session/PRNG state have no existing external contract,
# so they are versioned opaque blobs the core alone reads/writes -- no
# consumer outside save_snapshot()/load_snapshot() ever inspects their
# internal shape (WHAT not HOW: the boundary that matters is "round-trips
# through the core, opaque to everyone else").
struct SnapshotData $C.mapname("sched_snapshot_data_t") {
config @0: Conf4Sched;
jobs @1: SchedJobData;
queues @2: List(QueueConf);
workers @3: WorkerData;
resources @4: ResourceData;
resvals @5: ResValList;
users @6: List(SchedUser);
projects @7: List(SchedProject);
prngSeed @8: UInt64;
internalState @9: Data; # shared-resource tracking + unsatisfiable-resreq
# cache + session/iterator state, core-opaque
internalVersion @10: UInt32; # format tag for internalState, so a future
# incompatible internal layout fails loud
# instead of misinterpreting old bytes
}
struct SchedUser $C.mapname("xce_sched_user_t") {
name @0: Text; # matches xce_sched_user_t today (name only) --
# extend in lockstep if xce_sched_user_t gains fields
}
struct SchedProject $C.mapname("xce_sched_project_t") {
name @0: Text;
priority @1: Int32;
}
```
**Implementation Notes**
- Integration: `save_snapshot()`/`load_snapshot()` are the only functions that construct/consume `SnapshotData`; no other component touches `internalState`'s bytes.
- Validation: as-built, round-trip coverage for every new struct lives in `src/controller/testsuite/test_e2e_capnp.c` (task 7.5), not a dedicated `test_replay_format.c` — this repo's existing convention is one shared capnp round-trip test file per schema, and this spec's new structs were added to it rather than starting a second one; `internalVersion` mismatch is a dedicated negative test within it.
- Risks: `internalState`'s opaque-blob design trades wire-level inspectability for implementation flexibility — acceptable because AC 21 only requires the *core* to round-trip correctly, never that `xceadmin scheduler dump` render its internals meaningfully (dump can print `internalState`'s byte length and `internalVersion` only).
### Controller
#### SchedAdminOpcode
| Field | Detail |
|-------|--------|
| Intent | Pass-through dispatch for `OPCode_schedAdmin` in the controller's request handler |
| Requirements | 5.10 |
**Responsibilities & Constraints**
- Structurally mirrors the existing `OPCode_upgradeConfig` case (`cluster.c:670-705`) for the primary-gate/delegate/explicit-reply shape: primary-only gate via `ha_is_primary()`, explicit reply on every path, 5s timeout.
- Owns zero recording state — every field of `SchedAdminRequest`/`Reply` passes through untouched.
- **Blocking-wait scope decision (accepted, not hardened)**: unlike `OPCode_upgradeConfig` (in-process commit-barrier wait, already `COMMIT_WAIT_RESERVED`-admitted), this case blocks its `vtask` handler thread for up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock`. Reviewed against `ha-async-io.md`'s vtaskmgr admission rule and accepted as a boundary case (project owner, 2026-08-02): `xceadmin scheduler` commands are cluster-administrator-triggered, low-frequency, human-paced — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism (`COMMIT_WAIT_RESERVED`-style) is added for this opcode. Revisit if usage patterns change (e.g. automated/scripted recording control at scale).
**Dependencies**
- Inbound: XceAdminSchedulerCLI, via TLS+capnp (P0)
- Outbound: NormalDriver (P0, forwards the request)
- External: none
**Contracts**: API [x]
##### API Contract
| Method | Endpoint (opcode) | Request | Response | Errors |
|--------|--------------------|---------|----------|--------|
| Request/Reply | `OPCode_schedAdmin` | `SchedAdminRequest` | `SchedAdminReply` | `-EAGAIN` (not primary), `"scheduler not connected"`, `"request timeout"`, `"invalid record file name"`, `"already recording"`, `"not recording"` (full table: requirements.md Req 5 AC 10) |
**Implementation Notes**
- Integration: one new `case` in `myreq_handler`'s switch (`cluster.c`); no new file.
- Validation: `test_sched_admin_dispatch.c` (`src/controller/testsuite/`; named differently from this plan's original `test_sched_admin.c`, same coverage) asserting the reply-on-every-path property (standby rejection, disconnected-scheduler rejection, timeout) — mirrors existing `OPCode_upgradeConfig` test coverage style.
- Risks: no RBAC/auth check is visible for the comparable `OPCode_upgradeConfig` case in `cluster.c` (see `research.md`), and `OPCode_schedAdmin` inherits that same gap by deliberate decision — auth hardening for this opcode is deferred to a follow-up spec, not silently accepted as permanent.
### Tooling
#### XceAdminSchedulerCLI
| Field | Detail |
|-------|--------|
| Intent | `xceadmin scheduler record start/stop/status` and `xceadmin scheduler dump <FILE>` |
| Requirements | 5.10 |
**Responsibilities & Constraints**
- `cli.py` stays a thin `click` layer; all logic lives in the new `scheduler.py` module, mirroring `push.py`'s split.
- `record_start/stop/status()` go through `xceadmin.discovery.follow_redirects` + `decode_reply` (request/reply opcode, needs redirect-following, unlike `probe_config()`).
- `dump()` has no network dependency — decodes a local `.replay` file only.
**Dependencies**
- Inbound: operator CLI invocation (P0)
- Outbound: `xcecommon.wire`/`schema`, `xceadmin.discovery` (P0); SchedAdminOpcode via the wire (P0, for record/status only)
- External: none
**Contracts**: API [x]
##### API Contract
| Method | Endpoint | Request | Response | Errors |
|--------|----------|---------|----------|--------|
| CLI | `scheduler record start [FILE]` | `SchedAdminRequest{recordStart}` | `SchedAdminReply` | `XceAdminError` subclasses (connection, redirect, decode) |
| CLI | `scheduler record stop` | `SchedAdminRequest{recordStop}` | `SchedAdminReply` | same |
| CLI | `scheduler record status` | `SchedAdminRequest{recordStatus}` | `SchedAdminReply` | same |
| CLI | `scheduler dump <FILE>` | (local file) | formatted text to stdout | `SchemaNotFoundError`, file-not-found |
**Implementation Notes**
- Integration: `cli.py` imports `record_start`/`record_stop`/`record_status`/`dump` from `scheduler.py`, same import shape as today's `build_cluster_config`/`push_config`.
- Validation: `test_scheduler.py` — mock-TLS-server tests for the three record commands (mirrors `test_config_push_mock.py`), plus golden-file tests for `dump()` against a small fixture `.replay` file.
- Risks: none beyond ReplayFormat's (this component only formats/transmits, never interprets scheduling semantics).
## Data Models
Covered in full under **Components and Interfaces → ReplayFormat** above (new capnp structs) — not duplicated here per the dedup guidance in design-principles.md. `SnapshotData`'s reuse of existing `Conf4Sched`/`SchedJobData`/`WorkerData`/`ResourceData`/`ResValList`/`QueueConf` is the only cross-cutting data-model decision; see `research.md`, "Reusable capnp payload types," for the verification trail.
## Error Handling
### Error Strategy
Fail-loud, no silent fallback — consistent with the controller's existing "fail loud, stay alive" design invariant (`src/controller/CLAUDE.md`), applied to the scheduler process at the scope this feature touches.
### Error Categories and Responses
- **Configuration errors** (unregistered picker plugin, Req 2 AC 5): scheduler startup fails with a clear error — no silent fallback to `builtin_fcfs`.
- **Recording errors** (disk full, permission error, invalid `recordFile`): logged, recording state cleanly torn down (`recording=false`, file closed, `record_file_path` freed), scheduling continues unaffected (Req 5 AC 10, error handling) — recording failure is never fatal to the scheduler process.
- **Simulation errors** (missing/corrupt replay file, missing initial `SnapshotEntry`): simulation driver fails at startup with a clear message — no partial/best-effort simulation.
- **Admin-path errors** (`OPCode_schedAdmin`): every failure mode gets an explicit `{ok=false, error=...}` reply — never a dropped request (mirrors the "no silent drop" precedent already established for replication-class messages, `src/controller/CLAUDE.md`'s protocol checklist).
### Monitoring
No new metrics infrastructure — `SessionSummary` (Req 5 AC 12) provides session-duration diagnostics through the existing `.replay` recording path; no separate telemetry system is introduced.
## Testing Strategy
- **Unit — Picker contract** (`test_picker_fcfs.c`): feed `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly, assert `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order — verifies Req 3.2-3.4 without a live core.
- **Unit — Core event contract** (`test_core_events.c`): `core_push_events()` followed by `core_go()` with a stub picker asserts state-change hooks fire exactly once per event, synchronously, before `core_go()` returns — verifies Req 5.1-5.8.
- **Unit — Clock virtualization** (`test_clock.c`): `xce_core_now()` against the four mode rows in Req 5 AC 3's table (normal, discrete, real-time, fast/slow) — asserts the two-base+speed formula directly, no wall-clock flakiness (fixed `real_clock_base`/mocked `clock_gettime` delta).
- **Unit — Replay format round-trip** (as-built: `src/controller/testsuite/test_e2e_capnp.c`, not a dedicated `test_replay_format.c` — added to this repo's existing shared capnp round-trip test file instead): encode/decode every new capnp struct (`JobStateChangeEvent`, `JobUpdateEvent`, `DispatchDecision`, `PendDecision`, `PreemptDecision`, `SnapshotData`) — field-for-field equality after round-trip.
- **Integration — Deterministic replay** (`test_sim_equivalence.c`, Req 5.7/5.7.1): record a synthetic scheduling session through `NormalDriver`'s test harness → replay through `SimDriver` → assert `ReplayDecision` sequences match exactly. This is the design's highest-value test — it directly exercises the central invariant.
- **Integration — Recording lifecycle** (driver-level, Req 5.10-5.11): `record start` → submit synthetic events → `record stop` → verify file structure (`ReplayHeader`, `SnapshotEntry`, alternating `ReplayEvent`/`ReplaySession`, periodic checkpoints at the configured interval, final `endSession`).
- **Integration — `OPCode_schedAdmin` dispatch** (`test_sched_admin_dispatch.c`, controller-level; named differently from this plan's original `test_sched_admin.c`): standby rejection, disconnected-scheduler rejection, timeout, and successful forward/reply round trip — mirrors existing `OPCode_upgradeConfig` test coverage.
- **Integration — xceadmin scheduler CLI** (`test_scheduler.py`): mock-TLS-server tests for `record start/stop/status` (redirect-following, reply decoding) plus golden-file tests for `dump()` against a fixture `.replay` file.
- **E2E — HA soak regression** (Req 4.3, Req 5 per-phase verification): `/xce-test-ha` sequential and chaotic rounds after each phase, asserting zero new job loss/divergence attributable to this feature — required before each phase merges, not a one-time gate at the end.
- **E2E — REPL equivalence** (Req 5.22): `--interactive` with `step`/`continue` to completion produces the same decision sequence as batch `--sim` on the same replay file.
## Security Considerations
_Feature-specific decisions only; baseline RBAC/auth practices are in `.kiro/steering/security.md`._
- **`recordFile` path validation** is already fully specified at the requirements level (Req 5 AC 10, "Record file path validation" — reject `/`/`..`, resolve under configured `record_dir`, `realpath()` containment check before `fopen()`). This design's `NormalDriver` and `SchedAdminOpcode` components inherit that requirement directly. `record_dir` itself is provisioned only via a `--record-dir` scheduler startup CLI flag — never from `SchedAdminRequest` or any other request-path field — so the validation boundary can't be redirected by a caller.
- **`OPCode_schedAdmin` ships with no RBAC/auth check beyond the primary/standby gate — accepted scope decision, deferred hardening.** The comparable `OPCode_upgradeConfig` case has no visible RBAC/auth check in `cluster.c` either (see `research.md`). Project owner decision (2026-08-02): authentication for this opcode is enhanced in a follow-up spec, not blocking this one. Phase 3 ships `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current (unauthenticated-beyond-primary-gate) posture — this is a known, accepted gap, not a silent one.
- **`.replay` files may contain sensitive cluster state** (full job/user/resource snapshots) — file permissions on `record_dir` are an operational concern (steering-level, not new to this feature) but worth calling out since this is the first feature to write full scheduler snapshots to disk in a human-decodable format (`xceadmin scheduler dump`).
## Performance & Scalability
_Feature-specific targets only; general practices are in steering docs._
- **Recording overhead**: zero file I/O, zero extra allocation when `recording == false` — a single boolean branch per session (Req 5 AC 10, explicit requirement, inherited here as a design constraint on `NormalDriver`).
- **Determinism vs throughput**: `jbuf` cbuf capacity is permanently 1 (rendezvous, not a buffer) — this bounds picker→allocator throughput to strictly sequential handoff. This is an intentional trade-off (determinism over pipelining) already made at the requirements level (Req 5 AC 7) and is not revisited here; any future throughput work must find a different mechanism that preserves the ordering guarantee (e.g. deterministic batching with an explicit order tag), not simply raise the cbuf capacity.
- **Simulation fast mode**: `speed_num=0` means the virtual clock never advances between events at wall-clock cost — `--sim` on a large `.replay` file runs at CPU-bound speed, not wall-clock speed. No specific throughput target is set here (none given in requirements); implementation should avoid introducing any per-event syscall beyond what decoding the capnp stream requires.
- **`OPCode_schedAdmin` vtask occupancy**: each request blocks its vtask handler thread up to 5s (cross-process wait on the scheduler's reply) while holding `c->sched.lock` — no admission reservation, accepted as a boundary case given expected low, human-paced request volume (see `SchedAdminOpcode` component, Design Review 2026-08-02). Not bounded or load-tested against concurrent bursts; out of scope unless usage patterns change.
## Open Questions / Risks
1. **`OPCode_schedAdmin` authorization mechanism** (P2 — accepted gap, deferred). Ships at parity with `OPCode_upgradeConfig`'s current posture (primary/standby gate only, no RBAC/auth check). Project owner decision (2026-08-02): auth hardening for this opcode is a follow-up spec, not a Phase 3 blocker. Tracked in `.kiro/steering/roadmap.md`'s existing `peer-authentication-hardening` backlog item (updated 2026-08-02 to cover `OPCode_schedAdmin` alongside `OP_upgradeConfig` — same `validate_host()` gate closes both). See Security Considerations and `research.md`.
2. **`PreemptDecision`'s wire shape is pre-committed ahead of `scheduler-preemption`'s own design** (P1). Flagged as a revalidation trigger in Boundary Commitments; `scheduler-preemption`'s design should explicitly confirm or extend this shape early, not discover it mid-implementation.
3. **`SnapshotData.internalState` opaque-blob scope** (P2). If a future spec needs `xceadmin scheduler dump` to render shared-resource-tracking or unsatisfiable-resreq-cache contents meaningfully (not just byte length), `internalState` will need to become a structured capnp field instead of opaque `Data` — deferred here because no current requirement asks for it (AC 20's dump description only requires human-readable output for what dump already needs to show, and internals were never asked to be inspectable).
4. **`xce_sched_core_t`'s `config`/`plugins`/`jobs`/`queues`/`workers`/`users`/`projects` field block is left as `var ...` in requirements.md's data-structure summary** (not fully typed) — Phase 2 implementation must finalize these as concrete `kvmgr`/pointer types when `core.c` is written; this design does not re-type them since requirements.md already commits to "moved from `xce_scheduler_t`" as the authoritative source of truth for their shapes.
5. **`OPCode_schedAdmin` blocks a vtask handler thread up to 5s per request with no admission reservation** (P2 — accepted boundary case, deferred). Project owner decision (2026-08-02): `xceadmin scheduler` commands are cluster-administrator-triggered and low-frequency, not expected to burst concurrently — no `COMMIT_WAIT_RESERVED`-style admission gate added. Revisit only if usage patterns become automated/high-frequency. See `SchedAdminOpcode` component and `research.md`.
6. **Req 5 AC 3's tlog virtual-clock logging integration is built but never activated** (P2 — found by `/kiro-validate-impl`, 2026-08-07, not yet fixed). `core_clock.c`'s `xce_core_register_tlog_format()`/`xce_core_set_active_for_logging()` are unit-tested (`test_clock.c`) but have zero call sites in `main.c`/`sim.c`/`driver.c` — task 4.5 deliberately deferred wiring them to "a later task" (its own header comment), and no later task picked it up. The correct activation site is `sim.c`'s `sched_sim_driver_start()` (after `sim_set_virtual_clock()` seeds `virtual_clock_base` to a real epoch value from the `.replay` file) — NOT `main.c`'s normal-mode path, where `virtual_clock_base=0` would make `xce_core_now()` return raw `CLOCK_MONOTONIC` boot-relative nanoseconds, making log timestamps worse, not better. Left unfixed pending a paired deactivation: `g_active_core_for_logging` is one process-wide pointer, `sched_sim_driver_run()` has no fini function (its own comment: "no sim-driver stop/fini function exists") and multiple early-return paths, and several test binaries construct more than one stack-allocated `xce_sched_sim_driver_t` per process — activating without deactivating risks a real ASAN stack-use-after-scope hazard. See tasks.md's Implementation Notes for the full trace.
7. **`plugins/fcfs.c` vs `plugins/builtin_fcfs.c` split was undocumented in this design until 2026-08-07** (P3 — documentation gap only, now fixed here). Found by `/kiro-validate-impl`: the file-structure and BuiltinFcfsPicker sections above originally said `fcfs.c` would be rewritten; the actual implementation added a new `builtin_fcfs.c` and deliberately left `fcfs.c` untouched (task 1.2's compat-wrapper exercise target). Functionally correct and confirmed live (`builtin_fcfs` is the registered default), just wasn't reflected here until this update. Tasks 2.1-2.6 in tasks.md still carry no `_Verified:` annotations (unlike every other task group) — worth backfilling for traceability.

97
.kiro/specs/scheduler-core/research.md Normal file
View file

@ -0,0 +1,97 @@
# Research & Design Decisions: scheduler-core
## Summary
- **Feature**: `scheduler-core`
- **Discovery Scope**: Complex Integration (extension of an existing C daemon, new wire protocol, new cross-component opcode)
- **Key Findings**:
- `requirements.md` is unusually design-complete for a requirements doc — most C-level contracts (struct layouts, thread model, capnp wire framing) are already normative, not left to design. Design's real job here is: (1) the still-deferred `ReplayEvent`/`ReplayDecision` payload capnp schemas, (2) the concrete File Structure Plan, (3) closing two boundary tensions the requirements doc surfaces but doesn't resolve.
- Almost every payload type `ReplayEvent.body`'s union needs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) **already exists verbatim** in `src/schema/controller.capnp` — the requirements author already picked names that match the live schema. Only `JobStateChangeEvent` and `JobUpdateEvent` are genuinely new. This is a Build-vs-Adopt win: reuse, don't reinvent.
- The scheduler component's real file layout (`src/scheduler/src/{decision.c,handler.c,main.c,plugin.c,scheduler.c,plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c},resource.c,schedobj.c}`) is substantially larger than `.wolf/anatomy.md`'s cached listing (which only knew `handler.c`, `main.c`, `scheduler.c`). File Structure Plan below is built from the live `find`, not the stale cache.
## Research Log
### Existing scheduler component layout
- **Context**: Need concrete file paths for the File Structure Plan; `structure.md` gives the pattern (`include/`, `src/`, `testsuite/`, plugin subdir), but not the current file set.
- **Sources Consulted**: `find src/scheduler -name '*.c' -o -name '*.h'`, `src/scheduler/testsuite/meson.build`.
- **Findings**: `include/`: `decision.h`, `plugin.h`, `sched/types.h`, `scheduler.h`. `src/`: `decision.c`, `handler.c`, `main.c`, `plugin.c`, `scheduler.c`, `resource.c`, `schedobj.c`, `plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c}`. `testsuite/`: `test.c` (runner harness), `test_handlers.c`, `test_job_array_expand.c`, `test_job_array_wire_e2e.c`, `test_job_update.c` — standalone TUs, several `#include ../src/*.c` directly (white-box pattern) rather than linking.
- **Implications**: New core/driver files (`xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`) are new `src/` files, not modifications of `scheduler.c` wholesale — `scheduler.c` keeps today's `xce_scheduler_t`-owning code during Phase 1, then Phase 2 extracts state into a new `core.c`. Test files for the new pieces follow the existing standalone-TU-with-`#include`, one-binary-per-unit convention already established in `testsuite/meson.build`.
### Controller-side opcode dispatch location and pattern
- **Context**: AC 10 requires a new `OPCode_schedAdmin` case; need to confirm which file owns `myreq_handler` and the exact style of an existing opcode case for consistency.
- **Sources Consulted**: `grep -rln myreq_handler src/controller/*.c`, `cluster.c` around `OPCode_fetchfile` and `OPCode_upgradeConfig`.
- **Findings**: `myreq_handler` lives in `src/controller/cluster.c` (also referenced from `ha.c`/`ha_convergence.c`, but the dispatch switch itself is in `cluster.c`). The closest structural precedent is `OPCode_upgradeConfig` (`cluster.c:670-705`): gated by `ha_is_primary(&c->ha)` (reject standby with `-EAGAIN`, `RequestReply_data_nodata`), then delegates to a `handle_*()` function. **No RBAC/auth check is visible in that case block** — the comment trail (`cluster.c:672-693`, bug-fix 2026-07-25) discusses primary/standby gating exhaustively but never mentions an authentication check at this opcode. If `OPCode_upgradeConfig` has one, it happens earlier (TLS handshake / connection-accept time) or not at all at this layer.
- **Implications**: AC 10's claim "same RBAC role as config push" cannot be verified against a concrete gate in the code as it stands — carried into design.md as an **Open Question**, not asserted as a resolved fact. `OPCode_schedAdmin`'s case block should structurally mirror `OPCode_upgradeConfig` (primary-only gate, `handle_*()` delegation, explicit reply on every path) since that is the only comparable admin-opcode precedent in the file.
### xceadmin CLI extension pattern
- **Context**: AC 10's `xceadmin scheduler record start/stop/status` and AC 20's `xceadmin scheduler dump` need a concrete module plan consistent with the existing `config push`/`config build` implementation.
- **Sources Consulted**: `src/xceadmin/src/xceadmin/{cli.py,push.py,build.py,discovery.py}`, `src/xceadmin/src/xcecommon/{wire.py,schema.py}`.
- **Findings**: `cli.py` is a thin `click` layer only (`@cli.group()` per resource, `@group.command()` per verb) — all business logic lives in per-resource modules (`push.py`, `build.py`) that a) load the schema once via `xcecommon.schema.load_schema()`, b) build/decode capnp messages, c) go through `xcecommon.wire.connect`/`send_request` for the socket, and d) for request/reply opcodes, route through `xceadmin.discovery.follow_redirects` + `xceadmin.discovery.decode_reply` (which unwraps the `Request{opcode: OPCode_reply, body: replymsg}` envelope every reply is wrapped in — a real bug fixed 2026-07-30, documented in `discovery.py`'s own docstring). Every raising path uses one exception hierarchy (`XceAdminError` subclasses), caught once in `cli.py`.
- **Implications**: A new `xceadmin/scheduler.py` module is the natural home for `record_start()`, `record_stop()`, `record_status()`, and `dump()` — same shape as `push_config()`/`probe_config()`, reusing `discovery.follow_redirects`/`decode_reply` (this is a request/reply opcode per AC 10, so unlike `probe_config()` it DOES need redirect-following). `dump()` differs from the other three: it operates on a local `.replay` file only, no network call — it decodes a capnp stream with `xcecommon.schema.load_schema()`'s module object and formats each message type to text; it belongs in the same module for CLI cohesion but has no wire dependency.
### Reusable capnp payload types for ReplayEvent / SnapshotData
- **Context**: Requirements AC 4 explicitly defers the exact capnp schema for event payloads to design; AC 20's `ReplayEvent.body` union names `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `JobStateChangeEvent`, `JobUpdateEvent`, `QueueConf`, `SchedJobRusages` without defining the two `*Event` types.
- **Sources Consulted**: `grep -n '^struct ' src/schema/controller.capnp`.
- **Findings**: `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages` **all already exist** in `controller.capnp` with those exact names (the requirements author evidently cross-checked the schema before naming the union members). `JobStateChangeEvent` and `JobUpdateEvent` do not exist. The existing `Job` struct (`controller.capnp:288`) already carries `jobid` + `state`, and `JobAlloc`/`JobPendReason` exist for allocation/pend-reason payloads that the new `DispatchDecision`/`PendDecision` need.
- **Implications**: Design only needs to define 2 new event payload structs and 3 new decision payload structs (reusing `JobAlloc` and `JobPendReason` internally), not 9. See Data Models section of `design.md`.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Event-sourced core with pluggable driver (selected) | Core holds all scheduling state, driven purely by typed events + a pull-based clock; two driver implementations (normal, simulation) translate their respective input sources to the same event stream | Normal↔sim equivalence becomes a structural property, not a discipline; picker/allocator threading is unchanged from today, minimizing regression surface | Requires strict discipline (no wall-clock calls, no controller calls) inside the core — a single violation breaks the central invariant silently until someone runs `--sim-verify` | Already fully specified in requirements.md; design.md's job is to make the boundary and remaining schema concrete, not to re-litigate the pattern |
| Mode-flagged single scheduler (rejected in requirements) | Keep `xce_scheduler_t` as-is, add `if (sim_mode)` branches for clock/IO | No new types, smallest diff | Directly violates the central invariant requirements establishes (normal/sim would share code paths only by discipline, not by construction) — rejected by requirements.md Req 5 AC 1, not re-opened here | — |
## Design Decisions
### Decision: New capnp structs for JOB_STATE_CHANGE / JOB_UPDATE events
- **Context**: AC 4 specifies the field set (jobid, old_state, new_state, exit_code for state-change; jobid, changed_fields, full job record for update) but no capnp shape exists.
- **Alternatives Considered**:
1. Reuse `Job` alone and let the driver infer old/new state from two full records — rejected, doubles payload size for a two-field delta and loses `exit_code`.
2. Define thin new structs that carry only the delta plus embed `Job` where a full record is genuinely needed — selected.
- **Selected Approach**: `JobStateChangeEvent{jobid, oldState, newState, exitCode}`; `JobUpdateEvent{jobid, changedFields, job: Job}` (embeds the existing `Job` struct, not a re-derived one).
- **Rationale**: Matches AC 4's field table exactly; embedding `Job` for the update case (where the driver already has to build a full record) avoids a second job-shape definition; the state-change case stays flat since AC 4 requires no other job fields.
- **Trade-offs**: `JobUpdateEvent` duplicates the full `Job` payload even for a single-field priority change — acceptable, matches AC 4's explicit field table (`job: xce_job_t*` full record), not a design deviation.
- **Follow-up**: None — field set is normative from requirements, only the capnp wrapper is new.
### Decision: Decision payload structs (DispatchDecision / PendDecision / PreemptDecision) — and a boundary note on PreemptDecision
- **Context**: AC 5 fully specifies fields for all three decision types. Separately, the Boundary Context assigns "the preemption decision type" to the `scheduler-preemption` sibling spec.
- **Alternatives Considered**:
1. Omit `PreemptDecision` entirely from this design, leave a placeholder in the `ReplayDecision.body` union for `scheduler-preemption` to add later — rejected, AC 5's decision output table is a Requirement 5 AC in *this* spec, and the `on_decision` callback signature / `ReplayDecision` union must compile today (Phase 2) before `scheduler-preemption` exists.
2. Define `PreemptDecision`'s wire shape now per AC 5's field table, treat victim-selection *semantics* (not the wire shape) as what `scheduler-preemption` owns — selected.
- **Selected Approach**: `PreemptDecision{jobid, victims: List(PreemptVictim{jobid, workerid}), reason: JobPendReason}` defined in this spec's schema additions.
- **Rationale**: The wire struct is a passive data shape with no scheduling logic in it; nothing about defining it here binds `scheduler-preemption` to any victim-selection algorithm. Leaving the union member undefined would make `xce_decision_t`/`ReplayDecision` incomplete types this spec's own Phase 2 code needs to reference (`can_preempt()`'s caller, per requirements.md Req 1 AC 4's allocator pseudocode, already branches on a preempt outcome).
- **Trade-offs**: If `scheduler-preemption` later needs additional fields on `PreemptDecision`, that spec must extend this schema (a revalidation trigger, listed in design.md's Boundary Commitments) rather than owning it outright from scratch.
- **Follow-up**: Flag explicitly to whoever picks up `scheduler-preemption` that the wire shape is pre-committed here; the spec should confirm the field set is sufficient before implementation.
### Decision: `OPCode_schedAdmin` authorization mechanism — deferred to a follow-up spec (project owner call, 2026-08-02)
- **Context**: AC 10 states "same RBAC role as config push," but the only comparable opcode (`OPCode_upgradeConfig`) has no visible authorization check in `cluster.c` — only a primary/standby gate.
- **Alternatives Considered**:
1. Assert a specific mechanism (e.g. "add a `cluster:control` RBAC check") without verifying one exists for `config push` today — rejected, would document a fictitious parity claim.
2. Block Phase 3 until the implementer traces `config push`'s real auth boundary and applies it here too — rejected by project owner: not worth gating this spec's delivery on.
3. Ship `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current posture (primary-gate only, no RBAC check), document the gap explicitly, and track auth hardening as separate future work — **selected**.
- **Selected Approach**: Option 3. `design.md`'s Open Questions/Risks and Security Considerations sections record this as an accepted, deliberate gap rather than an unresolved blocker.
- **Rationale**: `config push`'s current authorization boundary could not be located in the controller-side handler during discovery. Rather than block this spec on retrofitting auth for two opcodes at once, the project owner chose to scope that work separately.
- **Trade-offs**: `OPCode_schedAdmin` (and, unchanged, `OPCode_upgradeConfig`) remain gated only by primary/standby routing until the follow-up lands — acceptable short-term since both are already TLS-connected, admin-tooling-only surfaces (xceadmin), not general network-reachable endpoints.
- **Follow-up**: Add an "auth hardening for admin opcodes (`OPCode_upgradeConfig`, `OPCode_schedAdmin`)" item to the project roadmap (see `design.md` Open Questions/Risks #1). Track as a future spec, not a task inside `scheduler-core`.
### Decision: `OPCode_schedAdmin`'s blocking cross-process wait — accepted as a boundary case, no admission reservation (project owner call, 2026-08-02)
- **Context**: Design review (2026-08-02) flagged that `myreq_handler` runs as a `vtask_t` handler (bounded 8-thread pool, `cluster.c:238`), and `OPCode_schedAdmin` blocks that thread up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock` — unlike `OPCode_upgradeConfig`'s in-process, `COMMIT_WAIT_RESERVED`-admitted commit-barrier wait. `ha-async-io.md`'s vtaskmgr rule requires admission reservation for bounded blocking and forbids unbounded blocking on this pool; the `raft-drive-async` family fixed prior real violations of this exact rule (`config-push`'s busy-poll+join was one of them).
- **Alternatives Considered**:
1. Add a `COMMIT_WAIT_RESERVED`-style admission reservation for `OPCode_schedAdmin` before Phase 3 ships — the design-review recommendation.
2. Make the scheduler forward asynchronous (vtask completion callback instead of blocking wait) — larger structural change, deferred as unnecessary for current usage.
3. Accept the blocking wait as a boundary case given expected request volume — **selected**.
- **Selected Approach**: Option 3. `xceadmin scheduler record`/`dump` commands are cluster-administrator-triggered, human-paced, low-frequency — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism is added.
- **Rationale**: The `COMMIT_WAIT_RESERVED` precedent exists because commit-barrier waits are on the hot path of every job submission (high, uncontrolled frequency). Admin recording control is operator-initiated and rare by nature; the risk this pattern protects against in the commit-barrier case doesn't apply at comparable severity here.
- **Trade-offs**: If usage patterns change (e.g. automated/scripted recording control invoked at scale, or a monitoring tool polling `record status` frequently), this becomes a live risk without warning — no test or admission gate would catch it.
- **Follow-up**: If `scheduler record`/`dump` usage ever becomes automated or high-frequency, revisit with a `COMMIT_WAIT_RESERVED`-style admission reservation.
## Risks & Mitigations
- **`OPCode_schedAdmin` ships without an RBAC/auth check beyond the primary/standby gate — accepted, not blocking.** Mitigation: documented explicitly in `design.md`; tracked as a future roadmap item (see Decision above), not a Phase 3 dependency.
- **`OPCode_schedAdmin` blocks a vtask thread for up to 5s per request with no admission reservation — accepted boundary case, not mitigated.** Rationale: low-frequency, human-triggered request class (see Decision above). Revisit only if usage patterns change.
- **`PreemptDecision` wire shape is pre-committed before `scheduler-preemption`'s own design exists** — mitigation: documented as a revalidation trigger; `scheduler-preemption` must review this shape early rather than discovering it mid-implementation.
- **`jbuf` capacity=1 determinism invariant depends on a `FIXME: make this configurable` comment never being resolved carelessly** — mitigation: already hard-pinned in requirements.md Req 5 AC 7 with an explicit Phase 2 task; design.md's File Structure Plan marks `xce_sched_core_init()` as the single place this is enforced.
## References
- `src/scheduler/CLAUDE.md`, `.kiro/steering/{structure.md,tech.md,ha-async-io.md}` — component layout and controller HA invariants.
- `src/schema/controller.capnp` — canonical wire schema; source of truth for reused struct names.
- `src/xceadmin/src/xceadmin/push.py`, `discovery.py` — reference pattern for the new `scheduler.py` xceadmin module.