From 7a5a864fb6a491e6563b70582f574275caf9eedb Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 8 Aug 2026 03:19:01 +0000 Subject: [PATCH] docs(scheduler-core): commit design.md/research.md, sync design.md with as-built code Both files existed only on disk since the design phase (2026-08-03) and were never committed despite ~40 implementation commits landing since. Also corrects design.md drift found by /kiro-validate-impl: the File Structure Plan and BuiltinFcfsPicker sections claimed plugins/fcfs.c would be rewritten in place, but the actual implementation added a new plugins/builtin_fcfs.c and deliberately kept fcfs.c untouched as the job_elect_fn compat-wrapper's exercise target. Updates the directory tree to the real file layout (core_events/session/clock.c, sim.c/ sim_loop.c/sim_record.c/sim_verify.c, snapshot/recording/repl splits, test_e2e_capnp.c vs the planned test_replay_format.c, test_sched_admin_dispatch.c vs test_sched_admin.c), and adds two Open Questions/Risks entries for the AC 3 tlog-logging-integration gap and this fcfs.c/builtin_fcfs.c documentation gap, both found during validation. Co-Authored-By: Claude Sonnet 5 --- .kiro/specs/scheduler-core/design.md | 794 +++++++++++++++++++++++++ .kiro/specs/scheduler-core/research.md | 97 +++ 2 files changed, 891 insertions(+) create mode 100644 .kiro/specs/scheduler-core/design.md create mode 100644 .kiro/specs/scheduler-core/research.md diff --git a/.kiro/specs/scheduler-core/design.md b/.kiro/specs/scheduler-core/design.md new file mode 100644 index 00000000..d9a3fa4c --- /dev/null +++ b/.kiro/specs/scheduler-core/design.md @@ -0,0 +1,794 @@ +# Technical Design: scheduler-core + +## Overview + +**Purpose**: This feature replaces the scheduler's hardcoded FCFS job-selection logic and its hard-wired coupling to the live controller with a pluggable picker interface and a passive, mode-agnostic scheduling core. The core is driven by either a normal-mode driver (live controller) or a simulation-mode driver (recorded `.replay` file), and both modes are required to produce byte-identical scheduling decisions given identical input — this is the spec's central, testable invariant. + +**Users**: Cluster operators configure job-picking policy without a rebuild (Req 2) and gain a `xceadmin scheduler record`/`dump` workflow. Scheduler developers gain a `--sim`/`--interactive` simulation driver to debug and tune scheduling policy against recorded production traces without a live cluster (Req 5). + +**Impact**: `xce_scheduler_t` (today: scheduling state + `apictx` + `cluster_cfg` in one struct, `scheduler.c`) splits into `xce_sched_core_t` (pure scheduling state, zero I/O) plus `xce_sched_driver_t` (normal, controller-attached) and `xce_sched_sim_driver_t` (simulation, replay-attached). The hardcoded `"xce_sched_fcfs"` string at `scheduler.c:43` is replaced by a configurable, plugin-registered picker. A new controller opcode (`OPCode_schedAdmin`) and a new xceadmin command group (`scheduler record`/`scheduler dump`) are added as pass-through admin plumbing. + +### Goals +- Job-picking policy configurable per-scheduler and per-queue without recompilation, with `builtin_fcfs` shipping as the zero-behavior-change default (Req 1-4). +- Scheduling core with zero controller I/O, wall-clock, or filesystem dependencies, driven identically by a normal or simulation driver (Req 5 AC 1-8). +- Runtime recording of live scheduling sessions to a self-contained `.replay` file, and a simulation driver + interactive REPL that replays it deterministically (Req 5 AC 9-22). +- Normal↔sim equivalence as a verifiable regression test: record a live session, replay it, diff the decisions — zero divergence (Req 5 AC 7.1). + +### Non-Goals +- New job-picking policies beyond `builtin_fcfs` (interface only; sibling work). +- Allocation iterators / `try_allocate()` internals (`scheduler-allocation`). +- Preemption policy / victim selection (`scheduler-preemption`) — this design defines `PreemptDecision`'s **wire shape** only (needed for `Req 5 AC 5`'s decision table to compile), not preemption logic. +- Hierarchical project resource quotas (`project-resource-ownership`). +- REPL state mutation, reverse execution, or scripting (explicitly deferred in Req 5 AC 22). + +## Boundary Commitments + +### This Spec Owns +- `xce_job_picker_iface_t` and the `builtin_fcfs` implementation. As-built: a new `src/scheduler/src/plugins/builtin_fcfs.c` implements the interface; `plugins/fcfs.c` was deliberately left unchanged (still `job_elect_fn`-only) as the live exercise target for `plugin.c`'s legacy compat wrapper (see Components — BuiltinFcfsPicker). +- `xce_sched_core_t`: all scheduling state, the picker/allocator thread pair, the driver↔core protocol (`core_push_events`, `core_go`, `on_decision`), and `xce_core_now()` clock virtualization. +- `xce_sched_driver_t` (normal-mode): controller event translation, decision publishing, and recording control (start/stop/status, periodic snapshot). +- `xce_sched_sim_driver_t` (simulation-mode): `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`, and the interactive REPL. +- The `.replay` capnp wire format (`ReplayHeader`, `SnapshotEntry`, `ReplayEvent`, `ReplaySession`, `ReplayDecision`, and their payload sub-structs) in `src/schema/controller.capnp`. +- The `OPCode_schedAdmin` request/reply pair and its **pass-through** dispatch case in the controller's `myreq_handler` (`src/controller/cluster.c`), plus its accompanying `src/controller/testsuite/` coverage — this spec's only footprint outside `src/scheduler/`. +- `xceadmin scheduler record start|stop|status` and `xceadmin scheduler dump ` (new `src/xceadmin/src/xceadmin/scheduler.py`). +- `job_pick_plugin` (global + per-queue) and `project_name` (per-queue) fields; `xce_sched_project_t` scaffolding (name + priority only — no quota logic). + +### Out of Boundary +- `try_allocate()` and the allocation iterator (`scheduler-allocation` owns the implementation; this spec only calls it from the allocator thread, unchanged signature). +- `can_preempt()` and victim-selection semantics (`scheduler-preemption` owns the algorithm; this spec owns only `PreemptDecision`'s wire shape, see Data Models). +- Hierarchical project quota, `resource_quota`/`resource_used`/`parent_project` on `xce_sched_project_t` (`project-resource-ownership`). +- Any controller state or persistence for recording (`OPCode_schedAdmin` is explicitly a stateless pass-through — the controller commits nothing, owns no WAL entry for it). +- REPL write-side mutation, semantic snapshot diff in xceadmin (both explicitly deferred, Req 5 AC 22). + +### Allowed Dependencies +- Existing plugin framework (`plugin.h`/`plugin.c`) — `xce_job_picker_iface_t` is a new plugin category registered through the same `plugin_register`/`key_find` mechanism, not a parallel system. +- Existing select-expression evaluator (`resreq.h`, `select_eval_ctx_t`) — used as-is by the allocator thread; the core does not re-implement or wrap it. +- libxce data-structure subset only inside the core: `ordq`, `dataq`, `cbuf`, `xlist`, `zarray`, `kvmgr`, `rtti` (Req 5 AC 3). The core must **not** link `netio`, `delay`, `xceapi`, or `idgen`. +- `xcecli_ctx_t` API (`xcecli_jobdispatch`, `xcecli_set_pend_reason`, `xcecli_preempt`) — normal-mode driver only, never referenced from `xce_sched_core_t` or `xce_sched_sim_driver_t`. +- `xcecommon.wire`/`xcecommon.schema`/`xceadmin.discovery` (Python) — the new `xceadmin/scheduler.py` module reuses these exactly as `push.py` does; it does not open its own socket or reload the schema. + +### Revalidation Triggers +- Any change to `xce_job_picker_iface_t`'s hook signatures — every picker plugin (present and future) must be re-verified. +- Any change to `jbuf` cbuf capacity away from 1 — invalidates the determinism argument in Req 5 AC 7; `scheduler-allocation` and `scheduler-preemption` must both re-verify ordering assumptions. +- Any change to the `.replay` capnp schema's message set or `ReplayDecision.body` union — `scheduler-preemption`'s `PreemptDecision` consumer and any external tooling reading `.replay` files must re-check compatibility. +- Any change to `OPCode_schedAdmin`'s request/reply shape — `xceadmin.scheduler` module must be updated in lockstep (same repo, same PR). + +## Architecture + +### Existing Architecture Analysis + +Today, `xce_scheduler_t` (`src/scheduler/include/scheduler.h`) owns scheduling state (`jobs`, `queues`, `workers`, `plugins` kvmgrs), controller I/O (`apictx: xcecli_ctx_t`, `cluster_cfg: xce_cluster_config_t`), and three threads (`jobpicker`, `allocator`, main) wired through three `cbuf`s (`wbuf` cap 1, `jbuf` cap `njobs`≡1 today, `decisions` cap `ndecisions`≡5). `main.c`'s loop calls `sched_update` → `sched_begin_session` → `sched_go` → drains `decisions` cbuf with a hardcoded `switch (decision->op)` → `sched_end_session`. The picker plugin (`fcfs.c`) is selected by a literal string (`scheduler.c:43`) and implements the old `xce_sched_plugin_t->job_elect_fn` hook, tightly coupled to `sched->queues`/`sched->jobs` kvmgrs directly. + +**Preserved without change**: the three-thread/three-cbuf pattern (`wbuf`→picker, `jbuf`→allocator, `decisions`→driver); `plugin_register`/`key_find` plugin lookup; `select_eval_ctx_t` per-session memoization; `sched_queue_is_ready()`/`sched_job_is_ready()` (currently stub `return 1;` in `scheduler.c`, called as-is by both today's `fcfs_elect_job` and the new `builtin_fcfs`). + +**What changes**: `xce_scheduler_t` splits along the controller-I/O seam into `xce_sched_core_t` (no `apictx`/`cluster_cfg`) and `xce_sched_driver_t` (holds them). The picker plugin category becomes `xce_job_picker_iface_t`, replacing `job_elect_fn` (deprecated with a compatibility wrapper, Req 5 AC "Migration from old job_elect_fn"). + +### Architecture Pattern & Boundary Map + +```mermaid +graph TB + subgraph DriverLayer[Driver Layer] + NormalDriver[Normal Driver] + SimDriver[Simulation Driver] + Controller[Live Controller] + ReplayFile[Replay File] + Controller --> NormalDriver + ReplayFile --> SimDriver + end + + subgraph CoreLayer[Scheduler Core] + Core[Scheduler Core State] + Picker[Job Picker Plugin] + Allocator[Allocator Thread] + Core --> Picker + Picker --> Allocator + end + + NormalDriver -->|core_push_events core_go| Core + SimDriver -->|core_push_events core_go| Core + Core -->|on_decision| NormalDriver + Core -->|on_decision| SimDriver + + NormalDriver -->|xcecli| Controller + SimDriver -->|write| RecordFile[Sim Output Replay File] + + subgraph AdminPath[Admin Control Path] + XceAdmin[xceadmin scheduler CLI] + SchedAdminOp[OPCode_schedAdmin dispatch] + XceAdmin --> SchedAdminOp + SchedAdminOp --> Controller + end + Controller -->|forward request-reply| NormalDriver +``` + +**Architecture Integration**: +- **Selected pattern**: Event-sourced core with a swappable driver (ports-and-adapters, applied narrowly — the core is the hexagon, the two drivers are the only adapters). Selected in requirements.md as the only pattern that makes normal↔sim equivalence a structural property rather than a discipline (see `research.md`, Architecture Pattern Evaluation). +- **Domain boundaries**: Core owns scheduling *decisions*; drivers own I/O translation *only* — no scheduling logic in either driver. `OPCode_schedAdmin` dispatch is a third, thin adapter (controller-side) that never touches core or driver state directly, only forwards bytes. +- **Existing patterns preserved**: three-thread/cbuf pipeline, plugin registration, select-expression memoization. +- **New components rationale**: `SimDriver` and the `.replay` format are net-new (no existing analog); `SchedAdminOpcode` is new but structurally mirrors the existing `OPCode_upgradeConfig` case (see `research.md`). +- **Steering compliance**: no new wall-clock/network dependency inside `src/scheduler/`'s core files (`ha-async-io.md`'s async-I/O discipline extended to a synchronous core by design, not violated — the core has no I/O at all); `.capnp` used for the new wire format per `structure.md`'s "Cap'n Proto for wire formats" convention. + +### Technology Stack + +| Layer | Choice / Version | Role in Feature | Notes | +|-------|------------------|------------------|-------| +| Scheduler core / drivers | C (gnu17) | `xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t` | No new language/runtime; existing `src/scheduler/` toolchain | +| Wire format | Cap'n Proto (existing `controller.capnp`) | `.replay` stream, `OPCode_schedAdmin` request/reply | Extends the existing schema file, not a new one | +| Admin CLI | Python / `click` (existing xceadmin stack) | `scheduler record`/`scheduler dump` commands | New module (`xceadmin/scheduler.py`) in the existing package, same `pycapnp` schema loader | +| Build | Meson ≥ 1.5 (existing) | New `.c` files added to `src/scheduler/meson.build`; testsuite entries | No new build system | + +### Dependency Direction + +`Picker Plugin Interface → Scheduler Core → {Normal Driver, Simulation Driver} → {Controller (xcecli), Replay File I/O}`. Controller-side `SchedAdminOpcode` dispatch depends only on the capnp schema (shared, not owned by the core) and forwards to the driver without depending on `xce_sched_core_t` internals. xceadmin's `scheduler.py` depends on `xcecommon`/`xceadmin.discovery` only — never on any C header. Each layer imports only from layers to its left; the core never imports driver headers (enforced by the "Allowed Dependencies" list above — no `xcecli_ctx_t`/`netio` symbol may appear in `core.c`). + +## File Structure Plan + +### Directory Structure + +**As-built** (updated 2026-08-07 after implementation; see tasks.md's `## Implementation Notes` for the full rationale behind each split — all were 800-line-file-cap splits made during implementation, not scope changes): + +``` +src/scheduler/ +├── include/ +│ ├── scheduler.h # unchanged Phase 1; Phase 2 shrinks to xce_scheduler_t compat shim +│ ├── plugin.h # + xce_job_picker_iface_t (Req 1 AC 2), deprecation of job_elect_fn +│ ├── core.h # xce_sched_core_t, xce_core_event_t, decision cb type +│ ├── driver.h # xce_sched_driver_t (normal-mode) +│ ├── sim.h # xce_sched_sim_driver_t -- built as sim.h, not sim_driver.h (task 11.1, documented deviation) +│ ├── recording.h # record start/stop/status types, ReplayEvent/ReplaySession writer contract +│ ├── repl.h # REPL hook wrappers consumed by sim_loop.c -- no-op unless sim->interactive +│ ├── decision.h # pre-existing; extended with XCE_DECISION_PREEMPT (task 4.4) +│ └── sched/types.h # + job_pick_plugin, project_name on xce_sched_queue_t (Req 2) +├── src/ +│ ├── scheduler.c # sched_job_picker(): job_elect_fn != NULL delegates to the legacy loop, NULL runs the generic xce_job_picker_iface_t loop +│ ├── core.c # xce_sched_core_init/fini, jbuf(cap=1)/decisions cbuf plumbing +│ ├── core_events.c # xce_core_push_events() -- split out of core.c (800-line cap; task 4.3) +│ ├── core_session.c # xce_core_begin_session/xce_core_end_session -- split out of core.c (task 4.4) +│ ├── core_clock.c # xce_core_now(), tlog format integration -- the ONE file allowed to call clock_gettime() (task 4.5) +│ ├── driver.c # normal-mode driver: capnp<->core event translation, decision publishing, deferred admin-request handling +│ ├── recording.c # record start/stop/status, header/snapshot writes +│ ├── recording_events.c # ReplayEvent writes -- split out of recording.c (task 9.2/9.3) +│ ├── recording_session.c # ReplaySession begin/end writes -- split out of recording.c (task 9.2/9.3) +│ ├── repl.c # REPL command loop core (execution control, Req 5 AC 22) +│ ├── repl_break.c # REPL breakpoints + event/decision hook wrappers -- split out of repl.c (task 12.2) +│ ├── repl_info.c # REPL state-inspection commands -- split out of repl.c (task 12.2) +│ ├── repl_params.c # REPL snapshot/simulation-parameter commands -- split out of repl.c (task 12.3) +│ ├── repl_internal.h # shared repl_*.c internal declarations (task 12.2) +│ ├── snapshot.c # save_snapshot()/load_snapshot() orchestration, picker rebuild +│ ├── snapshot_wire.c # SnapshotData capnp encode/decode -- split out of snapshot.c (task 8.1) +│ ├── snapshot_load.c # snapshot restore application -- split out of snapshot.c (task 8.1) +│ ├── sim.c # simulation startup (Req 5 AC 13): replay header/snapshot load, virtual clock seed +│ ├── sim_loop.c # simulation event loop + REPL hook call sites (Req 5 AC 14/18) +│ ├── sim_record.c # --sim-output ReplayEvent/ReplaySession writes -- split out of sim_loop.c (task 11.5) +│ ├── sim_verify.c # --sim-verify decision comparator -- split out of sim_loop.c (task 11.4) +│ ├── handler.c # unchanged — capnp push/notify enqueue into pend_updates; scheduler_req_handler dispatches SchedAdminRequest +│ ├── plugin.c # + picker plugin registration category, + job_elect_fn compat + # wrapper (adapts old-style job_elect_fn-only plugins to + # xce_job_picker_iface_t, logs a deprecation warning at register time) +│ └── plugins/ +│ ├── builtin_fcfs.c # the native xce_job_picker_iface_t default picker (Req 3) -- NEW file, see note below +│ └── fcfs.c # UNCHANGED -- still registers job_elect_fn only; kept as the live exercise target for plugin.c's compat wrapper (task 1.2). NOT rewritten as originally planned here; see BuiltinFcfsPicker component notes. +├── testsuite/ +│ ├── test_picker_iface.c, test_picker_fcfs.c, test_picker_fcfs_session.c, test_picker_wiring.c, +│ │ test_picker_config_fields.c, test_picker_core_loop_boundary.c, test_picker_pendreason_factory_audit.c, +│ │ test_plugin_picker_registration.c # picker interface + builtin_fcfs + registration/compat-wrapper coverage +│ ├── test_core_events.c, test_core_session.c, test_core_full_cycle.c, test_core_isolation.c, +│ │ test_core_jbuf_capacity_pin.c # core contract + determinism-pin coverage +│ ├── test_clock.c # xce_core_now() two-base+speed formula +│ ├── test_driver_translate.c, test_driver_decision_cb.c, test_driver_session_ordering.c +│ ├── test_recording_path_validation.c, test_record_lifecycle.c, test_admin_request.c +│ ├── test_snapshot_save_load.c +│ ├── test_sim_startup.c, test_sim_event_loop.c, test_sim_isolation.c, test_sim_alt_events.c, +│ │ test_sim_verify.c, test_sim_record_output.c, test_sim_equivalence.c +│ └── test_repl_execution_control.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c, +│ test_repl_equivalence.c +src/controller/ +├── cluster.c # + case OPCode_schedAdmin (Req 5 AC 10) +└── testsuite/ + └── test_sched_admin_dispatch.c # standby rejection, disconnected-scheduler, timeout, success round-trip + # (name differs from this plan's original test_sched_admin.c; same coverage, + # registered in controller_test.c same as existing opcode tests) +src/schema/ +└── controller.capnp # + SchedAdminRequest/Reply, ReplayHeader/.../ReplayDecision + payload structs (see Data Models); + # + WalCatchupEntry.payload/batchPayloads switched Data->List(UInt8) (bug-1505, follow-up + # outside this spec's boundary, same codec-crash class as SnapshotData.internalState/bug-1504) +src/xceadmin/src/xceadmin/ +├── cli.py # + @cli.group() scheduler; nested @scheduler.group() record with + # start/stop/status commands, plus @scheduler.command() dump (thin layer only) +└── scheduler.py # record_start/stop/status(), dump() — mirrors push.py's pattern +src/xceadmin/tests/ +├── test_scheduler.py # mock-TLS-server tests for record start/stop/status + dump parsing +└── test_cli_scheduler_mock.py # additional CLI-level mock-server coverage (task 10.6, beyond this plan's original file list) +``` + +### Modified Files +- `src/scheduler/src/scheduler.c` — Phase 1: hardcoded picker string removed; Phase 2: `sched_job_picker()` branches `job_elect_fn != NULL` (legacy plugins, delegates to their own self-contained loop) vs `NULL` (native `xce_job_picker_iface_t` pickers, generic per-session loop) — `scheduler.c` was not retired; it remains the picker-dispatch and allocator-loop home. +- `src/scheduler/src/plugins/fcfs.c` — **as-built: left unchanged**, not rewritten. `plugins/builtin_fcfs.c` (new file) implements `xce_job_picker_iface_t` and is the registered default (`schedobj.c`); `fcfs.c` keeps registering `job_elect_fn` only, deliberately, as the live exercise target for `plugin.c`'s legacy compat wrapper (task 1.2) — see BuiltinFcfsPicker component notes for the full rationale. +- `src/scheduler/src/main.c` — decision-drain `switch` replaced by `on_decision` callback registration (Req 5 AC 6); also gained the `--sim`/`--interactive`/`--record-dir`/`--sim-*` CLI surface (tasks 9.1, 11.1). +- `src/controller/cluster.c` — one new `case OPCode_schedAdmin:` block, structurally mirroring `OPCode_upgradeConfig` (primary-gate, delegate, explicit reply every path). +- `src/schema/controller.capnp` — additive only: new `OPCode_schedAdmin` enum value, new structs, two new union members on existing `Request.body`/`RequestReply.data` unions (ordinals `@31`/`@19` respectively — next available, verified against current schema in `research.md`). +- `src/xceadmin/src/xceadmin/cli.py` — one new `@cli.group() scheduler`, containing a nested `@scheduler.group() record` (with `start`/`stop`/`status` commands) plus a sibling `@scheduler.command() dump` — matching the literal CLI syntax `xceadmin scheduler record start`/`xceadmin scheduler dump`, not 4 flat commands. + +## System Flows + +### Normal-Mode Scheduling Round (driver ↔ core) + +```mermaid +sequenceDiagram + participant Ctrl as Controller + participant Drv as Normal Driver + participant Core as Scheduler Core + participant Pick as Picker Thread + participant Alloc as Allocator Thread + + Ctrl->>Drv: capnp push or notify + Drv->>Drv: buffer into pend_updates + Drv->>Core: core_push_events + Core->>Pick: state change hooks, synchronous, picker idle + Drv->>Core: core_go, non blocking + Core->>Pick: signal via wbuf + Pick->>Pick: begin_session then next_job loop + Pick->>Alloc: send jobid via jbuf per job + Alloc->>Alloc: try_allocate then can_preempt + Alloc->>Drv: send decision via decisions cbuf + Drv->>Ctrl: on_decision publishes dispatch, pend reason, or preempt + Alloc->>Drv: end of session marker via decisions cbuf + Drv->>Core: core_end_session +``` + +Key decision not obvious from the diagram: decisions publish to the controller **immediately per-decision**, not batched at session end (Req 5, "Decision publishing") — the drain loop in the driver thread interleaves `cbuf_get`/`on_decision` one at a time. + +### `OPCode_schedAdmin` Recording Control (admin path) + +```mermaid +sequenceDiagram + participant CLI as XceAdmin SchedulerModule + participant Ctrl as Controller ClusterDispatch + participant Sched as Scheduler NormalDriver + + CLI->>Ctrl: schedAdmin request, action plus recordFile + Ctrl->>Ctrl: check ha_is_primary, reject standby + Ctrl->>Sched: forward via scheduler endpoint, five second timeout + Sched->>Sched: validate recordFile, reject or resolve under record_dir + Sched->>Sched: save_snapshot then write header and snapshot entry + Sched-->>Ctrl: schedAdmin reply, ok plus recording state and counters + Ctrl-->>CLI: relay reply to caller +``` + +Key decision not obvious from the diagram: the controller is a pure pass-through — it does not interpret `SchedAdminRequest` fields beyond the opcode dispatch, and owns no recording state (Boundary Commitments, "This Spec Owns" — recording logic lives entirely in the scheduler driver). + +## Requirements Traceability + +| Requirement | Summary | Components | Interfaces | Flows | +|---|---|---|---|---| +| 1.1 | Core loop calls `next_job()` only, no ordering logic | SchedulerCore, JobPicker | `xce_job_picker_iface_t.next_job` | Normal-Mode Round | +| 1.2 | Picker hook categories (lifecycle/state/session/post-alloc/optional) | JobPicker | `xce_job_picker_iface_t` | — | +| 1.3 | Three-thread cbuf model | SchedulerCore | `wbuf`/`jbuf`/`decisions` cbufs | Normal-Mode Round | +| 1.4 | Picker/allocator/main-loop pseudocode contract | SchedulerCore, JobPicker | `core_push_events`/`core_go` | Normal-Mode Round | +| 1.5 | Cancel-safe, session-aware picker | JobPicker | `begin_session`/`end_session`/`next_job` | — | +| 1.6 | Replace hardcoded picker string | SchedulerCore | `plugin_register`/`key_find` | — | +| 1.7 | Preserve thread architecture, decision publish via callback | SchedulerCore | `on_decision` cb | Normal-Mode Round | +| 2.1 | `job_pick_plugin` on `xce_scheduler_t`, default `builtin_fcfs` | SchedulerCore | — | — | +| 2.2 | `job_pick_plugin` per-queue override | JobPicker | `xce_sched_queue_t.job_pick_plugin` | — | +| 2.3 | `project_name` on queue | SchedulerCore | `xce_sched_queue_t.project_name` | — | +| 2.4 | Config-driven picker name, no rebuild | SchedulerCore | scheduler config | — | +| 2.5 | Unregistered plugin fails loud | SchedulerCore | `plugin_register`/`key_find` | — | +| 2.6 | Picker contract documented (cancel-safe, session-aware) | JobPicker | `xce_job_picker_iface_t` | — | +| 3.1 | `builtin_fcfs` auto-registered as default | BuiltinFcfsPicker | plugin registration | — | +| 3.2 | Hook-by-hook FCFS implementation | BuiltinFcfsPicker | `xce_job_picker_iface_t` | — | +| 3.3 | Private `ordq_t` state, no core queries in `next_job` | BuiltinFcfsPicker | — | — | +| 3.4 | Ordering equivalence with today's `fcfs_elect_job` | BuiltinFcfsPicker | — | — | +| 3.5 | Dedicated picker thread retained, cancel points | BuiltinFcfsPicker, SchedulerCore | `next_job`/`cbuf_get(wbuf)` | Normal-Mode Round | +| 3.6 | `end_session` no-op, `pendJobs` carries over | BuiltinFcfsPicker | — | — | +| 4.1 | Job-picking order regression check | BuiltinFcfsPicker | — | test_picker_fcfs.c | +| 4.2 | Existing scheduler unit tests pass unmodified | SchedulerCore | — | testsuite/ | +| 4.3 | HA soak — no job loss/divergence | SchedulerCore | — | `/xce-test-ha` | +| 5.1 | Mode-agnostic core, no mode flag | SchedulerCore | `xce_sched_core_init()` | — | +| 5.2 | Passive event-driven interface, zero I/O deps | SchedulerCore | `core_push_events`/`on_decision` | — | +| 5.3 | Virtualized clock, two-base+speed formula | SchedulerCore | `xce_core_now()` | — | +| 5.4 | Core event types (typed payloads) | SchedulerCore | `xce_core_event_t` | — | +| 5.5 | Decision output types (DISPATCH/PEND/PREEMPT) | SchedulerCore | `xce_decision_t` | — | +| 5.6 | Decision output callback | SchedulerCore, NormalDriver | `xce_core_decision_cb_t` | Normal-Mode Round | +| 5.7 | Deterministic guarantee + normal↔sim equivalence | SchedulerCore | `jbuf` cap=1, hash-iteration audit, PRNG seed | test_sim_equivalence.c | +| 5.8 | Driver pending-event queue | NormalDriver | `pend_updates` dataq | Normal-Mode Round | +| 5.9 | Controller integration (capnp translate, publish decisions) | NormalDriver | `xcecli_ctx_t` | Normal-Mode Round | +| 5.10 | Runtime recording control (`OPCode_schedAdmin`) | NormalDriver, SchedAdminOpcode, XceAdminSchedulerCLI | `SchedAdminRequest`/`Reply` | Admin Path | +| 5.11 | Periodic snapshot during recording | NormalDriver | `record_snapshot_interval` | — | +| 5.12 | Session response time metrics | SchedulerCore | `SessionSummary` | — | +| 5.13 | Simulation mode startup (load header+snapshot) | SimDriver | `.replay` load sequence | — | +| 5.14 | Discrete event-driven virtual clock | SimDriver | `--sim-speed` | — | +| 5.15 | `--sim-verify` deterministic replay verification | SimDriver | `decision_eq` | — | +| 5.16 | `--sim-alt-events` what-if branching | SimDriver | alt-events file | — | +| 5.17 | Simulation isolation (no controller deps) | SimDriver | `xce_sched_sim_driver_t` | — | +| 5.18 | `ReplaySession` as session boundary | SimDriver | event loop | — | +| 5.19 | `--sim-output` simulation record file | SimDriver | `.replay` writer | — | +| 5.20 | `.replay` capnp stream format | ReplayFormat | schema structs | — | +| 5.21 | Snapshot serialization / picker rebuild | SchedulerCore, ReplayFormat | `save_snapshot`/`load_snapshot` | — | +| 5.22 | Interactive REPL | SimDriver | command loop | — | +| 6.1 | Resreq immutability | SchedulerCore | — | — | +| 6.2 | Pend reason factory (`make_pendreason()`) | SchedulerCore | `lib/preason.c` | — | +| 6.3 | Pointer-identity fast path | SchedulerCore | — | — | +| 6.4 | Select expression memoization preserved | SchedulerCore | `select_eval_ctx_t` | — | +| 6.5 | Snapshot forward-compatible with object sharing | ReplayFormat | `SnapshotData` | — | + +## Components and Interfaces + +| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | +|-----------|--------------|--------|---------------|---------------------------|-----------| +| JobPicker (interface) | Scheduler Core | Defines the pluggable job-selection contract | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 | plugin.h (P0) | Service | +| BuiltinFcfsPicker | Scheduler Core | Default picker replicating today's FCFS order | 3.1-3.6, 4.1 | JobPicker (P0) | Service | +| SchedulerCore | Scheduler Core | Passive, mode-agnostic scheduling engine | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 | JobPicker (P0), allocation/preemption (P1, sibling specs) | Service, State | +| NormalDriver | Scheduler Driver | Controller-attached driver, recording control | 5.6, 5.8-5.11 | SchedulerCore (P0), `xcecli_ctx_t` (P0) | Service, Event | +| SimDriver | Scheduler Driver | Replay-attached driver, virtual clock, REPL | 5.13-5.19, 5.22 | SchedulerCore (P0), ReplayFormat (P0) | Service, State | +| ReplayFormat | Wire Protocol | `.replay` capnp schema, `SnapshotData` | 5.20, 5.21, 6.5 | `controller.capnp` (P0) | Batch | +| SchedAdminOpcode | Controller | Pass-through dispatch for `OPCode_schedAdmin` | 5.10 | `cluster.c` `myreq_handler` (P0) | API | +| XceAdminSchedulerCLI | Tooling | `scheduler record`/`scheduler dump` commands | 5.10 | `xcecommon.wire`/`discovery` (P0) | API | + +### Scheduler Core + +#### JobPicker + +| Field | Detail | +|-------|--------| +| Intent | Contract every job-selection strategy implements; hides all ordering logic from the core loop | +| Requirements | 1.1, 1.2, 1.4, 1.5, 2.2, 2.6 | + +**Responsibilities & Constraints** +- Owns iteration order over pending jobs; the core never inspects queue/job internals for ordering. +- Cancel-safe (`pthread_cancel` points in `next_job()`) and session-scoped (`begin_session` resets iteration state). +- State-change hooks run synchronously in the driver thread during `core_push_events()`; session hooks run in the picker thread; post-allocation hooks run in the allocator thread (Req 1 AC 3). + +**Dependencies** +- Inbound: SchedulerCore — invokes every hook (P0) +- Outbound: none — picker implementations must not call back into core internals outside the hook parameters they're given + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] + +##### Service Interface +```c +typedef struct { + int (*init)(void* picker, xce_sched_core_t* core); + void (*fini)(void* picker); + int (*begin_session)(void* picker); + xce_sched_job_t* (*next_job)(void* picker); + int (*end_session)(void* picker); + int (*on_dispatch)(void* picker, uint64_t jobid); + int (*on_pend)(void* picker, uint64_t jobid); + int (*on_queue_update)(void* picker, xce_sched_queue_t* queue); + int (*on_job_push)(void* picker, xce_sched_job_t* job); + int (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state); + int (*on_job_update)(void* picker, xce_sched_job_t* job); + int (*on_config)(void* picker, xce_conf4sched_t* config); /* MAY be NULL */ + int (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); /* MAY be NULL */ +} xce_job_picker_iface_t; +``` +- Preconditions: `init()` called exactly once before any other hook; `begin_session()` precedes every `next_job()` sequence. +- Postconditions: `next_job()` returns NULL exactly when no ready job remains for the current session. +- Invariants: hook implementations never block indefinitely (picker thread holds no lock the allocator needs); `next_job()` never allocates a job the same session has already returned. + +**Implementation Notes** +- Integration: registered via `plugin_register`/`key_find` under the picker plugin category (new `plugin.c` registration table entry). +- Validation: `test_picker_fcfs.c` feeds hooks directly and asserts `next_job()` order — no live core needed (Req 3 AC 3's testability rationale). +- Risks: a picker implementation that queries core state directly during `next_job()` (bypassing the hook-fed private state pattern) would still compile but silently reintroduce lock contention — not enforceable at the interface level, called out for code review. + +#### BuiltinFcfsPicker + +| Field | Detail | +|-------|--------| +| Intent | Default `xce_job_picker_iface_t` implementation; replicates today's `fcfs_elect_job` ordering exactly through the new interface | +| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1 | + +**Responsibilities & Constraints** +- Maintains private state only — `ordq_t queues` (ordered by `queue->priority desc`) and, per queue, an `ordq_t pendJobs` (ordered by `priority desc, queue_time asc`). Never queries `sched->queues`/`sched->jobs` during `next_job()` (Req 3 AC 3). +- Every hook's behavior is fully specified in requirements.md Req 3 AC 2's hook-by-hook table — not repeated here; this block covers integration and testability, not the algorithm. +- Auto-registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` at scheduler startup (Req 3 AC 1). +- `end_session()` is a no-op; undispatched jobs carry over in `pendJobs` to the next round unchanged (Req 3 AC 6). + +**Dependencies** +- Inbound: SchedulerCore — invokes every `xce_job_picker_iface_t` hook, same as any picker (P0) +- Outbound: none — implements the interface, depends on nothing beyond it + +**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ] + +##### Service Interface +Implements `xce_job_picker_iface_t` (struct shown under JobPicker, above) with the `fcfs_plugin_data_t` private state described in Responsibilities. No additional public surface beyond the interface itself. +- Preconditions/Postconditions/Invariants: inherits JobPicker's contract in full; adds no relaxations or extensions. + +**Implementation Notes** +- Integration, as-built: implemented in a **new** file, `plugins/builtin_fcfs.c`, registered as `"builtin_fcfs"` and set as the default `job_pick_plugin` (`schedobj.c`) — NOT a rewrite of `plugins/fcfs.c` in place, as originally planned above. `fcfs.c` was deliberately left untouched (still `job_elect_fn`-only, unregistered as default) because task 1.2's `job_elect_fn` compat wrapper needs a real legacy-style plugin to exercise; `fcfs.c` fills that role, and `scheduler.c`'s `sched_job_picker()` dispatch (`job_elect_fn != NULL` vs `NULL`) depends on both variants existing side by side. Functionally equivalent to the original plan (one default picker implementing the new interface, ordering-equivalent to legacy `fcfs_elect_job`); only the file-layout mechanics differ. +- Validation: `test_picker_fcfs.c` is this component's primary test — feeds `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly and asserts `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order, matching today's `fcfs_elect_job` (Req 4.1's regression check). +- Risks: ordering-equivalence regression is the main risk (any drift from today's `fcfs_elect_job` output is a behavior change operators didn't opt into) — mitigated by `test_picker_fcfs.c` asserting against the documented ordering rule directly, plus the HA soak in Req 4.3. + +#### SchedulerCore + +| Field | Detail | +|-------|--------| +| Intent | Owns all scheduling state; drives the picker/allocator threads; exposes the driver↔core protocol | +| Requirements | 1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4 | + +**Responsibilities & Constraints** +- Zero direct dependency on controller communication, capnp, network I/O, filesystem, or wall-clock time (Req 5 AC 2) — enforced by the "Allowed Dependencies" link list, not just convention. +- `jbuf` cbuf capacity is hardcoded to 1 inside `xce_sched_core_init()` — never a parameter (Req 5 AC 7, pinned against the existing `njobs` FIXME in `scheduler.c:384`). +- All internal time via `xce_core_now()`; no direct `time()`/`clock_gettime()`/`gettimeofday()` calls anywhere in `core.c`. + +**Dependencies** +- Inbound: NormalDriver, SimDriver — both call `core_push_events`/`core_go`, register `on_decision` (P0) +- Outbound: JobPicker (P0); `try_allocate()`/`can_preempt()` from sibling specs (P1 — signature-stable dependency, no behavior assumption) +- External: none + +**Contracts**: Service [x] / State [x] + +##### Service Interface +```c +int xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed); +int xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents); +int xce_core_go(xce_sched_core_t*); +int xce_core_end_session(xce_sched_core_t*); +uint64_t xce_core_now(xce_sched_core_t*); +int xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out); +int xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in); +void xce_sched_core_fini(xce_sched_core_t*); +``` +- Preconditions: `core_push_events()` only valid when no session is active (`session_active == 0`); `core_go()` returns `-EINVAL` if a session is already active. `xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()` are likewise only valid when `session_active == 0` — both return `-EBUSY` if called mid-session, rather than silently capturing or restoring torn state. Every caller in this design (record start, periodic checkpoint, simulation startup) already invokes them between sessions by construction; the `-EBUSY` return is a defensive contract check, not a path any correct caller should hit. +- Postconditions: after `on_decision` delivers the end-of-session `NULL` marker, `session_active` is guaranteed to transition to 0 on the subsequent `core_end_session()` call. +- Invariants: given identical `(config, event sequence)`, `core_push_events`+`core_go` produce an identical decision sequence (Req 5 AC 7) — the whole point of the design. + +##### State Management +- State model: scheduling state (jobs/queues/workers/resources/resvals/users/projects) held in `kvmgr`s inside `xce_sched_core_t`, mirroring today's `xce_scheduler_t` layout minus `apictx`/`cluster_cfg`. +- Persistence & consistency: no persistence inside the core itself — `save_snapshot()`/`load_snapshot()` are the only serialization boundary, and only the driver decides when to invoke them. +- Concurrency strategy: three-thread cbuf pipeline unchanged from today; clock atoms (`virtual_clock_base`, `real_clock_base`, `speed_num`, `speed_den`) are lock-free `_Atomic uint64_t`, updated by the driver as a group before each event dispatch. + +**Implementation Notes** +- Integration: `core_push_events`/`core_go`/`on_decision` are the only surface either driver touches — no driver reaches into `xce_sched_core_t` internals. +- Validation: `test_core_events.c` (contract-level), `test_sim_equivalence.c` (record→replay→diff, Req 5 AC 7.1 — the spec's central regression test). +- Risks: the `kvmgr` iteration-order audit (Req 5 AC 7, "Stabilized hash table iteration") is a Phase 2 deliverable with real but bounded scope — jobs/queues/users/projects only (per `research.md` and requirements.md's explicit Phase 2 task). + +### Scheduler Driver + +#### NormalDriver + +| Field | Detail | +|-------|--------| +| Intent | Connects the core to a live controller; owns recording control | +| Requirements | 5.6, 5.8, 5.9, 5.10, 5.11 | + +**Responsibilities & Constraints** +- Translates capnp push/notify into `xce_core_event_t`, buffers between sessions in `pend_updates` (existing dataq swap mechanism). +- Publishes each decision immediately via `xcecli_jobdispatch`/`xcecli_set_pend_reason`/`xcecli_preempt` as it drains from `decisions` cbuf — not batched at session end. +- Recording is boolean-gated: zero file I/O, zero extra allocation when `recording == false` (Req 5 AC 10, Performance). +- Validates `recordFile` before ever calling `fopen()`: reject `/` or `..`, resolve under `record_dir`, confirm containment via `realpath()` (Req 5 AC 10, "Record file path validation" — this design inherits, not re-derives, that requirement). +- `record_dir` is provisioned exactly one way: a `--record-dir DIR` scheduler startup CLI flag (default `/var/log/xce/`), parsed alongside the existing normal-mode arguments in `main.c` — never a capnp field on any request path (`SchedAdminRequest` included), so nothing reachable over the wire can redirect where recordings are written. Matches the existing `--sim-*` flag family's provisioning style (CLI-only, not config-pushed). + +**Dependencies** +- Inbound: `main.c` (process entry point) (P0) +- Outbound: SchedulerCore (P0), `xcecli_ctx_t` (P0) +- External: `SchedAdminOpcode` dispatch (P1 — receives forwarded requests, not a caller relationship) + +**Contracts**: Service [x] / Event [x] + +##### Service Interface +```c +typedef struct { + xce_sched_core_t core; + xcecli_ctx_t apictx; + xce_cluster_config_t cluster_cfg; + dataq_t* pend_updates[2]; + char* record_dir; /* from --record-dir CLI flag at startup; default "/var/log/xce/"; + never settable via any request-path field */ + char* record_file_path; + FILE* record_file; + bool recording; + uint64_t record_event_count; + uint64_t record_snapshot_interval; + uint64_t record_start_ts; +} xce_sched_driver_t; + +int sched_driver_run(xce_sched_driver_t*, xce_conf4sched_t*, xce_cluster_config_t*, void* handler); +int sched_driver_record_start(xce_sched_driver_t*, const char* record_file /* bare filename or NULL */); +int sched_driver_record_stop(xce_sched_driver_t*); +int sched_driver_record_status(xce_sched_driver_t*, SchedAdminReply* out); +``` +- Preconditions: `record_start` fails with `-EALREADY`-equivalent if already recording; `record_stop` fails if not recording. +- Postconditions: on `record_start` success, `ReplayHeader` + initial `SnapshotEntry` are durably on disk before the reply is sent. +- Invariants: `recording == true` iff `record_file != NULL`. + +##### Event Contract +- Subscribed events: capnp push/notify from the controller (existing handler.c path, unchanged). +- Published events: decisions to the controller via `xcecli_ctx_t`; `SchedAdminReply` back through the forwarding path on recording-control requests. +- Ordering / delivery guarantees: decisions publish in allocator output order; recording writes events/decisions in the same order they cross the driver thread. + +**Implementation Notes** +- Integration: `main.c`'s mode-selection branch calls `sched_driver_run()` when `--sim` is absent. +- Validation: recording lifecycle tested against a real filesystem (temp dir) — start/stop/status round trip, path-validation rejection cases, disk-full simulated via a read-only mount point. +- Risks: `OPCode_schedAdmin` ships with the same primary/standby gate as `OPCode_upgradeConfig` and no additional RBAC/auth check — a deliberate scope decision (project owner, 2026-08-02), not an oversight. Strengthening authentication for this opcode (and `config push`, if it also lacks one) is deferred to a follow-up spec. See Open Questions/Risks. + +#### SimDriver + +| Field | Detail | +|-------|--------| +| Intent | Connects the core to a `.replay` file; owns virtual clock, verification, and the REPL | +| Requirements | 5.13-5.19, 5.22 | + +**Responsibilities & Constraints** +- Contains no `xcecli_ctx_t`, `xce_cluster_config_t`, capnp socket, or controller RPC of any kind (Req 5 AC 17) — enforced by the struct definition itself having no such fields. +- Discrete virtual clock: jumps directly to the next event's timestamp, never a fixed tick (Req 5 AC 14). +- REPL is a single `fgets` loop in the same thread as the simulation loop — no REPL thread (Req 5 AC 22, Implementation note). + +**Dependencies** +- Inbound: `main.c` (P0, when `--sim` is present) +- Outbound: SchedulerCore (P0), ReplayFormat (P0) +- External: none + +**Contracts**: Service [x] / State [x] + +##### Service Interface +```c +typedef struct { + xce_sched_core_t core; + FILE *replay_file, *alt_events_file, *record_file; + uint64_t sim_speed_num, sim_speed_den; + bool verify; + bool interactive; +} xce_sched_sim_driver_t; + +int sched_sim_driver_run(xce_sched_sim_driver_t*, /* CLI arguments struct */ const void* args); +``` +- Preconditions: replay file's second message must be a `SnapshotEntry` (rejected otherwise, Req 5 AC 20). +- Postconditions: with `--sim-output`, every session cycle appends exactly one `ReplaySession` to the output file before the next `ReplayEvent` batch begins. +- Invariants: REPL `step`/`continue` to completion produces byte-identical output to batch `--sim` (Req 5 AC 22, "Deterministic replay is preserved"). + +##### State Management +- State model: same `xce_sched_core_t` as normal mode, loaded via `load_snapshot()` instead of live events. +- Persistence & consistency: `--sim-output` writes a new independent `.replay` file; never mutates the input file. +- Concurrency strategy: single-threaded driver loop around the same core thread pair; REPL blocks the driver loop intentionally (no wall-clock deadline in simulation). + +**Implementation Notes** +- Integration: `main.c`'s mode-selection branch calls `sched_sim_driver_run()` when `--sim` is present. +- Validation: `test_sim_equivalence.c` drives this component directly (record via NormalDriver's test harness, replay via this one, diff). +- Risks: none beyond what's already captured under SchedulerCore (the determinism guarantee is a core property; SimDriver only needs to not violate it, e.g. by never bypassing `core_push_events`/`core_go`). + +### Wire Protocol + +#### ReplayFormat + +| Field | Detail | +|-------|--------| +| Intent | `.replay` capnp stream schema and `SnapshotData` | +| Requirements | 5.20, 5.21, 6.5 | + +**Responsibilities & Constraints** +- Additive-only changes to `controller.capnp` — reuses existing structs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) wherever they already exist (verified in `research.md`); only defines new structs where none exist. +- `SnapshotData` field shapes track the *actual current* C struct field sets (e.g. `xce_sched_user_t` is `{name}` only today, `xce_sched_project_t` is `{name}` only today) — the schema does not invent fields the C structs don't have yet. + +**Dependencies** +- Inbound: SchedulerCore (`save_snapshot`/`load_snapshot`), NormalDriver (recording writer), SimDriver (reader/writer) (all P0) +- Outbound: none +- External: none + +**Contracts**: Batch [x] + +##### Batch / Job Contract +- Trigger: `record start` (normal mode) or `--sim-output` (simulation mode) write a `.replay` stream incrementally; `--sim`/`xceadmin scheduler dump` read one. +- Input / validation: first message must be `ReplayHeader`; second must be `SnapshotEntry` (Req 5 AC 20) — both enforced at read time, not just documented. +- Output / destination: append-only file; `record stop` / clean simulation completion writes `ReplayEvent{endSession}`. +- Idempotency & recovery: a write failure mid-message truncates cleanly to the last complete message (Req 5 AC 10, error handling) — no partial-message corruption possible because each message is a full capnp frame written atomically at the `fwrite` level. + +**New capnp structs** (additive to `controller.capnp`; `ReplayHeader`/`ReplayEvent`/`ReplaySession`/`ReplayDecision`/`SessionSummary` wrapper structs already fully specified in requirements.md Req 5 AC 20 — not repeated here): + +```capnp +# New event payloads (JOBS_PUSH/WORKERS_PUSH/etc reuse existing Conf4Sched, +# SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages — +# verified present in controller.capnp, research.md). + +struct JobStateChangeEvent $C.mapname("job_state_change_event_t") { + jobid @0: UInt64; + oldState @1: UInt32; + newState @2: UInt32; + exitCode @3: Int32; +} + +struct JobUpdateEvent $C.mapname("job_update_event_t") { + jobid @0: UInt64; + changedFields @1: UInt32; # bitmask: JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ + job @2: Job; # full updated record; reuses existing Job struct +} + +# New decision payloads for ReplayDecision.body (Req 5 AC 5). +# DISPATCH/PEND reuse existing JobAlloc/JobPendReason; PREEMPT is new +# (wire shape owned here per Boundary Commitments; victim-selection +# semantics owned by scheduler-preemption — see research.md). +# +# AC 5's field list for DISPATCH is "jobid, workerid, allocations[]" +# (singular workerid). JobAlloc already generalizes this correctly for +# multi-host jobs: JobAlloc.jobid covers jobid, JobAlloc.allocs is +# List(HostAlloc) with each HostAlloc.host as that allocation's worker +# id and HostAlloc.allocs: List(ResAlloc) as its resource/amount pairs +# -- verified against controller.capnp, no new fields needed. jobid +# lives ONLY in JobAlloc.jobid (not duplicated at DispatchDecision +# level) to avoid two unsynchronized sources of truth for the same job. + +struct DispatchDecision $C.mapname("sched_dispatch_decision_t") { + alloc @0: JobAlloc; +} + +struct PendDecision $C.mapname("sched_pend_decision_t") { + jobid @0: UInt64; + reason @1: JobPendReason; +} + +struct PreemptVictim $C.mapname("preempt_victim_t") { + jobid @0: UInt64; + workerid @1: Text; +} + +struct PreemptDecision $C.mapname("sched_preempt_decision_t") { + jobid @0: UInt64; + victims @1: List(PreemptVictim); + reason @2: JobPendReason; +} + +# SnapshotData — full core-state checklist from requirements.md Req 5 AC 21, +# given a concrete wire shape. Reuses existing List(...) payload structs +# for domain objects already carried elsewhere on the wire; the two +# scheduler-internal-only caches (shared-resource tracking, unsatisfiable- +# resreq cache) and session/PRNG state have no existing external contract, +# so they are versioned opaque blobs the core alone reads/writes -- no +# consumer outside save_snapshot()/load_snapshot() ever inspects their +# internal shape (WHAT not HOW: the boundary that matters is "round-trips +# through the core, opaque to everyone else"). + +struct SnapshotData $C.mapname("sched_snapshot_data_t") { + config @0: Conf4Sched; + jobs @1: SchedJobData; + queues @2: List(QueueConf); + workers @3: WorkerData; + resources @4: ResourceData; + resvals @5: ResValList; + users @6: List(SchedUser); + projects @7: List(SchedProject); + prngSeed @8: UInt64; + internalState @9: Data; # shared-resource tracking + unsatisfiable-resreq + # cache + session/iterator state, core-opaque + internalVersion @10: UInt32; # format tag for internalState, so a future + # incompatible internal layout fails loud + # instead of misinterpreting old bytes +} + +struct SchedUser $C.mapname("xce_sched_user_t") { + name @0: Text; # matches xce_sched_user_t today (name only) -- + # extend in lockstep if xce_sched_user_t gains fields +} + +struct SchedProject $C.mapname("xce_sched_project_t") { + name @0: Text; + priority @1: Int32; +} +``` + +**Implementation Notes** +- Integration: `save_snapshot()`/`load_snapshot()` are the only functions that construct/consume `SnapshotData`; no other component touches `internalState`'s bytes. +- Validation: as-built, round-trip coverage for every new struct lives in `src/controller/testsuite/test_e2e_capnp.c` (task 7.5), not a dedicated `test_replay_format.c` — this repo's existing convention is one shared capnp round-trip test file per schema, and this spec's new structs were added to it rather than starting a second one; `internalVersion` mismatch is a dedicated negative test within it. +- Risks: `internalState`'s opaque-blob design trades wire-level inspectability for implementation flexibility — acceptable because AC 21 only requires the *core* to round-trip correctly, never that `xceadmin scheduler dump` render its internals meaningfully (dump can print `internalState`'s byte length and `internalVersion` only). + +### Controller + +#### SchedAdminOpcode + +| Field | Detail | +|-------|--------| +| Intent | Pass-through dispatch for `OPCode_schedAdmin` in the controller's request handler | +| Requirements | 5.10 | + +**Responsibilities & Constraints** +- Structurally mirrors the existing `OPCode_upgradeConfig` case (`cluster.c:670-705`) for the primary-gate/delegate/explicit-reply shape: primary-only gate via `ha_is_primary()`, explicit reply on every path, 5s timeout. +- Owns zero recording state — every field of `SchedAdminRequest`/`Reply` passes through untouched. +- **Blocking-wait scope decision (accepted, not hardened)**: unlike `OPCode_upgradeConfig` (in-process commit-barrier wait, already `COMMIT_WAIT_RESERVED`-admitted), this case blocks its `vtask` handler thread for up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock`. Reviewed against `ha-async-io.md`'s vtaskmgr admission rule and accepted as a boundary case (project owner, 2026-08-02): `xceadmin scheduler` commands are cluster-administrator-triggered, low-frequency, human-paced — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism (`COMMIT_WAIT_RESERVED`-style) is added for this opcode. Revisit if usage patterns change (e.g. automated/scripted recording control at scale). + +**Dependencies** +- Inbound: XceAdminSchedulerCLI, via TLS+capnp (P0) +- Outbound: NormalDriver (P0, forwards the request) +- External: none + +**Contracts**: API [x] + +##### API Contract +| Method | Endpoint (opcode) | Request | Response | Errors | +|--------|--------------------|---------|----------|--------| +| Request/Reply | `OPCode_schedAdmin` | `SchedAdminRequest` | `SchedAdminReply` | `-EAGAIN` (not primary), `"scheduler not connected"`, `"request timeout"`, `"invalid record file name"`, `"already recording"`, `"not recording"` (full table: requirements.md Req 5 AC 10) | + +**Implementation Notes** +- Integration: one new `case` in `myreq_handler`'s switch (`cluster.c`); no new file. +- Validation: `test_sched_admin_dispatch.c` (`src/controller/testsuite/`; named differently from this plan's original `test_sched_admin.c`, same coverage) asserting the reply-on-every-path property (standby rejection, disconnected-scheduler rejection, timeout) — mirrors existing `OPCode_upgradeConfig` test coverage style. +- Risks: no RBAC/auth check is visible for the comparable `OPCode_upgradeConfig` case in `cluster.c` (see `research.md`), and `OPCode_schedAdmin` inherits that same gap by deliberate decision — auth hardening for this opcode is deferred to a follow-up spec, not silently accepted as permanent. + +### Tooling + +#### XceAdminSchedulerCLI + +| Field | Detail | +|-------|--------| +| Intent | `xceadmin scheduler record start/stop/status` and `xceadmin scheduler dump ` | +| Requirements | 5.10 | + +**Responsibilities & Constraints** +- `cli.py` stays a thin `click` layer; all logic lives in the new `scheduler.py` module, mirroring `push.py`'s split. +- `record_start/stop/status()` go through `xceadmin.discovery.follow_redirects` + `decode_reply` (request/reply opcode, needs redirect-following, unlike `probe_config()`). +- `dump()` has no network dependency — decodes a local `.replay` file only. + +**Dependencies** +- Inbound: operator CLI invocation (P0) +- Outbound: `xcecommon.wire`/`schema`, `xceadmin.discovery` (P0); SchedAdminOpcode via the wire (P0, for record/status only) +- External: none + +**Contracts**: API [x] + +##### API Contract +| Method | Endpoint | Request | Response | Errors | +|--------|----------|---------|----------|--------| +| CLI | `scheduler record start [FILE]` | `SchedAdminRequest{recordStart}` | `SchedAdminReply` | `XceAdminError` subclasses (connection, redirect, decode) | +| CLI | `scheduler record stop` | `SchedAdminRequest{recordStop}` | `SchedAdminReply` | same | +| CLI | `scheduler record status` | `SchedAdminRequest{recordStatus}` | `SchedAdminReply` | same | +| CLI | `scheduler dump ` | (local file) | formatted text to stdout | `SchemaNotFoundError`, file-not-found | + +**Implementation Notes** +- Integration: `cli.py` imports `record_start`/`record_stop`/`record_status`/`dump` from `scheduler.py`, same import shape as today's `build_cluster_config`/`push_config`. +- Validation: `test_scheduler.py` — mock-TLS-server tests for the three record commands (mirrors `test_config_push_mock.py`), plus golden-file tests for `dump()` against a small fixture `.replay` file. +- Risks: none beyond ReplayFormat's (this component only formats/transmits, never interprets scheduling semantics). + +## Data Models + +Covered in full under **Components and Interfaces → ReplayFormat** above (new capnp structs) — not duplicated here per the dedup guidance in design-principles.md. `SnapshotData`'s reuse of existing `Conf4Sched`/`SchedJobData`/`WorkerData`/`ResourceData`/`ResValList`/`QueueConf` is the only cross-cutting data-model decision; see `research.md`, "Reusable capnp payload types," for the verification trail. + +## Error Handling + +### Error Strategy +Fail-loud, no silent fallback — consistent with the controller's existing "fail loud, stay alive" design invariant (`src/controller/CLAUDE.md`), applied to the scheduler process at the scope this feature touches. + +### Error Categories and Responses +- **Configuration errors** (unregistered picker plugin, Req 2 AC 5): scheduler startup fails with a clear error — no silent fallback to `builtin_fcfs`. +- **Recording errors** (disk full, permission error, invalid `recordFile`): logged, recording state cleanly torn down (`recording=false`, file closed, `record_file_path` freed), scheduling continues unaffected (Req 5 AC 10, error handling) — recording failure is never fatal to the scheduler process. +- **Simulation errors** (missing/corrupt replay file, missing initial `SnapshotEntry`): simulation driver fails at startup with a clear message — no partial/best-effort simulation. +- **Admin-path errors** (`OPCode_schedAdmin`): every failure mode gets an explicit `{ok=false, error=...}` reply — never a dropped request (mirrors the "no silent drop" precedent already established for replication-class messages, `src/controller/CLAUDE.md`'s protocol checklist). + +### Monitoring +No new metrics infrastructure — `SessionSummary` (Req 5 AC 12) provides session-duration diagnostics through the existing `.replay` recording path; no separate telemetry system is introduced. + +## Testing Strategy + +- **Unit — Picker contract** (`test_picker_fcfs.c`): feed `on_queue_update`/`on_job_push`/`on_job_state_change` hooks directly, assert `next_job()` returns jobs in `(queue priority desc, job priority desc, queue_time asc)` order — verifies Req 3.2-3.4 without a live core. +- **Unit — Core event contract** (`test_core_events.c`): `core_push_events()` followed by `core_go()` with a stub picker asserts state-change hooks fire exactly once per event, synchronously, before `core_go()` returns — verifies Req 5.1-5.8. +- **Unit — Clock virtualization** (`test_clock.c`): `xce_core_now()` against the four mode rows in Req 5 AC 3's table (normal, discrete, real-time, fast/slow) — asserts the two-base+speed formula directly, no wall-clock flakiness (fixed `real_clock_base`/mocked `clock_gettime` delta). +- **Unit — Replay format round-trip** (as-built: `src/controller/testsuite/test_e2e_capnp.c`, not a dedicated `test_replay_format.c` — added to this repo's existing shared capnp round-trip test file instead): encode/decode every new capnp struct (`JobStateChangeEvent`, `JobUpdateEvent`, `DispatchDecision`, `PendDecision`, `PreemptDecision`, `SnapshotData`) — field-for-field equality after round-trip. +- **Integration — Deterministic replay** (`test_sim_equivalence.c`, Req 5.7/5.7.1): record a synthetic scheduling session through `NormalDriver`'s test harness → replay through `SimDriver` → assert `ReplayDecision` sequences match exactly. This is the design's highest-value test — it directly exercises the central invariant. +- **Integration — Recording lifecycle** (driver-level, Req 5.10-5.11): `record start` → submit synthetic events → `record stop` → verify file structure (`ReplayHeader`, `SnapshotEntry`, alternating `ReplayEvent`/`ReplaySession`, periodic checkpoints at the configured interval, final `endSession`). +- **Integration — `OPCode_schedAdmin` dispatch** (`test_sched_admin_dispatch.c`, controller-level; named differently from this plan's original `test_sched_admin.c`): standby rejection, disconnected-scheduler rejection, timeout, and successful forward/reply round trip — mirrors existing `OPCode_upgradeConfig` test coverage. +- **Integration — xceadmin scheduler CLI** (`test_scheduler.py`): mock-TLS-server tests for `record start/stop/status` (redirect-following, reply decoding) plus golden-file tests for `dump()` against a fixture `.replay` file. +- **E2E — HA soak regression** (Req 4.3, Req 5 per-phase verification): `/xce-test-ha` sequential and chaotic rounds after each phase, asserting zero new job loss/divergence attributable to this feature — required before each phase merges, not a one-time gate at the end. +- **E2E — REPL equivalence** (Req 5.22): `--interactive` with `step`/`continue` to completion produces the same decision sequence as batch `--sim` on the same replay file. + +## Security Considerations + +_Feature-specific decisions only; baseline RBAC/auth practices are in `.kiro/steering/security.md`._ + +- **`recordFile` path validation** is already fully specified at the requirements level (Req 5 AC 10, "Record file path validation" — reject `/`/`..`, resolve under configured `record_dir`, `realpath()` containment check before `fopen()`). This design's `NormalDriver` and `SchedAdminOpcode` components inherit that requirement directly. `record_dir` itself is provisioned only via a `--record-dir` scheduler startup CLI flag — never from `SchedAdminRequest` or any other request-path field — so the validation boundary can't be redirected by a caller. +- **`OPCode_schedAdmin` ships with no RBAC/auth check beyond the primary/standby gate — accepted scope decision, deferred hardening.** The comparable `OPCode_upgradeConfig` case has no visible RBAC/auth check in `cluster.c` either (see `research.md`). Project owner decision (2026-08-02): authentication for this opcode is enhanced in a follow-up spec, not blocking this one. Phase 3 ships `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current (unauthenticated-beyond-primary-gate) posture — this is a known, accepted gap, not a silent one. +- **`.replay` files may contain sensitive cluster state** (full job/user/resource snapshots) — file permissions on `record_dir` are an operational concern (steering-level, not new to this feature) but worth calling out since this is the first feature to write full scheduler snapshots to disk in a human-decodable format (`xceadmin scheduler dump`). + +## Performance & Scalability + +_Feature-specific targets only; general practices are in steering docs._ + +- **Recording overhead**: zero file I/O, zero extra allocation when `recording == false` — a single boolean branch per session (Req 5 AC 10, explicit requirement, inherited here as a design constraint on `NormalDriver`). +- **Determinism vs throughput**: `jbuf` cbuf capacity is permanently 1 (rendezvous, not a buffer) — this bounds picker→allocator throughput to strictly sequential handoff. This is an intentional trade-off (determinism over pipelining) already made at the requirements level (Req 5 AC 7) and is not revisited here; any future throughput work must find a different mechanism that preserves the ordering guarantee (e.g. deterministic batching with an explicit order tag), not simply raise the cbuf capacity. +- **Simulation fast mode**: `speed_num=0` means the virtual clock never advances between events at wall-clock cost — `--sim` on a large `.replay` file runs at CPU-bound speed, not wall-clock speed. No specific throughput target is set here (none given in requirements); implementation should avoid introducing any per-event syscall beyond what decoding the capnp stream requires. +- **`OPCode_schedAdmin` vtask occupancy**: each request blocks its vtask handler thread up to 5s (cross-process wait on the scheduler's reply) while holding `c->sched.lock` — no admission reservation, accepted as a boundary case given expected low, human-paced request volume (see `SchedAdminOpcode` component, Design Review 2026-08-02). Not bounded or load-tested against concurrent bursts; out of scope unless usage patterns change. + +## Open Questions / Risks + +1. **`OPCode_schedAdmin` authorization mechanism** (P2 — accepted gap, deferred). Ships at parity with `OPCode_upgradeConfig`'s current posture (primary/standby gate only, no RBAC/auth check). Project owner decision (2026-08-02): auth hardening for this opcode is a follow-up spec, not a Phase 3 blocker. Tracked in `.kiro/steering/roadmap.md`'s existing `peer-authentication-hardening` backlog item (updated 2026-08-02 to cover `OPCode_schedAdmin` alongside `OP_upgradeConfig` — same `validate_host()` gate closes both). See Security Considerations and `research.md`. +2. **`PreemptDecision`'s wire shape is pre-committed ahead of `scheduler-preemption`'s own design** (P1). Flagged as a revalidation trigger in Boundary Commitments; `scheduler-preemption`'s design should explicitly confirm or extend this shape early, not discover it mid-implementation. +3. **`SnapshotData.internalState` opaque-blob scope** (P2). If a future spec needs `xceadmin scheduler dump` to render shared-resource-tracking or unsatisfiable-resreq-cache contents meaningfully (not just byte length), `internalState` will need to become a structured capnp field instead of opaque `Data` — deferred here because no current requirement asks for it (AC 20's dump description only requires human-readable output for what dump already needs to show, and internals were never asked to be inspectable). +4. **`xce_sched_core_t`'s `config`/`plugins`/`jobs`/`queues`/`workers`/`users`/`projects` field block is left as `var ...` in requirements.md's data-structure summary** (not fully typed) — Phase 2 implementation must finalize these as concrete `kvmgr`/pointer types when `core.c` is written; this design does not re-type them since requirements.md already commits to "moved from `xce_scheduler_t`" as the authoritative source of truth for their shapes. +5. **`OPCode_schedAdmin` blocks a vtask handler thread up to 5s per request with no admission reservation** (P2 — accepted boundary case, deferred). Project owner decision (2026-08-02): `xceadmin scheduler` commands are cluster-administrator-triggered and low-frequency, not expected to burst concurrently — no `COMMIT_WAIT_RESERVED`-style admission gate added. Revisit only if usage patterns become automated/high-frequency. See `SchedAdminOpcode` component and `research.md`. +6. **Req 5 AC 3's tlog virtual-clock logging integration is built but never activated** (P2 — found by `/kiro-validate-impl`, 2026-08-07, not yet fixed). `core_clock.c`'s `xce_core_register_tlog_format()`/`xce_core_set_active_for_logging()` are unit-tested (`test_clock.c`) but have zero call sites in `main.c`/`sim.c`/`driver.c` — task 4.5 deliberately deferred wiring them to "a later task" (its own header comment), and no later task picked it up. The correct activation site is `sim.c`'s `sched_sim_driver_start()` (after `sim_set_virtual_clock()` seeds `virtual_clock_base` to a real epoch value from the `.replay` file) — NOT `main.c`'s normal-mode path, where `virtual_clock_base=0` would make `xce_core_now()` return raw `CLOCK_MONOTONIC` boot-relative nanoseconds, making log timestamps worse, not better. Left unfixed pending a paired deactivation: `g_active_core_for_logging` is one process-wide pointer, `sched_sim_driver_run()` has no fini function (its own comment: "no sim-driver stop/fini function exists") and multiple early-return paths, and several test binaries construct more than one stack-allocated `xce_sched_sim_driver_t` per process — activating without deactivating risks a real ASAN stack-use-after-scope hazard. See tasks.md's Implementation Notes for the full trace. +7. **`plugins/fcfs.c` vs `plugins/builtin_fcfs.c` split was undocumented in this design until 2026-08-07** (P3 — documentation gap only, now fixed here). Found by `/kiro-validate-impl`: the file-structure and BuiltinFcfsPicker sections above originally said `fcfs.c` would be rewritten; the actual implementation added a new `builtin_fcfs.c` and deliberately left `fcfs.c` untouched (task 1.2's compat-wrapper exercise target). Functionally correct and confirmed live (`builtin_fcfs` is the registered default), just wasn't reflected here until this update. Tasks 2.1-2.6 in tasks.md still carry no `_Verified:` annotations (unlike every other task group) — worth backfilling for traceability. diff --git a/.kiro/specs/scheduler-core/research.md b/.kiro/specs/scheduler-core/research.md new file mode 100644 index 00000000..741e8472 --- /dev/null +++ b/.kiro/specs/scheduler-core/research.md @@ -0,0 +1,97 @@ +# Research & Design Decisions: scheduler-core + +## Summary +- **Feature**: `scheduler-core` +- **Discovery Scope**: Complex Integration (extension of an existing C daemon, new wire protocol, new cross-component opcode) +- **Key Findings**: + - `requirements.md` is unusually design-complete for a requirements doc — most C-level contracts (struct layouts, thread model, capnp wire framing) are already normative, not left to design. Design's real job here is: (1) the still-deferred `ReplayEvent`/`ReplayDecision` payload capnp schemas, (2) the concrete File Structure Plan, (3) closing two boundary tensions the requirements doc surfaces but doesn't resolve. + - Almost every payload type `ReplayEvent.body`'s union needs (`Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages`) **already exists verbatim** in `src/schema/controller.capnp` — the requirements author already picked names that match the live schema. Only `JobStateChangeEvent` and `JobUpdateEvent` are genuinely new. This is a Build-vs-Adopt win: reuse, don't reinvent. + - The scheduler component's real file layout (`src/scheduler/src/{decision.c,handler.c,main.c,plugin.c,scheduler.c,plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c},resource.c,schedobj.c}`) is substantially larger than `.wolf/anatomy.md`'s cached listing (which only knew `handler.c`, `main.c`, `scheduler.c`). File Structure Plan below is built from the live `find`, not the stale cache. + +## Research Log + +### Existing scheduler component layout +- **Context**: Need concrete file paths for the File Structure Plan; `structure.md` gives the pattern (`include/`, `src/`, `testsuite/`, plugin subdir), but not the current file set. +- **Sources Consulted**: `find src/scheduler -name '*.c' -o -name '*.h'`, `src/scheduler/testsuite/meson.build`. +- **Findings**: `include/`: `decision.h`, `plugin.h`, `sched/types.h`, `scheduler.h`. `src/`: `decision.c`, `handler.c`, `main.c`, `plugin.c`, `scheduler.c`, `resource.c`, `schedobj.c`, `plugins/{fcfs.c,limit.c,parjob_allocator.c,seqjob_allocator.c}`. `testsuite/`: `test.c` (runner harness), `test_handlers.c`, `test_job_array_expand.c`, `test_job_array_wire_e2e.c`, `test_job_update.c` — standalone TUs, several `#include ../src/*.c` directly (white-box pattern) rather than linking. +- **Implications**: New core/driver files (`xce_sched_core_t`, `xce_sched_driver_t`, `xce_sched_sim_driver_t`) are new `src/` files, not modifications of `scheduler.c` wholesale — `scheduler.c` keeps today's `xce_scheduler_t`-owning code during Phase 1, then Phase 2 extracts state into a new `core.c`. Test files for the new pieces follow the existing standalone-TU-with-`#include`, one-binary-per-unit convention already established in `testsuite/meson.build`. + +### Controller-side opcode dispatch location and pattern +- **Context**: AC 10 requires a new `OPCode_schedAdmin` case; need to confirm which file owns `myreq_handler` and the exact style of an existing opcode case for consistency. +- **Sources Consulted**: `grep -rln myreq_handler src/controller/*.c`, `cluster.c` around `OPCode_fetchfile` and `OPCode_upgradeConfig`. +- **Findings**: `myreq_handler` lives in `src/controller/cluster.c` (also referenced from `ha.c`/`ha_convergence.c`, but the dispatch switch itself is in `cluster.c`). The closest structural precedent is `OPCode_upgradeConfig` (`cluster.c:670-705`): gated by `ha_is_primary(&c->ha)` (reject standby with `-EAGAIN`, `RequestReply_data_nodata`), then delegates to a `handle_*()` function. **No RBAC/auth check is visible in that case block** — the comment trail (`cluster.c:672-693`, bug-fix 2026-07-25) discusses primary/standby gating exhaustively but never mentions an authentication check at this opcode. If `OPCode_upgradeConfig` has one, it happens earlier (TLS handshake / connection-accept time) or not at all at this layer. +- **Implications**: AC 10's claim "same RBAC role as config push" cannot be verified against a concrete gate in the code as it stands — carried into design.md as an **Open Question**, not asserted as a resolved fact. `OPCode_schedAdmin`'s case block should structurally mirror `OPCode_upgradeConfig` (primary-only gate, `handle_*()` delegation, explicit reply on every path) since that is the only comparable admin-opcode precedent in the file. + +### xceadmin CLI extension pattern +- **Context**: AC 10's `xceadmin scheduler record start/stop/status` and AC 20's `xceadmin scheduler dump` need a concrete module plan consistent with the existing `config push`/`config build` implementation. +- **Sources Consulted**: `src/xceadmin/src/xceadmin/{cli.py,push.py,build.py,discovery.py}`, `src/xceadmin/src/xcecommon/{wire.py,schema.py}`. +- **Findings**: `cli.py` is a thin `click` layer only (`@cli.group()` per resource, `@group.command()` per verb) — all business logic lives in per-resource modules (`push.py`, `build.py`) that a) load the schema once via `xcecommon.schema.load_schema()`, b) build/decode capnp messages, c) go through `xcecommon.wire.connect`/`send_request` for the socket, and d) for request/reply opcodes, route through `xceadmin.discovery.follow_redirects` + `xceadmin.discovery.decode_reply` (which unwraps the `Request{opcode: OPCode_reply, body: replymsg}` envelope every reply is wrapped in — a real bug fixed 2026-07-30, documented in `discovery.py`'s own docstring). Every raising path uses one exception hierarchy (`XceAdminError` subclasses), caught once in `cli.py`. +- **Implications**: A new `xceadmin/scheduler.py` module is the natural home for `record_start()`, `record_stop()`, `record_status()`, and `dump()` — same shape as `push_config()`/`probe_config()`, reusing `discovery.follow_redirects`/`decode_reply` (this is a request/reply opcode per AC 10, so unlike `probe_config()` it DOES need redirect-following). `dump()` differs from the other three: it operates on a local `.replay` file only, no network call — it decodes a capnp stream with `xcecommon.schema.load_schema()`'s module object and formats each message type to text; it belongs in the same module for CLI cohesion but has no wire dependency. + +### Reusable capnp payload types for ReplayEvent / SnapshotData +- **Context**: Requirements AC 4 explicitly defers the exact capnp schema for event payloads to design; AC 20's `ReplayEvent.body` union names `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `JobStateChangeEvent`, `JobUpdateEvent`, `QueueConf`, `SchedJobRusages` without defining the two `*Event` types. +- **Sources Consulted**: `grep -n '^struct ' src/schema/controller.capnp`. +- **Findings**: `Conf4Sched`, `SchedJobData`, `WorkerData`, `ResourceData`, `ResValList`, `QueueConf`, `SchedJobRusages` **all already exist** in `controller.capnp` with those exact names (the requirements author evidently cross-checked the schema before naming the union members). `JobStateChangeEvent` and `JobUpdateEvent` do not exist. The existing `Job` struct (`controller.capnp:288`) already carries `jobid` + `state`, and `JobAlloc`/`JobPendReason` exist for allocation/pend-reason payloads that the new `DispatchDecision`/`PendDecision` need. +- **Implications**: Design only needs to define 2 new event payload structs and 3 new decision payload structs (reusing `JobAlloc` and `JobPendReason` internally), not 9. See Data Models section of `design.md`. + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks / Limitations | Notes | +|--------|-------------|-----------|---------------------|-------| +| Event-sourced core with pluggable driver (selected) | Core holds all scheduling state, driven purely by typed events + a pull-based clock; two driver implementations (normal, simulation) translate their respective input sources to the same event stream | Normal↔sim equivalence becomes a structural property, not a discipline; picker/allocator threading is unchanged from today, minimizing regression surface | Requires strict discipline (no wall-clock calls, no controller calls) inside the core — a single violation breaks the central invariant silently until someone runs `--sim-verify` | Already fully specified in requirements.md; design.md's job is to make the boundary and remaining schema concrete, not to re-litigate the pattern | +| Mode-flagged single scheduler (rejected in requirements) | Keep `xce_scheduler_t` as-is, add `if (sim_mode)` branches for clock/IO | No new types, smallest diff | Directly violates the central invariant requirements establishes (normal/sim would share code paths only by discipline, not by construction) — rejected by requirements.md Req 5 AC 1, not re-opened here | — | + +## Design Decisions + +### Decision: New capnp structs for JOB_STATE_CHANGE / JOB_UPDATE events +- **Context**: AC 4 specifies the field set (jobid, old_state, new_state, exit_code for state-change; jobid, changed_fields, full job record for update) but no capnp shape exists. +- **Alternatives Considered**: + 1. Reuse `Job` alone and let the driver infer old/new state from two full records — rejected, doubles payload size for a two-field delta and loses `exit_code`. + 2. Define thin new structs that carry only the delta plus embed `Job` where a full record is genuinely needed — selected. +- **Selected Approach**: `JobStateChangeEvent{jobid, oldState, newState, exitCode}`; `JobUpdateEvent{jobid, changedFields, job: Job}` (embeds the existing `Job` struct, not a re-derived one). +- **Rationale**: Matches AC 4's field table exactly; embedding `Job` for the update case (where the driver already has to build a full record) avoids a second job-shape definition; the state-change case stays flat since AC 4 requires no other job fields. +- **Trade-offs**: `JobUpdateEvent` duplicates the full `Job` payload even for a single-field priority change — acceptable, matches AC 4's explicit field table (`job: xce_job_t*` full record), not a design deviation. +- **Follow-up**: None — field set is normative from requirements, only the capnp wrapper is new. + +### Decision: Decision payload structs (DispatchDecision / PendDecision / PreemptDecision) — and a boundary note on PreemptDecision +- **Context**: AC 5 fully specifies fields for all three decision types. Separately, the Boundary Context assigns "the preemption decision type" to the `scheduler-preemption` sibling spec. +- **Alternatives Considered**: + 1. Omit `PreemptDecision` entirely from this design, leave a placeholder in the `ReplayDecision.body` union for `scheduler-preemption` to add later — rejected, AC 5's decision output table is a Requirement 5 AC in *this* spec, and the `on_decision` callback signature / `ReplayDecision` union must compile today (Phase 2) before `scheduler-preemption` exists. + 2. Define `PreemptDecision`'s wire shape now per AC 5's field table, treat victim-selection *semantics* (not the wire shape) as what `scheduler-preemption` owns — selected. +- **Selected Approach**: `PreemptDecision{jobid, victims: List(PreemptVictim{jobid, workerid}), reason: JobPendReason}` defined in this spec's schema additions. +- **Rationale**: The wire struct is a passive data shape with no scheduling logic in it; nothing about defining it here binds `scheduler-preemption` to any victim-selection algorithm. Leaving the union member undefined would make `xce_decision_t`/`ReplayDecision` incomplete types this spec's own Phase 2 code needs to reference (`can_preempt()`'s caller, per requirements.md Req 1 AC 4's allocator pseudocode, already branches on a preempt outcome). +- **Trade-offs**: If `scheduler-preemption` later needs additional fields on `PreemptDecision`, that spec must extend this schema (a revalidation trigger, listed in design.md's Boundary Commitments) rather than owning it outright from scratch. +- **Follow-up**: Flag explicitly to whoever picks up `scheduler-preemption` that the wire shape is pre-committed here; the spec should confirm the field set is sufficient before implementation. + +### Decision: `OPCode_schedAdmin` authorization mechanism — deferred to a follow-up spec (project owner call, 2026-08-02) +- **Context**: AC 10 states "same RBAC role as config push," but the only comparable opcode (`OPCode_upgradeConfig`) has no visible authorization check in `cluster.c` — only a primary/standby gate. +- **Alternatives Considered**: + 1. Assert a specific mechanism (e.g. "add a `cluster:control` RBAC check") without verifying one exists for `config push` today — rejected, would document a fictitious parity claim. + 2. Block Phase 3 until the implementer traces `config push`'s real auth boundary and applies it here too — rejected by project owner: not worth gating this spec's delivery on. + 3. Ship `OPCode_schedAdmin` at parity with `OPCode_upgradeConfig`'s current posture (primary-gate only, no RBAC check), document the gap explicitly, and track auth hardening as separate future work — **selected**. +- **Selected Approach**: Option 3. `design.md`'s Open Questions/Risks and Security Considerations sections record this as an accepted, deliberate gap rather than an unresolved blocker. +- **Rationale**: `config push`'s current authorization boundary could not be located in the controller-side handler during discovery. Rather than block this spec on retrofitting auth for two opcodes at once, the project owner chose to scope that work separately. +- **Trade-offs**: `OPCode_schedAdmin` (and, unchanged, `OPCode_upgradeConfig`) remain gated only by primary/standby routing until the follow-up lands — acceptable short-term since both are already TLS-connected, admin-tooling-only surfaces (xceadmin), not general network-reachable endpoints. +- **Follow-up**: Add an "auth hardening for admin opcodes (`OPCode_upgradeConfig`, `OPCode_schedAdmin`)" item to the project roadmap (see `design.md` Open Questions/Risks #1). Track as a future spec, not a task inside `scheduler-core`. + +### Decision: `OPCode_schedAdmin`'s blocking cross-process wait — accepted as a boundary case, no admission reservation (project owner call, 2026-08-02) +- **Context**: Design review (2026-08-02) flagged that `myreq_handler` runs as a `vtask_t` handler (bounded 8-thread pool, `cluster.c:238`), and `OPCode_schedAdmin` blocks that thread up to 5s waiting on a cross-process reply from the scheduler daemon while holding `c->sched.lock` — unlike `OPCode_upgradeConfig`'s in-process, `COMMIT_WAIT_RESERVED`-admitted commit-barrier wait. `ha-async-io.md`'s vtaskmgr rule requires admission reservation for bounded blocking and forbids unbounded blocking on this pool; the `raft-drive-async` family fixed prior real violations of this exact rule (`config-push`'s busy-poll+join was one of them). +- **Alternatives Considered**: + 1. Add a `COMMIT_WAIT_RESERVED`-style admission reservation for `OPCode_schedAdmin` before Phase 3 ships — the design-review recommendation. + 2. Make the scheduler forward asynchronous (vtask completion callback instead of blocking wait) — larger structural change, deferred as unnecessary for current usage. + 3. Accept the blocking wait as a boundary case given expected request volume — **selected**. +- **Selected Approach**: Option 3. `xceadmin scheduler record`/`dump` commands are cluster-administrator-triggered, human-paced, low-frequency — not a request class expected to burst concurrently the way heartbeat/replication traffic does. No admission-reservation mechanism is added. +- **Rationale**: The `COMMIT_WAIT_RESERVED` precedent exists because commit-barrier waits are on the hot path of every job submission (high, uncontrolled frequency). Admin recording control is operator-initiated and rare by nature; the risk this pattern protects against in the commit-barrier case doesn't apply at comparable severity here. +- **Trade-offs**: If usage patterns change (e.g. automated/scripted recording control invoked at scale, or a monitoring tool polling `record status` frequently), this becomes a live risk without warning — no test or admission gate would catch it. +- **Follow-up**: If `scheduler record`/`dump` usage ever becomes automated or high-frequency, revisit with a `COMMIT_WAIT_RESERVED`-style admission reservation. + +## Risks & Mitigations +- **`OPCode_schedAdmin` ships without an RBAC/auth check beyond the primary/standby gate — accepted, not blocking.** Mitigation: documented explicitly in `design.md`; tracked as a future roadmap item (see Decision above), not a Phase 3 dependency. +- **`OPCode_schedAdmin` blocks a vtask thread for up to 5s per request with no admission reservation — accepted boundary case, not mitigated.** Rationale: low-frequency, human-triggered request class (see Decision above). Revisit only if usage patterns change. +- **`PreemptDecision` wire shape is pre-committed before `scheduler-preemption`'s own design exists** — mitigation: documented as a revalidation trigger; `scheduler-preemption` must review this shape early rather than discovering it mid-implementation. +- **`jbuf` capacity=1 determinism invariant depends on a `FIXME: make this configurable` comment never being resolved carelessly** — mitigation: already hard-pinned in requirements.md Req 5 AC 7 with an explicit Phase 2 task; design.md's File Structure Plan marks `xce_sched_core_init()` as the single place this is enforced. + +## References +- `src/scheduler/CLAUDE.md`, `.kiro/steering/{structure.md,tech.md,ha-async-io.md}` — component layout and controller HA invariants. +- `src/schema/controller.capnp` — canonical wire schema; source of truth for reused struct names. +- `src/xceadmin/src/xceadmin/push.py`, `discovery.py` — reference pattern for the new `scheduler.py` xceadmin module.