myxce/.kiro/specs/scheduler-core/design.md

Technical Design: scheduler-core

Overview

Purpose: This feature replaces the scheduler's hardcoded FCFS job-selection logic and its hard-wired coupling to the live controller with a pluggable picker interface and a passive, mode-agnostic scheduling core. The core is driven by either a normal-mode driver (live controller) or a simulation-mode driver (recorded .replay file), and both modes are required to produce byte-identical scheduling decisions given identical input — this is the spec's central, testable invariant.

Users: Cluster operators configure job-picking policy without a rebuild (Req 2) and gain a xceadmin scheduler record/dump workflow. Scheduler developers gain a --sim/--interactive simulation driver to debug and tune scheduling policy against recorded production traces without a live cluster (Req 5).

Impact: xce_scheduler_t (today: scheduling state + apictx + cluster_cfg in one struct, scheduler.c) splits into xce_sched_core_t (pure scheduling state, zero I/O) plus xce_sched_driver_t (normal, controller-attached) and xce_sched_sim_driver_t (simulation, replay-attached). The hardcoded "xce_sched_fcfs" string at scheduler.c:43 is replaced by a configurable, plugin-registered picker. A new controller opcode (OPCode_schedAdmin) and a new xceadmin command group (scheduler record/scheduler dump) are added as pass-through admin plumbing.

Goals

• Job-picking policy configurable per-scheduler and per-queue without recompilation, with builtin_fcfs shipping as the zero-behavior-change default (Req 1-4).
• Scheduling core with zero controller I/O, wall-clock, or filesystem dependencies, driven identically by a normal or simulation driver (Req 5 AC 1-8).
• Runtime recording of live scheduling sessions to a self-contained .replay file, and a simulation driver + interactive REPL that replays it deterministically (Req 5 AC 9-22).
• Normal↔sim equivalence as a verifiable regression test: record a live session, replay it, diff the decisions — zero divergence (Req 5 AC 7.1).

Non-Goals

• New job-picking policies beyond builtin_fcfs (interface only; sibling work).
• Allocation iterators / try_allocate() internals (scheduler-allocation).
• Preemption policy / victim selection (scheduler-preemption) — this design defines PreemptDecision's wire shape only (needed for Req 5 AC 5's decision table to compile), not preemption logic.
• Hierarchical project resource quotas (project-resource-ownership).
• REPL state mutation, reverse execution, or scripting (explicitly deferred in Req 5 AC 22).

Boundary Commitments

This Spec Owns

• xce_job_picker_iface_t and the builtin_fcfs implementation. As-built: a new src/scheduler/src/plugins/builtin_fcfs.c implements the interface; plugins/fcfs.c was deliberately left unchanged (still job_elect_fn-only) as the live exercise target for plugin.c's legacy compat wrapper (see Components — BuiltinFcfsPicker).
• xce_sched_core_t: all scheduling state, the picker/allocator thread pair, the driver↔core protocol (core_push_events, core_go, on_decision), and xce_core_now() clock virtualization.
• xce_sched_driver_t (normal-mode): controller event translation, decision publishing, and recording control (start/stop/status, periodic snapshot).
• xce_sched_sim_driver_t (simulation-mode): .replay file loading, discrete virtual clock, --sim-verify/--sim-alt-events/--sim-output, and the interactive REPL.
• The .replay capnp wire format (ReplayHeader, SnapshotEntry, ReplayEvent, ReplaySession, ReplayDecision, and their payload sub-structs) in src/schema/controller.capnp.
• The OPCode_schedAdmin request/reply pair and its pass-through dispatch case in the controller's myreq_handler (src/controller/cluster.c), plus its accompanying src/controller/testsuite/ coverage — this spec's only footprint outside src/scheduler/.
• xceadmin scheduler record start|stop|status and xceadmin scheduler dump <FILE> (new src/xceadmin/src/xceadmin/scheduler.py).
• job_pick_plugin (global + per-queue) and project_name (per-queue) fields; xce_sched_project_t scaffolding (name + priority only — no quota logic).

Out of Boundary

• try_allocate() and the allocation iterator (scheduler-allocation owns the implementation; this spec only calls it from the allocator thread, unchanged signature).
• can_preempt() and victim-selection semantics (scheduler-preemption owns the algorithm; this spec owns only PreemptDecision's wire shape, see Data Models).
• Hierarchical project quota, resource_quota/resource_used/parent_project on xce_sched_project_t (project-resource-ownership).
• Any controller state or persistence for recording (OPCode_schedAdmin is explicitly a stateless pass-through — the controller commits nothing, owns no WAL entry for it).
• REPL write-side mutation, semantic snapshot diff in xceadmin (both explicitly deferred, Req 5 AC 22).

Allowed Dependencies

• Existing plugin framework (plugin.h/plugin.c) — xce_job_picker_iface_t is a new plugin category registered through the same plugin_register/key_find mechanism, not a parallel system.
• Existing select-expression evaluator (resreq.h, select_eval_ctx_t) — used as-is by the allocator thread; the core does not re-implement or wrap it.
• libxce data-structure subset only inside the core: ordq, dataq, cbuf, xlist, zarray, kvmgr, rtti (Req 5 AC 3). The core must not link netio, delay, xceapi, or idgen.
• xcecli_ctx_t API (xcecli_jobdispatch, xcecli_set_pend_reason, xcecli_preempt) — normal-mode driver only, never referenced from xce_sched_core_t or xce_sched_sim_driver_t.
• xcecommon.wire/xcecommon.schema/xceadmin.discovery (Python) — the new xceadmin/scheduler.py module reuses these exactly as push.py does; it does not open its own socket or reload the schema.

Revalidation Triggers

• Any change to xce_job_picker_iface_t's hook signatures — every picker plugin (present and future) must be re-verified.
• Any change to jbuf cbuf capacity away from 1 — invalidates the determinism argument in Req 5 AC 7; scheduler-allocation and scheduler-preemption must both re-verify ordering assumptions.
• Any change to the .replay capnp schema's message set or ReplayDecision.body union — scheduler-preemption's PreemptDecision consumer and any external tooling reading .replay files must re-check compatibility.
• Any change to OPCode_schedAdmin's request/reply shape — xceadmin.scheduler module must be updated in lockstep (same repo, same PR).

Architecture

Existing Architecture Analysis

Today, xce_scheduler_t (src/scheduler/include/scheduler.h) owns scheduling state (jobs, queues, workers, plugins kvmgrs), controller I/O (apictx: xcecli_ctx_t, cluster_cfg: xce_cluster_config_t), and three threads (jobpicker, allocator, main) wired through three cbufs (wbuf cap 1, jbuf cap njobs≡1 today, decisions cap ndecisions≡5). main.c's loop calls sched_update → sched_begin_session → sched_go → drains decisions cbuf with a hardcoded switch (decision->op) → sched_end_session. The picker plugin (fcfs.c) is selected by a literal string (scheduler.c:43) and implements the old xce_sched_plugin_t->job_elect_fn hook, tightly coupled to sched->queues/sched->jobs kvmgrs directly.

Preserved without change: the three-thread/three-cbuf pattern (wbuf→picker, jbuf→allocator, decisions→driver); plugin_register/key_find plugin lookup; select_eval_ctx_t per-session memoization; sched_queue_is_ready()/sched_job_is_ready() (currently stub return 1; in scheduler.c, called as-is by both today's fcfs_elect_job and the new builtin_fcfs).

What changes: xce_scheduler_t splits along the controller-I/O seam into xce_sched_core_t (no apictx/cluster_cfg) and xce_sched_driver_t (holds them). The picker plugin category becomes xce_job_picker_iface_t, replacing job_elect_fn (deprecated with a compatibility wrapper, Req 5 AC "Migration from old job_elect_fn").

Architecture Pattern & Boundary Map

graph TB
    subgraph DriverLayer[Driver Layer]
        NormalDriver[Normal Driver]
        SimDriver[Simulation Driver]
        Controller[Live Controller]
        ReplayFile[Replay File]
        Controller --> NormalDriver
        ReplayFile --> SimDriver
    end

    subgraph CoreLayer[Scheduler Core]
        Core[Scheduler Core State]
        Picker[Job Picker Plugin]
        Allocator[Allocator Thread]
        Core --> Picker
        Picker --> Allocator
    end

    NormalDriver -->|core_push_events core_go| Core
    SimDriver -->|core_push_events core_go| Core
    Core -->|on_decision| NormalDriver
    Core -->|on_decision| SimDriver

    NormalDriver -->|xcecli| Controller
    SimDriver -->|write| RecordFile[Sim Output Replay File]

    subgraph AdminPath[Admin Control Path]
        XceAdmin[xceadmin scheduler CLI]
        SchedAdminOp[OPCode_schedAdmin dispatch]
        XceAdmin --> SchedAdminOp
        SchedAdminOp --> Controller
    end
    Controller -->|forward request-reply| NormalDriver

Architecture Integration:

• Selected pattern: Event-sourced core with a swappable driver (ports-and-adapters, applied narrowly — the core is the hexagon, the two drivers are the only adapters). Selected in requirements.md as the only pattern that makes normal↔sim equivalence a structural property rather than a discipline (see research.md, Architecture Pattern Evaluation).
• Domain boundaries: Core owns scheduling decisions; drivers own I/O translation only — no scheduling logic in either driver. OPCode_schedAdmin dispatch is a third, thin adapter (controller-side) that never touches core or driver state directly, only forwards bytes.
• Existing patterns preserved: three-thread/cbuf pipeline, plugin registration, select-expression memoization.
• New components rationale: SimDriver and the .replay format are net-new (no existing analog); SchedAdminOpcode is new but structurally mirrors the existing OPCode_upgradeConfig case (see research.md).
• Steering compliance: no new wall-clock/network dependency inside src/scheduler/'s core files (ha-async-io.md's async-I/O discipline extended to a synchronous core by design, not violated — the core has no I/O at all); .capnp used for the new wire format per structure.md's "Cap'n Proto for wire formats" convention.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Scheduler core / drivers	C (gnu17)	xce_sched_core_t, xce_sched_driver_t, xce_sched_sim_driver_t	No new language/runtime; existing src/scheduler/ toolchain
Wire format	Cap'n Proto (existing controller.capnp)	.replay stream, OPCode_schedAdmin request/reply	Extends the existing schema file, not a new one
Admin CLI	Python / click (existing xceadmin stack)	scheduler record/scheduler dump commands	New module (xceadmin/scheduler.py) in the existing package, same pycapnp schema loader
Build	Meson ≥ 1.5 (existing)	New .c files added to src/scheduler/meson.build; testsuite entries	No new build system

Dependency Direction

Picker Plugin Interface → Scheduler Core → {Normal Driver, Simulation Driver} → {Controller (xcecli), Replay File I/O}. Controller-side SchedAdminOpcode dispatch depends only on the capnp schema (shared, not owned by the core) and forwards to the driver without depending on xce_sched_core_t internals. xceadmin's scheduler.py depends on xcecommon/xceadmin.discovery only — never on any C header. Each layer imports only from layers to its left; the core never imports driver headers (enforced by the "Allowed Dependencies" list above — no xcecli_ctx_t/netio symbol may appear in core.c).

File Structure Plan

Directory Structure

As-built (updated 2026-08-07 after implementation; see tasks.md's ## Implementation Notes for the full rationale behind each split — all were 800-line-file-cap splits made during implementation, not scope changes):

src/scheduler/
├── include/
│   ├── scheduler.h        # unchanged Phase 1; Phase 2 shrinks to xce_scheduler_t compat shim
│   ├── plugin.h            # + xce_job_picker_iface_t (Req 1 AC 2), deprecation of job_elect_fn
│   ├── core.h              # xce_sched_core_t, xce_core_event_t, decision cb type
│   ├── driver.h            # xce_sched_driver_t (normal-mode)
│   ├── sim.h                # xce_sched_sim_driver_t -- built as sim.h, not sim_driver.h (task 11.1, documented deviation)
│   ├── recording.h          # record start/stop/status types, ReplayEvent/ReplaySession writer contract
│   ├── repl.h                # REPL hook wrappers consumed by sim_loop.c -- no-op unless sim->interactive
│   ├── decision.h           # pre-existing; extended with XCE_DECISION_PREEMPT (task 4.4)
│   └── sched/types.h       # + job_pick_plugin, project_name on xce_sched_queue_t (Req 2)
├── src/
│   ├── scheduler.c         # sched_job_picker(): job_elect_fn != NULL delegates to the legacy loop, NULL runs the generic xce_job_picker_iface_t loop
│   ├── core.c              # xce_sched_core_init/fini, jbuf(cap=1)/decisions cbuf plumbing
│   ├── core_events.c        # xce_core_push_events() -- split out of core.c (800-line cap; task 4.3)
│   ├── core_session.c       # xce_core_begin_session/xce_core_end_session -- split out of core.c (task 4.4)
│   ├── core_clock.c         # xce_core_now(), tlog format integration -- the ONE file allowed to call clock_gettime() (task 4.5)
│   ├── driver.c             # normal-mode driver: capnp<->core event translation, decision publishing, deferred admin-request handling
│   ├── recording.c          # record start/stop/status, header/snapshot writes
│   ├── recording_events.c   # ReplayEvent writes -- split out of recording.c (task 9.2/9.3)
│   ├── recording_session.c  # ReplaySession begin/end writes -- split out of recording.c (task 9.2/9.3)
│   ├── repl.c               # REPL command loop core (execution control, Req 5 AC 22)
│   ├── repl_break.c          # REPL breakpoints + event/decision hook wrappers -- split out of repl.c (task 12.2)
│   ├── repl_info.c           # REPL state-inspection commands -- split out of repl.c (task 12.2)
│   ├── repl_params.c         # REPL snapshot/simulation-parameter commands -- split out of repl.c (task 12.3)
│   ├── repl_internal.h       # shared repl_*.c internal declarations (task 12.2)
│   ├── snapshot.c           # save_snapshot()/load_snapshot() orchestration, picker rebuild
│   ├── snapshot_wire.c       # SnapshotData capnp encode/decode -- split out of snapshot.c (task 8.1)
│   ├── snapshot_load.c       # snapshot restore application -- split out of snapshot.c (task 8.1)
│   ├── sim.c                 # simulation startup (Req 5 AC 13): replay header/snapshot load, virtual clock seed
│   ├── sim_loop.c            # simulation event loop + REPL hook call sites (Req 5 AC 14/18)
│   ├── sim_record.c          # --sim-output ReplayEvent/ReplaySession writes -- split out of sim_loop.c (task 11.5)
│   ├── sim_verify.c          # --sim-verify decision comparator -- split out of sim_loop.c (task 11.4)
│   ├── handler.c            # unchanged — capnp push/notify enqueue into pend_updates; scheduler_req_handler dispatches SchedAdminRequest
│   ├── plugin.c             # + picker plugin registration category, + job_elect_fn compat
                              #   wrapper (adapts old-style job_elect_fn-only plugins to
                              #   xce_job_picker_iface_t, logs a deprecation warning at register time)
│   └── plugins/
│       ├── builtin_fcfs.c   # the native xce_job_picker_iface_t default picker (Req 3) -- NEW file, see note below
│       └── fcfs.c           # UNCHANGED -- still registers job_elect_fn only; kept as the live exercise target for plugin.c's compat wrapper (task 1.2). NOT rewritten as originally planned here; see BuiltinFcfsPicker component notes.
├── testsuite/
│   ├── test_picker_iface.c, test_picker_fcfs.c, test_picker_fcfs_session.c, test_picker_wiring.c,
│   │   test_picker_config_fields.c, test_picker_core_loop_boundary.c, test_picker_pendreason_factory_audit.c,
│   │   test_plugin_picker_registration.c        # picker interface + builtin_fcfs + registration/compat-wrapper coverage
│   ├── test_core_events.c, test_core_session.c, test_core_full_cycle.c, test_core_isolation.c,
│   │   test_core_jbuf_capacity_pin.c            # core contract + determinism-pin coverage
│   ├── test_clock.c                             # xce_core_now() two-base+speed formula
│   ├── test_driver_translate.c, test_driver_decision_cb.c, test_driver_session_ordering.c
│   ├── test_recording_path_validation.c, test_record_lifecycle.c, test_admin_request.c
│   ├── test_snapshot_save_load.c
│   ├── test_sim_startup.c, test_sim_event_loop.c, test_sim_isolation.c, test_sim_alt_events.c,
│   │   test_sim_verify.c, test_sim_record_output.c, test_sim_equivalence.c
│   └── test_repl_execution_control.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c,
│       test_repl_equivalence.c
src/controller/
├── cluster.c                 # + case OPCode_schedAdmin (Req 5 AC 10)
└── testsuite/
    └── test_sched_admin_dispatch.c  # standby rejection, disconnected-scheduler, timeout, success round-trip
                                      # (name differs from this plan's original test_sched_admin.c; same coverage,
                                      #  registered in controller_test.c same as existing opcode tests)
src/schema/
└── controller.capnp          # + SchedAdminRequest/Reply, ReplayHeader/.../ReplayDecision + payload structs (see Data Models);
                               #   + WalCatchupEntry.payload/batchPayloads switched Data->List(UInt8) (bug-1505, follow-up
                               #   outside this spec's boundary, same codec-crash class as SnapshotData.internalState/bug-1504)
src/xceadmin/src/xceadmin/
├── cli.py                   # + @cli.group() scheduler; nested @scheduler.group() record with
                              #   start/stop/status commands, plus @scheduler.command() dump (thin layer only)
└── scheduler.py              # record_start/stop/status(), dump() — mirrors push.py's pattern
src/xceadmin/tests/
├── test_scheduler.py         # mock-TLS-server tests for record start/stop/status + dump parsing
└── test_cli_scheduler_mock.py # additional CLI-level mock-server coverage (task 10.6, beyond this plan's original file list)

Modified Files

• src/scheduler/src/scheduler.c — Phase 1: hardcoded picker string removed; Phase 2: sched_job_picker() branches job_elect_fn != NULL (legacy plugins, delegates to their own self-contained loop) vs NULL (native xce_job_picker_iface_t pickers, generic per-session loop) — scheduler.c was not retired; it remains the picker-dispatch and allocator-loop home.
• src/scheduler/src/plugins/fcfs.c — as-built: left unchanged, not rewritten. plugins/builtin_fcfs.c (new file) implements xce_job_picker_iface_t and is the registered default (schedobj.c); fcfs.c keeps registering job_elect_fn only, deliberately, as the live exercise target for plugin.c's legacy compat wrapper (task 1.2) — see BuiltinFcfsPicker component notes for the full rationale.
• src/scheduler/src/main.c — decision-drain switch replaced by on_decision callback registration (Req 5 AC 6); also gained the --sim/--interactive/--record-dir/--sim-* CLI surface (tasks 9.1, 11.1).
• src/controller/cluster.c — one new case OPCode_schedAdmin: block, structurally mirroring OPCode_upgradeConfig (primary-gate, delegate, explicit reply every path).
• src/schema/controller.capnp — additive only: new OPCode_schedAdmin enum value, new structs, two new union members on existing Request.body/RequestReply.data unions (ordinals @31/@19 respectively — next available, verified against current schema in research.md).
• src/xceadmin/src/xceadmin/cli.py — one new @cli.group() scheduler, containing a nested @scheduler.group() record (with start/stop/status commands) plus a sibling @scheduler.command() dump — matching the literal CLI syntax xceadmin scheduler record start/xceadmin scheduler dump, not 4 flat commands.

System Flows

Normal-Mode Scheduling Round (driver ↔ core)

sequenceDiagram
    participant Ctrl as Controller
    participant Drv as Normal Driver
    participant Core as Scheduler Core
    participant Pick as Picker Thread
    participant Alloc as Allocator Thread

    Ctrl->>Drv: capnp push or notify
    Drv->>Drv: buffer into pend_updates
    Drv->>Core: core_push_events
    Core->>Pick: state change hooks, synchronous, picker idle
    Drv->>Core: core_go, non blocking
    Core->>Pick: signal via wbuf
    Pick->>Pick: begin_session then next_job loop
    Pick->>Alloc: send jobid via jbuf per job
    Alloc->>Alloc: try_allocate then can_preempt
    Alloc->>Drv: send decision via decisions cbuf
    Drv->>Ctrl: on_decision publishes dispatch, pend reason, or preempt
    Alloc->>Drv: end of session marker via decisions cbuf
    Drv->>Core: core_end_session

Key decision not obvious from the diagram: decisions publish to the controller immediately per-decision, not batched at session end (Req 5, "Decision publishing") — the drain loop in the driver thread interleaves cbuf_get/on_decision one at a time.

OPCode_schedAdmin Recording Control (admin path)

sequenceDiagram
    participant CLI as XceAdmin SchedulerModule
    participant Ctrl as Controller ClusterDispatch
    participant Sched as Scheduler NormalDriver

    CLI->>Ctrl: schedAdmin request, action plus recordFile
    Ctrl->>Ctrl: check ha_is_primary, reject standby
    Ctrl->>Sched: forward via scheduler endpoint, five second timeout
    Sched->>Sched: validate recordFile, reject or resolve under record_dir
    Sched->>Sched: save_snapshot then write header and snapshot entry
    Sched-->>Ctrl: schedAdmin reply, ok plus recording state and counters
    Ctrl-->>CLI: relay reply to caller

Key decision not obvious from the diagram: the controller is a pure pass-through — it does not interpret SchedAdminRequest fields beyond the opcode dispatch, and owns no recording state (Boundary Commitments, "This Spec Owns" — recording logic lives entirely in the scheduler driver).

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Core loop calls next_job() only, no ordering logic	SchedulerCore, JobPicker	xce_job_picker_iface_t.next_job	Normal-Mode Round
1.2	Picker hook categories (lifecycle/state/session/post-alloc/optional)	JobPicker	xce_job_picker_iface_t	—
1.3	Three-thread cbuf model	SchedulerCore	wbuf/jbuf/decisions cbufs	Normal-Mode Round
1.4	Picker/allocator/main-loop pseudocode contract	SchedulerCore, JobPicker	core_push_events/core_go	Normal-Mode Round
1.5	Cancel-safe, session-aware picker	JobPicker	begin_session/end_session/next_job	—
1.6	Replace hardcoded picker string	SchedulerCore	plugin_register/key_find	—
1.7	Preserve thread architecture, decision publish via callback	SchedulerCore	on_decision cb	Normal-Mode Round
2.1	job_pick_plugin on xce_scheduler_t, default builtin_fcfs	SchedulerCore	—	—
2.2	job_pick_plugin per-queue override	JobPicker	xce_sched_queue_t.job_pick_plugin	—
2.3	project_name on queue	SchedulerCore	xce_sched_queue_t.project_name	—
2.4	Config-driven picker name, no rebuild	SchedulerCore	scheduler config	—
2.5	Unregistered plugin fails loud	SchedulerCore	plugin_register/key_find	—
2.6	Picker contract documented (cancel-safe, session-aware)	JobPicker	xce_job_picker_iface_t	—
3.1	builtin_fcfs auto-registered as default	BuiltinFcfsPicker	plugin registration	—
3.2	Hook-by-hook FCFS implementation	BuiltinFcfsPicker	xce_job_picker_iface_t	—
3.3	Private ordq_t state, no core queries in next_job	BuiltinFcfsPicker	—	—
3.4	Ordering equivalence with today's fcfs_elect_job	BuiltinFcfsPicker	—	—
3.5	Dedicated picker thread retained, cancel points	BuiltinFcfsPicker, SchedulerCore	next_job/cbuf_get(wbuf)	Normal-Mode Round
3.6	end_session no-op, pendJobs carries over	BuiltinFcfsPicker	—	—
4.1	Job-picking order regression check	BuiltinFcfsPicker	—	test_picker_fcfs.c
4.2	Existing scheduler unit tests pass unmodified	SchedulerCore	—	testsuite/
4.3	HA soak — no job loss/divergence	SchedulerCore	—	/xce-test-ha
5.1	Mode-agnostic core, no mode flag	SchedulerCore	xce_sched_core_init()	—
5.2	Passive event-driven interface, zero I/O deps	SchedulerCore	core_push_events/on_decision	—
5.3	Virtualized clock, two-base+speed formula	SchedulerCore	xce_core_now()	—
5.4	Core event types (typed payloads)	SchedulerCore	xce_core_event_t	—
5.5	Decision output types (DISPATCH/PEND/PREEMPT)	SchedulerCore	xce_decision_t	—
5.6	Decision output callback	SchedulerCore, NormalDriver	xce_core_decision_cb_t	Normal-Mode Round
5.7	Deterministic guarantee + normal↔sim equivalence	SchedulerCore	jbuf cap=1, hash-iteration audit, PRNG seed	test_sim_equivalence.c
5.8	Driver pending-event queue	NormalDriver	pend_updates dataq	Normal-Mode Round
5.9	Controller integration (capnp translate, publish decisions)	NormalDriver	xcecli_ctx_t	Normal-Mode Round
5.10	Runtime recording control (OPCode_schedAdmin)	NormalDriver, SchedAdminOpcode, XceAdminSchedulerCLI	SchedAdminRequest/Reply	Admin Path
5.11	Periodic snapshot during recording	NormalDriver	record_snapshot_interval	—
5.12	Session response time metrics	SchedulerCore	SessionSummary	—
5.13	Simulation mode startup (load header+snapshot)	SimDriver	.replay load sequence	—
5.14	Discrete event-driven virtual clock	SimDriver	--sim-speed	—
5.15	--sim-verify deterministic replay verification	SimDriver	decision_eq	—
5.16	--sim-alt-events what-if branching	SimDriver	alt-events file	—
5.17	Simulation isolation (no controller deps)	SimDriver	xce_sched_sim_driver_t	—
5.18	ReplaySession as session boundary	SimDriver	event loop	—
5.19	--sim-output simulation record file	SimDriver	.replay writer	—
5.20	.replay capnp stream format	ReplayFormat	schema structs	—
5.21	Snapshot serialization / picker rebuild	SchedulerCore, ReplayFormat	save_snapshot/load_snapshot	—
5.22	Interactive REPL	SimDriver	command loop	—
6.1	Resreq immutability	SchedulerCore	—	—
6.2	Pend reason factory (make_pendreason())	SchedulerCore	lib/preason.c	—
6.3	Pointer-identity fast path	SchedulerCore	—	—
6.4	Select expression memoization preserved	SchedulerCore	select_eval_ctx_t	—
6.5	Snapshot forward-compatible with object sharing	ReplayFormat	SnapshotData	—

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
JobPicker (interface)	Scheduler Core	Defines the pluggable job-selection contract	1.1, 1.2, 1.4, 1.5, 2.2, 2.6	plugin.h (P0)	Service
BuiltinFcfsPicker	Scheduler Core	Default picker replicating today's FCFS order	3.1-3.6, 4.1	JobPicker (P0)	Service
SchedulerCore	Scheduler Core	Passive, mode-agnostic scheduling engine	1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4	JobPicker (P0), allocation/preemption (P1, sibling specs)	Service, State
NormalDriver	Scheduler Driver	Controller-attached driver, recording control	5.6, 5.8-5.11	SchedulerCore (P0), xcecli_ctx_t (P0)	Service, Event
SimDriver	Scheduler Driver	Replay-attached driver, virtual clock, REPL	5.13-5.19, 5.22	SchedulerCore (P0), ReplayFormat (P0)	Service, State
ReplayFormat	Wire Protocol	.replay capnp schema, SnapshotData	5.20, 5.21, 6.5	controller.capnp (P0)	Batch
SchedAdminOpcode	Controller	Pass-through dispatch for OPCode_schedAdmin	5.10	cluster.c myreq_handler (P0)	API
XceAdminSchedulerCLI	Tooling	scheduler record/scheduler dump commands	5.10	xcecommon.wire/discovery (P0)	API

Scheduler Core

JobPicker

Field	Detail
Intent	Contract every job-selection strategy implements; hides all ordering logic from the core loop
Requirements	1.1, 1.2, 1.4, 1.5, 2.2, 2.6

Responsibilities & Constraints

• Owns iteration order over pending jobs; the core never inspects queue/job internals for ordering.
• Cancel-safe (pthread_cancel points in next_job()) and session-scoped (begin_session resets iteration state).
• State-change hooks run synchronously in the driver thread during core_push_events(); session hooks run in the picker thread; post-allocation hooks run in the allocator thread (Req 1 AC 3).

Dependencies

• Inbound: SchedulerCore — invokes every hook (P0)
• Outbound: none — picker implementations must not call back into core internals outside the hook parameters they're given

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

typedef struct {
    int  (*init)(void* picker, xce_sched_core_t* core);
    void (*fini)(void* picker);
    int  (*begin_session)(void* picker);
    xce_sched_job_t* (*next_job)(void* picker);
    int  (*end_session)(void* picker);
    int  (*on_dispatch)(void* picker, uint64_t jobid);
    int  (*on_pend)(void* picker, uint64_t jobid);
    int  (*on_queue_update)(void* picker, xce_sched_queue_t* queue);
    int  (*on_job_push)(void* picker, xce_sched_job_t* job);
    int  (*on_job_state_change)(void* picker, xce_sched_job_t* job, uint32_t state);
    int  (*on_job_update)(void* picker, xce_sched_job_t* job);
    int  (*on_config)(void* picker, xce_conf4sched_t* config);       /* MAY be NULL */
    int  (*on_job_rusage)(void* picker, xce_sched_job_t* job, xce_job_rusages_t* rusages); /* MAY be NULL */
} xce_job_picker_iface_t;

• Preconditions: init() called exactly once before any other hook; begin_session() precedes every next_job() sequence.
• Postconditions: next_job() returns NULL exactly when no ready job remains for the current session.
• Invariants: hook implementations never block indefinitely (picker thread holds no lock the allocator needs); next_job() never allocates a job the same session has already returned.

Implementation Notes

• Integration: registered via plugin_register/key_find under the picker plugin category (new plugin.c registration table entry).
• Validation: test_picker_fcfs.c feeds hooks directly and asserts next_job() order — no live core needed (Req 3 AC 3's testability rationale).
• Risks: a picker implementation that queries core state directly during next_job() (bypassing the hook-fed private state pattern) would still compile but silently reintroduce lock contention — not enforceable at the interface level, called out for code review.

BuiltinFcfsPicker

Field	Detail
Intent	Default xce_job_picker_iface_t implementation; replicates today's fcfs_elect_job ordering exactly through the new interface
Requirements	3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 4.1

Responsibilities & Constraints

• Maintains private state only — ordq_t queues (ordered by queue->priority desc) and, per queue, an ordq_t pendJobs (ordered by priority desc, queue_time asc). Never queries sched->queues/sched->jobs during next_job() (Req 3 AC 3).
• Every hook's behavior is fully specified in requirements.md Req 3 AC 2's hook-by-hook table — not repeated here; this block covers integration and testability, not the algorithm.
• Auto-registered as "builtin_fcfs" and set as the default job_pick_plugin at scheduler startup (Req 3 AC 1).
• end_session() is a no-op; undispatched jobs carry over in pendJobs to the next round unchanged (Req 3 AC 6).

Dependencies

• Inbound: SchedulerCore — invokes every xce_job_picker_iface_t hook, same as any picker (P0)
• Outbound: none — implements the interface, depends on nothing beyond it

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

Implements xce_job_picker_iface_t (struct shown under JobPicker, above) with the fcfs_plugin_data_t private state described in Responsibilities. No additional public surface beyond the interface itself.

• Preconditions/Postconditions/Invariants: inherits JobPicker's contract in full; adds no relaxations or extensions.

Implementation Notes

• Integration, as-built: implemented in a new file, plugins/builtin_fcfs.c, registered as "builtin_fcfs" and set as the default job_pick_plugin (schedobj.c) — NOT a rewrite of plugins/fcfs.c in place, as originally planned above. fcfs.c was deliberately left untouched (still job_elect_fn-only, unregistered as default) because task 1.2's job_elect_fn compat wrapper needs a real legacy-style plugin to exercise; fcfs.c fills that role, and scheduler.c's sched_job_picker() dispatch (job_elect_fn != NULL vs NULL) depends on both variants existing side by side. Functionally equivalent to the original plan (one default picker implementing the new interface, ordering-equivalent to legacy fcfs_elect_job); only the file-layout mechanics differ.
• Validation: test_picker_fcfs.c is this component's primary test — feeds on_queue_update/on_job_push/on_job_state_change hooks directly and asserts next_job() returns jobs in (queue priority desc, job priority desc, queue_time asc) order, matching today's fcfs_elect_job (Req 4.1's regression check).
• Risks: ordering-equivalence regression is the main risk (any drift from today's fcfs_elect_job output is a behavior change operators didn't opt into) — mitigated by test_picker_fcfs.c asserting against the documented ordering rule directly, plus the HA soak in Req 4.3.

SchedulerCore

Field	Detail
Intent	Owns all scheduling state; drives the picker/allocator threads; exposes the driver↔core protocol
Requirements	1.1, 1.3-1.4, 1.6-1.7, 2.1, 2.3-2.5, 3.5, 4.2-4.3, 5.1-5.7, 5.12, 5.21, 6.1-6.4

Responsibilities & Constraints

• Zero direct dependency on controller communication, capnp, network I/O, filesystem, or wall-clock time (Req 5 AC 2) — enforced by the "Allowed Dependencies" link list, not just convention.
• jbuf cbuf capacity is hardcoded to 1 inside xce_sched_core_init() — never a parameter (Req 5 AC 7, pinned against the existing njobs FIXME in scheduler.c:384).
• All internal time via xce_core_now(); no direct time()/clock_gettime()/gettimeofday() calls anywhere in core.c.

Dependencies

• Inbound: NormalDriver, SimDriver — both call core_push_events/core_go, register on_decision (P0)
• Outbound: JobPicker (P0); try_allocate()/can_preempt() from sibling specs (P1 — signature-stable dependency, no behavior assumption)
• External: none

Contracts: Service [x] / State [x]

Service Interface

int      xce_sched_core_init(xce_sched_core_t*, xce_core_decision_cb_t, void* driver_ctx, uint64_t prng_seed);
int      xce_core_push_events(xce_sched_core_t*, xce_core_event_t* events, int nevents);
int      xce_core_go(xce_sched_core_t*);
int      xce_core_end_session(xce_sched_core_t*);
uint64_t xce_core_now(xce_sched_core_t*);
int      xce_sched_core_save_snapshot(xce_sched_core_t*, SnapshotData* out);
int      xce_sched_core_load_snapshot(xce_sched_core_t*, const SnapshotData* in);
void     xce_sched_core_fini(xce_sched_core_t*);

• Preconditions: core_push_events() only valid when no session is active (session_active == 0); core_go() returns -EINVAL if a session is already active. xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot() are likewise only valid when session_active == 0 — both return -EBUSY if called mid-session, rather than silently capturing or restoring torn state. Every caller in this design (record start, periodic checkpoint, simulation startup) already invokes them between sessions by construction; the -EBUSY return is a defensive contract check, not a path any correct caller should hit.
• Postconditions: after on_decision delivers the end-of-session NULL marker, session_active is guaranteed to transition to 0 on the subsequent core_end_session() call.
• Invariants: given identical (config, event sequence), core_push_events+core_go produce an identical decision sequence (Req 5 AC 7) — the whole point of the design.

State Management

• State model: scheduling state (jobs/queues/workers/resources/resvals/users/projects) held in kvmgrs inside xce_sched_core_t, mirroring today's xce_scheduler_t layout minus apictx/cluster_cfg.
• Persistence & consistency: no persistence inside the core itself — save_snapshot()/load_snapshot() are the only serialization boundary, and only the driver decides when to invoke them.
• Concurrency strategy: three-thread cbuf pipeline unchanged from today; clock atoms (virtual_clock_base, real_clock_base, speed_num, speed_den) are lock-free _Atomic uint64_t, updated by the driver as a group before each event dispatch.

Implementation Notes

• Integration: core_push_events/core_go/on_decision are the only surface either driver touches — no driver reaches into xce_sched_core_t internals.
• Validation: test_core_events.c (contract-level), test_sim_equivalence.c (record→replay→diff, Req 5 AC 7.1 — the spec's central regression test).
• Risks: the kvmgr iteration-order audit (Req 5 AC 7, "Stabilized hash table iteration") is a Phase 2 deliverable with real but bounded scope — jobs/queues/users/projects only (per research.md and requirements.md's explicit Phase 2 task).

Scheduler Driver

NormalDriver

Field	Detail
Intent	Connects the core to a live controller; owns recording control
Requirements	5.6, 5.8, 5.9, 5.10, 5.11

Responsibilities & Constraints

• Translates capnp push/notify into xce_core_event_t, buffers between sessions in pend_updates (existing dataq swap mechanism).
• Publishes each decision immediately via xcecli_jobdispatch/xcecli_set_pend_reason/xcecli_preempt as it drains from decisions cbuf — not batched at session end.
• Recording is boolean-gated: zero file I/O, zero extra allocation when recording == false (Req 5 AC 10, Performance).
• Validates recordFile before ever calling fopen(): reject / or .., resolve under record_dir, confirm containment via realpath() (Req 5 AC 10, "Record file path validation" — this design inherits, not re-derives, that requirement).
• record_dir is provisioned exactly one way: a --record-dir DIR scheduler startup CLI flag (default /var/log/xce/), parsed alongside the existing normal-mode arguments in main.c — never a capnp field on any request path (SchedAdminRequest included), so nothing reachable over the wire can redirect where recordings are written. Matches the existing --sim-* flag family's provisioning style (CLI-only, not config-pushed).

Dependencies

• Inbound: main.c (process entry point) (P0)
• Outbound: SchedulerCore (P0), xcecli_ctx_t (P0)
• External: SchedAdminOpcode dispatch (P1 — receives forwarded requests, not a caller relationship)

Contracts: Service [x] / Event [x]

Service Interface

typedef struct {
    xce_sched_core_t     core;
    xcecli_ctx_t         apictx;
    xce_cluster_config_t cluster_cfg;
    dataq_t*             pend_updates[2];
    char*    record_dir;              /* from --record-dir CLI flag at startup; default "/var/log/xce/";
                                          never settable via any request-path field */
    char*    record_file_path;
    FILE*    record_file;
    bool     recording;
    uint64_t record_event_count;
    uint64_t record_snapshot_interval;
    uint64_t record_start_ts;
} xce_sched_driver_t;

int sched_driver_run(xce_sched_driver_t*, xce_conf4sched_t*, xce_cluster_config_t*, void* handler);
int sched_driver_record_start(xce_sched_driver_t*, const char* record_file /* bare filename or NULL */);
int sched_driver_record_stop(xce_sched_driver_t*);
int sched_driver_record_status(xce_sched_driver_t*, SchedAdminReply* out);

• Preconditions: record_start fails with -EALREADY-equivalent if already recording; record_stop fails if not recording.
• Postconditions: on record_start success, ReplayHeader + initial SnapshotEntry are durably on disk before the reply is sent.
• Invariants: recording == true iff record_file != NULL.

Event Contract

• Subscribed events: capnp push/notify from the controller (existing handler.c path, unchanged).
• Published events: decisions to the controller via xcecli_ctx_t; SchedAdminReply back through the forwarding path on recording-control requests.
• Ordering / delivery guarantees: decisions publish in allocator output order; recording writes events/decisions in the same order they cross the driver thread.

Implementation Notes

• Integration: main.c's mode-selection branch calls sched_driver_run() when --sim is absent.
• Validation: recording lifecycle tested against a real filesystem (temp dir) — start/stop/status round trip, path-validation rejection cases, disk-full simulated via a read-only mount point.
• Risks: OPCode_schedAdmin ships with the same primary/standby gate as OPCode_upgradeConfig and no additional RBAC/auth check — a deliberate scope decision (project owner, 2026-08-02), not an oversight. Strengthening authentication for this opcode (and config push, if it also lacks one) is deferred to a follow-up spec. See Open Questions/Risks.

SimDriver

Field	Detail
Intent	Connects the core to a .replay file; owns virtual clock, verification, and the REPL
Requirements	5.13-5.19, 5.22

Responsibilities & Constraints

• Contains no xcecli_ctx_t, xce_cluster_config_t, capnp socket, or controller RPC of any kind (Req 5 AC 17) — enforced by the struct definition itself having no such fields.
• Discrete virtual clock: jumps directly to the next event's timestamp, never a fixed tick (Req 5 AC 14).
• REPL is a single fgets loop in the same thread as the simulation loop — no REPL thread (Req 5 AC 22, Implementation note).

Dependencies

• Inbound: main.c (P0, when --sim is present)
• Outbound: SchedulerCore (P0), ReplayFormat (P0)
• External: none

Contracts: Service [x] / State [x]

Service Interface

typedef struct {
    xce_sched_core_t  core;
    FILE *replay_file, *alt_events_file, *record_file;
    uint64_t sim_speed_num, sim_speed_den;
    bool     verify;
    bool     interactive;
} xce_sched_sim_driver_t;

int sched_sim_driver_run(xce_sched_sim_driver_t*, /* CLI arguments struct */ const void* args);

• Preconditions: replay file's second message must be a SnapshotEntry (rejected otherwise, Req 5 AC 20).
• Postconditions: with --sim-output, every session cycle appends exactly one ReplaySession to the output file before the next ReplayEvent batch begins.
• Invariants: REPL step/continue to completion produces byte-identical output to batch --sim (Req 5 AC 22, "Deterministic replay is preserved").

State Management

• State model: same xce_sched_core_t as normal mode, loaded via load_snapshot() instead of live events.
• Persistence & consistency: --sim-output writes a new independent .replay file; never mutates the input file.
• Concurrency strategy: single-threaded driver loop around the same core thread pair; REPL blocks the driver loop intentionally (no wall-clock deadline in simulation).

Implementation Notes

• Integration: main.c's mode-selection branch calls sched_sim_driver_run() when --sim is present.
• Validation: test_sim_equivalence.c drives this component directly (record via NormalDriver's test harness, replay via this one, diff).
• Risks: none beyond what's already captured under SchedulerCore (the determinism guarantee is a core property; SimDriver only needs to not violate it, e.g. by never bypassing core_push_events/core_go).

Wire Protocol

ReplayFormat

Field	Detail
Intent	.replay capnp stream schema and SnapshotData
Requirements	5.20, 5.21, 6.5

Responsibilities & Constraints

• Additive-only changes to controller.capnp — reuses existing structs (Conf4Sched, SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages) wherever they already exist (verified in research.md); only defines new structs where none exist.
• SnapshotData field shapes track the actual current C struct field sets (e.g. xce_sched_user_t is {name} only today, xce_sched_project_t is {name} only today) — the schema does not invent fields the C structs don't have yet.

Dependencies

• Inbound: SchedulerCore (save_snapshot/load_snapshot), NormalDriver (recording writer), SimDriver (reader/writer) (all P0)
• Outbound: none
• External: none

Contracts: Batch [x]

Batch / Job Contract

• Trigger: record start (normal mode) or --sim-output (simulation mode) write a .replay stream incrementally; --sim/xceadmin scheduler dump read one.
• Input / validation: first message must be ReplayHeader; second must be SnapshotEntry (Req 5 AC 20) — both enforced at read time, not just documented.
• Output / destination: append-only file; record stop / clean simulation completion writes ReplayEvent{endSession}.
• Idempotency & recovery: a write failure mid-message truncates cleanly to the last complete message (Req 5 AC 10, error handling) — no partial-message corruption possible because each message is a full capnp frame written atomically at the fwrite level.

New capnp structs (additive to controller.capnp; ReplayHeader/ReplayEvent/ReplaySession/ReplayDecision/SessionSummary wrapper structs already fully specified in requirements.md Req 5 AC 20 — not repeated here):

# New event payloads (JOBS_PUSH/WORKERS_PUSH/etc reuse existing Conf4Sched,
# SchedJobData, WorkerData, ResourceData, ResValList, QueueConf, SchedJobRusages —
# verified present in controller.capnp, research.md).

struct JobStateChangeEvent $C.mapname("job_state_change_event_t") {
  jobid    @0: UInt64;
  oldState @1: UInt32;
  newState @2: UInt32;
  exitCode @3: Int32;
}

struct JobUpdateEvent $C.mapname("job_update_event_t") {
  jobid         @0: UInt64;
  changedFields @1: UInt32;   # bitmask: JOB_F_PRIORITY, JOB_F_QUEUE, JOB_F_RESREQ
  job           @2: Job;      # full updated record; reuses existing Job struct
}

# New decision payloads for ReplayDecision.body (Req 5 AC 5).
# DISPATCH/PEND reuse existing JobAlloc/JobPendReason; PREEMPT is new
# (wire shape owned here per Boundary Commitments; victim-selection
# semantics owned by scheduler-preemption — see research.md).
#
# AC 5's field list for DISPATCH is "jobid, workerid, allocations[]"
# (singular workerid). JobAlloc already generalizes this correctly for
# multi-host jobs: JobAlloc.jobid covers jobid, JobAlloc.allocs is
# List(HostAlloc) with each HostAlloc.host as that allocation's worker
# id and HostAlloc.allocs: List(ResAlloc) as its resource/amount pairs
# -- verified against controller.capnp, no new fields needed. jobid
# lives ONLY in JobAlloc.jobid (not duplicated at DispatchDecision
# level) to avoid two unsynchronized sources of truth for the same job.

struct DispatchDecision $C.mapname("sched_dispatch_decision_t") {
  alloc  @0: JobAlloc;
}

struct PendDecision $C.mapname("sched_pend_decision_t") {
  jobid  @0: UInt64;
  reason @1: JobPendReason;
}

struct PreemptVictim $C.mapname("preempt_victim_t") {
  jobid    @0: UInt64;
  workerid @1: Text;
}

struct PreemptDecision $C.mapname("sched_preempt_decision_t") {
  jobid   @0: UInt64;
  victims @1: List(PreemptVictim);
  reason  @2: JobPendReason;
}

# SnapshotData — full core-state checklist from requirements.md Req 5 AC 21,
# given a concrete wire shape. Reuses existing List(...) payload structs
# for domain objects already carried elsewhere on the wire; the two
# scheduler-internal-only caches (shared-resource tracking, unsatisfiable-
# resreq cache) and session/PRNG state have no existing external contract,
# so they are versioned opaque blobs the core alone reads/writes -- no
# consumer outside save_snapshot()/load_snapshot() ever inspects their
# internal shape (WHAT not HOW: the boundary that matters is "round-trips
# through the core, opaque to everyone else").

struct SnapshotData $C.mapname("sched_snapshot_data_t") {
  config          @0: Conf4Sched;
  jobs            @1: SchedJobData;
  queues          @2: List(QueueConf);
  workers         @3: WorkerData;
  resources       @4: ResourceData;
  resvals         @5: ResValList;
  users           @6: List(SchedUser);
  projects        @7: List(SchedProject);
  prngSeed        @8: UInt64;
  internalState   @9: Data;   # shared-resource tracking + unsatisfiable-resreq
                              # cache + session/iterator state, core-opaque
  internalVersion @10: UInt32; # format tag for internalState, so a future
                                # incompatible internal layout fails loud
                                # instead of misinterpreting old bytes
}

struct SchedUser $C.mapname("xce_sched_user_t") {
  name @0: Text;   # matches xce_sched_user_t today (name only) --
                   # extend in lockstep if xce_sched_user_t gains fields
}

struct SchedProject $C.mapname("xce_sched_project_t") {
  name     @0: Text;
  priority @1: Int32;
}

Implementation Notes

• Integration: save_snapshot()/load_snapshot() are the only functions that construct/consume SnapshotData; no other component touches internalState's bytes.
• Validation: as-built, round-trip coverage for every new struct lives in src/controller/testsuite/test_e2e_capnp.c (task 7.5), not a dedicated test_replay_format.c — this repo's existing convention is one shared capnp round-trip test file per schema, and this spec's new structs were added to it rather than starting a second one; internalVersion mismatch is a dedicated negative test within it.
• Risks: internalState's opaque-blob design trades wire-level inspectability for implementation flexibility — acceptable because AC 21 only requires the core to round-trip correctly, never that xceadmin scheduler dump render its internals meaningfully (dump can print internalState's byte length and internalVersion only).

Controller

SchedAdminOpcode

Field	Detail
Intent	Pass-through dispatch for OPCode_schedAdmin in the controller's request handler
Requirements	5.10

Responsibilities & Constraints

• Structurally mirrors the existing OPCode_upgradeConfig case (cluster.c:670-705) for the primary-gate/delegate/explicit-reply shape: primary-only gate via ha_is_primary(), explicit reply on every path, 5s timeout.
• Owns zero recording state — every field of SchedAdminRequest/Reply passes through untouched.
• Blocking-wait admission (hardened 2026-08-19, design review follow-up): like OPCode_upgradeConfig (in-process commit-barrier wait, COMMIT_WAIT_RESERVED-admitted), this case now also admission-gates its blocking wait. Originally (accepted, project owner, 2026-08-02) this case blocked its vtask handler thread for up to 5s waiting on a cross-process reply from the scheduler daemon while holding c->sched.lock, with no reservation — a boundary case reviewed against ha-async-io.md's vtaskmgr admission rule and accepted on the grounds that xceadmin scheduler commands are cluster-administrator-triggered, low-frequency, human-paced traffic, not expected to burst concurrently the way heartbeat/replication traffic does. A retrospective design review flagged this as a real (if low-probability) starvation risk under a burst of concurrent/scripted requests, since each blocked vtask thread occupies a worker for up to 5s and c->sched.lock already serializes them one at a time — N concurrent requests could occupy N threads for up to N×5s. Fix: vtaskmgr_blocking_acquire(&c->vtaskmgr)/vtaskmgr_blocking_release(&c->vtaskmgr) now wrap the ENTIRE forward+wait region (acquired before mylock_lock(&c->sched.lock), released on every exit path) — the same shared budget commit.c's wait_admit_locked()/wait_release_locked() already gate on (wired via commit_set_pool(&c->commit, &c->vtaskmgr)). Admission-exhausted requests fast-reject with {ok=false, error="server busy, try again"} before ever touching c->sched.lock — never queued. See requirements.md Req 5 AC 10's own Amendment for the full rationale.

Dependencies

• Inbound: XceAdminSchedulerCLI, via TLS+capnp (P0)
• Outbound: NormalDriver (P0, forwards the request)
• External: none

Contracts: API [x]

API Contract

Method	Endpoint (opcode)	Request	Response	Errors
Request/Reply	OPCode_schedAdmin	SchedAdminRequest	SchedAdminReply	-EAGAIN (not primary), "scheduler not connected", "request timeout", "invalid record file name", "already recording", "not recording", "server busy, try again" (admission exhausted) (full table: requirements.md Req 5 AC 10)

Implementation Notes

• Integration: one new case in myreq_handler's switch (cluster.c); no new file.
• Validation: test_sched_admin_dispatch.c (src/controller/testsuite/; named differently from this plan's original test_sched_admin.c, same coverage) asserting the reply-on-every-path property (standby rejection, disconnected-scheduler rejection, timeout) — mirrors existing OPCode_upgradeConfig test coverage style.
• Risks: no RBAC/auth check is visible for the comparable OPCode_upgradeConfig case in cluster.c (see research.md), and OPCode_schedAdmin inherits that same gap by deliberate decision — auth hardening for this opcode is deferred to a follow-up spec, not silently accepted as permanent.

Tooling

XceAdminSchedulerCLI

Field	Detail
Intent	xceadmin scheduler record start/stop/status and xceadmin scheduler dump <FILE>
Requirements	5.10

Responsibilities & Constraints

• cli.py stays a thin click layer; all logic lives in the new scheduler.py module, mirroring push.py's split.
• record_start/stop/status() go through xceadmin.discovery.follow_redirects + decode_reply (request/reply opcode, needs redirect-following, unlike probe_config()).
• dump() has no network dependency — decodes a local .replay file only.

Dependencies

• Inbound: operator CLI invocation (P0)
• Outbound: xcecommon.wire/schema, xceadmin.discovery (P0); SchedAdminOpcode via the wire (P0, for record/status only)
• External: none

Contracts: API [x]

API Contract

Method	Endpoint	Request	Response	Errors
CLI	scheduler record start [FILE]	SchedAdminRequest{recordStart}	SchedAdminReply	XceAdminError subclasses (connection, redirect, decode)
CLI	scheduler record stop	SchedAdminRequest{recordStop}	SchedAdminReply	same
CLI	scheduler record status	SchedAdminRequest{recordStatus}	SchedAdminReply	same
CLI	scheduler dump <FILE>	(local file)	formatted text to stdout	SchemaNotFoundError, file-not-found

Implementation Notes

• Integration: cli.py imports record_start/record_stop/record_status/dump from scheduler.py, same import shape as today's build_cluster_config/push_config.
• Validation: test_scheduler.py — mock-TLS-server tests for the three record commands (mirrors test_config_push_mock.py), plus golden-file tests for dump() against a small fixture .replay file.
• Risks: none beyond ReplayFormat's (this component only formats/transmits, never interprets scheduling semantics).

Data Models

Covered in full under Components and Interfaces → ReplayFormat above (new capnp structs) — not duplicated here per the dedup guidance in design-principles.md. SnapshotData's reuse of existing Conf4Sched/SchedJobData/WorkerData/ResourceData/ResValList/QueueConf is the only cross-cutting data-model decision; see research.md, "Reusable capnp payload types," for the verification trail.

Error Handling

Error Strategy

Fail-loud, no silent fallback — consistent with the controller's existing "fail loud, stay alive" design invariant (src/controller/CLAUDE.md), applied to the scheduler process at the scope this feature touches.

Error Categories and Responses

• Configuration errors (unregistered picker plugin, Req 2 AC 5): scheduler startup fails with a clear error — no silent fallback to builtin_fcfs.
• Recording errors (disk full, permission error, invalid recordFile): logged, recording state cleanly torn down (recording=false, file closed, record_file_path freed), scheduling continues unaffected (Req 5 AC 10, error handling) — recording failure is never fatal to the scheduler process.
• Simulation errors (missing/corrupt replay file, missing initial SnapshotEntry): simulation driver fails at startup with a clear message — no partial/best-effort simulation.
• Admin-path errors (OPCode_schedAdmin): every failure mode gets an explicit {ok=false, error=...} reply — never a dropped request (mirrors the "no silent drop" precedent already established for replication-class messages, src/controller/CLAUDE.md's protocol checklist).

Monitoring

No new metrics infrastructure — SessionSummary (Req 5 AC 12) provides session-duration diagnostics through the existing .replay recording path; no separate telemetry system is introduced.

Testing Strategy

• Unit — Picker contract (test_picker_fcfs.c): feed on_queue_update/on_job_push/on_job_state_change hooks directly, assert next_job() returns jobs in (queue priority desc, job priority desc, queue_time asc) order — verifies Req 3.2-3.4 without a live core.
• Unit — Core event contract (test_core_events.c): core_push_events() followed by core_go() with a stub picker asserts state-change hooks fire exactly once per event, synchronously, before core_go() returns — verifies Req 5.1-5.8.
• Unit — Clock virtualization (test_clock.c): xce_core_now() against the four mode rows in Req 5 AC 3's table (normal, discrete, real-time, fast/slow) — asserts the two-base+speed formula directly, no wall-clock flakiness (fixed real_clock_base/mocked clock_gettime delta).
• Unit — Replay format round-trip (as-built: src/controller/testsuite/test_e2e_capnp.c, not a dedicated test_replay_format.c — added to this repo's existing shared capnp round-trip test file instead): encode/decode every new capnp struct (JobStateChangeEvent, JobUpdateEvent, DispatchDecision, PendDecision, PreemptDecision, SnapshotData) — field-for-field equality after round-trip.
• Integration — Deterministic replay (test_sim_equivalence.c, Req 5.7/5.7.1): record a synthetic scheduling session through NormalDriver's test harness → replay through SimDriver → assert ReplayDecision sequences match exactly. This is the design's highest-value test — it directly exercises the central invariant.
• Integration — Recording lifecycle (driver-level, Req 5.10-5.11): record start → submit synthetic events → record stop → verify file structure (ReplayHeader, SnapshotEntry, alternating ReplayEvent/ReplaySession, periodic checkpoints at the configured interval, final endSession).
• Integration — OPCode_schedAdmin dispatch (test_sched_admin_dispatch.c, controller-level; named differently from this plan's original test_sched_admin.c): standby rejection, disconnected-scheduler rejection, timeout, and successful forward/reply round trip — mirrors existing OPCode_upgradeConfig test coverage.
• Integration — xceadmin scheduler CLI (test_scheduler.py): mock-TLS-server tests for record start/stop/status (redirect-following, reply decoding) plus golden-file tests for dump() against a fixture .replay file.
• E2E — HA soak regression (Req 4.3, Req 5 per-phase verification): /xce-test-ha sequential and chaotic rounds after each phase, asserting zero new job loss/divergence attributable to this feature — required before each phase merges, not a one-time gate at the end.
• E2E — REPL equivalence (Req 5.22): --interactive with step/continue to completion produces the same decision sequence as batch --sim on the same replay file.

Security Considerations

Feature-specific decisions only; baseline RBAC/auth practices are in .kiro/steering/security.md.

• recordFile path validation is already fully specified at the requirements level (Req 5 AC 10, "Record file path validation" — reject //.., resolve under configured record_dir, realpath() containment check before fopen()). This design's NormalDriver and SchedAdminOpcode components inherit that requirement directly. record_dir itself is provisioned only via a --record-dir scheduler startup CLI flag — never from SchedAdminRequest or any other request-path field — so the validation boundary can't be redirected by a caller.
• OPCode_schedAdmin ships with no RBAC/auth check beyond the primary/standby gate — accepted scope decision, deferred hardening. The comparable OPCode_upgradeConfig case has no visible RBAC/auth check in cluster.c either (see research.md). Project owner decision (2026-08-02): authentication for this opcode is enhanced in a follow-up spec, not blocking this one. Phase 3 ships OPCode_schedAdmin at parity with OPCode_upgradeConfig's current (unauthenticated-beyond-primary-gate) posture — this is a known, accepted gap, not a silent one.
• .replay files may contain sensitive cluster state (full job/user/resource snapshots) — file permissions on record_dir are an operational concern (steering-level, not new to this feature) but worth calling out since this is the first feature to write full scheduler snapshots to disk in a human-decodable format (xceadmin scheduler dump).

Performance & Scalability

Feature-specific targets only; general practices are in steering docs.

• Recording overhead: zero file I/O, zero extra allocation when recording == false — a single boolean branch per session (Req 5 AC 10, explicit requirement, inherited here as a design constraint on NormalDriver).
• Determinism vs throughput: jbuf cbuf capacity is permanently 1 (rendezvous, not a buffer) — this bounds picker→allocator throughput to strictly sequential handoff. This is an intentional trade-off (determinism over pipelining) already made at the requirements level (Req 5 AC 7) and is not revisited here; any future throughput work must find a different mechanism that preserves the ordering guarantee (e.g. deterministic batching with an explicit order tag), not simply raise the cbuf capacity.
• Simulation fast mode: speed_num=0 means the virtual clock never advances between events at wall-clock cost — --sim on a large .replay file runs at CPU-bound speed, not wall-clock speed. No specific throughput target is set here (none given in requirements); implementation should avoid introducing any per-event syscall beyond what decoding the capnp stream requires.
• OPCode_schedAdmin vtask occupancy (hardened 2026-08-19): each request blocks its vtask handler thread up to 5s (cross-process wait on the scheduler's reply) while holding c->sched.lock — now gated by the same vtaskmgr_blocking_acquire/_release admission reservation commit.c's commit-wait barrier shares (see SchedAdminOpcode component above). Admission-exhausted bursts fast-reject instead of occupying vtask threads; not separately load-tested against a real concurrent burst (unit-tested only, test_schedadmin_fast_rejects_when_admission_exhausted), but the mechanism itself is the same one already proven under test_vtask_admission.c and production commit-wait traffic.

Open Questions / Risks

1. OPCode_schedAdmin authorization mechanism (P2 — accepted gap, deferred). Ships at parity with OPCode_upgradeConfig's current posture (primary/standby gate only, no RBAC/auth check). Project owner decision (2026-08-02): auth hardening for this opcode is a follow-up spec, not a Phase 3 blocker. Tracked in .kiro/steering/roadmap.md's existing peer-authentication-hardening backlog item (updated 2026-08-02 to cover OPCode_schedAdmin alongside OP_upgradeConfig — same validate_host() gate closes both). See Security Considerations and research.md.
2. PreemptDecision's wire shape is pre-committed ahead of scheduler-preemption's own design (P1). Flagged as a revalidation trigger in Boundary Commitments; scheduler-preemption's design should explicitly confirm or extend this shape early, not discover it mid-implementation.
3. SnapshotData.internalState opaque-blob scope (P2). If a future spec needs xceadmin scheduler dump to render shared-resource-tracking or unsatisfiable-resreq-cache contents meaningfully (not just byte length), internalState will need to become a structured capnp field instead of opaque Data — deferred here because no current requirement asks for it (AC 20's dump description only requires human-readable output for what dump already needs to show, and internals were never asked to be inspectable).
4. xce_sched_core_t's config/plugins/jobs/queues/workers/users/projects field block is left as var ... in requirements.md's data-structure summary (not fully typed) — Phase 2 implementation must finalize these as concrete kvmgr/pointer types when core.c is written; this design does not re-type them since requirements.md already commits to "moved from xce_scheduler_t" as the authoritative source of truth for their shapes.
5. OPCode_schedAdmin blocks a vtask handler thread up to 5s per request with no admission reservation FIXED 2026-08-19 (design-review follow-up). Now gated by vtaskmgr_blocking_acquire/_release against the same c->vtaskmgr reservation commit.c's commit-wait barrier shares; admission-exhausted requests fast-reject with {ok=false, error="server busy, try again"} instead of blocking. See SchedAdminOpcode component above and research.md's "SUPERSEDES" note.
6. Req 5 AC 3's tlog virtual-clock logging integration is built but never activated FIXED 2026-08-19 (design-review follow-up). sched_sim_driver_start() (sim.c) now calls xce_core_register_tlog_format()/xce_core_set_active_for_logging(&sim->core) right after sim_set_virtual_clock(), exactly at the site this entry identified; sched_sim_driver_run() (sim_loop.c) deactivates (xce_core_set_active_for_logging(NULL)) on all 7 of its return paths to avoid the ASAN stack-use-after-scope hazard this entry flagged — confirmed by fixing a real instance of exactly that hazard, caught live when the new test was added (bug-1598, test_sim_event_loop.c's fixtures switched to static storage). No sim-driver fini function was added; the per-return-path deactivation is the shipped mitigation instead.
7. plugins/fcfs.c vs plugins/builtin_fcfs.c split was undocumented in this design until 2026-08-07 (P3 — documentation gap only, now fixed here). Found by /kiro-validate-impl: the file-structure and BuiltinFcfsPicker sections above originally said fcfs.c would be rewritten; the actual implementation added a new builtin_fcfs.c and deliberately left fcfs.c untouched (task 1.2's compat-wrapper exercise target). Functionally correct and confirmed live (builtin_fcfs is the registered default), just wasn't reflected here until this update. Tasks 2.1-2.6 in tasks.md still carry no _Verified: annotations (unlike every other task group) — worth backfilling for traceability.