From d9d0613598ef9c238401ca854526039f4e55aa2c Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 8 Aug 2026 03:21:57 +0000 Subject: [PATCH] docs: add scheduler recording/simulation/REPL to deployment guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "Deploying the Scheduler" section predated scheduler-core and only covered normal-mode startup. Adds session recording (--record-dir, xceadmin scheduler record start|stop|status, filename validation), xceadmin scheduler dump for offline .replay inspection, and simulation mode (--sim, --sim-speed, --sim-verify, --sim-alt-events, --sim-output, --interactive REPL) — all verified against the actual argp help output and xceadmin cli.py source, not assumed. Co-Authored-By: Claude Sonnet 5 --- src/docs/deployment.md | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/src/docs/deployment.md b/src/docs/deployment.md index 82f3b8d4..ab6f726b 100644 --- a/src/docs/deployment.md +++ b/src/docs/deployment.md @@ -175,7 +175,46 @@ xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.lo - **Gotcha**: the default controller target is hard-coded to `localhost:9990`. There is no `-m`/`-p` equivalent — the *only* way to point the scheduler at any other host or port, even a single non-HA controller, is `--peers`. Forgetting `--peers` silently targets `localhost:9990` regardless of where your controller actually is. - `-v` is repeatable for more verbose logging; default log file is `/tmp/xce_scheduler.log`. - No TLS — plaintext to the controller, same as the worker. -- Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary, not dynamically loaded plugins. +- Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary through the picker plugin interface (`builtin_fcfs` is the registered default), not dynamically loaded. + +### Session recording (live cluster) + +Every normal-mode scheduler can record its scheduling sessions to a `.replay` file for later offline replay/debugging — off by default, no overhead when disabled: + +```bash +xce_scheduler --peers "ctrl-1=controller-host:9990" --record-dir /var/log/xce/scheduler-recordings -v +``` + +- `--record-dir DIR` sets where recordings are written (default `/var/log/xce/`). This is the **only** way `record_dir` is provisioned — there is no per-request override. +- Recording itself is controlled at runtime, not via a startup flag, through the controller: + ```bash + xceadmin scheduler record start [FILENAME] --controller 127.0.0.1:9991 # filename optional, auto-generated if omitted + xceadmin scheduler record status --controller 127.0.0.1:9991 + xceadmin scheduler record stop --controller 127.0.0.1:9991 + ``` + These go through a new `OPCode_schedAdmin` request the controller pass-through-forwards to its attached scheduler (`xceadmin` → controller → scheduler → reply, same primary-redirect behavior as `config push`). The controller holds no recording state itself. +- `FILENAME` (if given) must be a bare filename — no `/` and no `..` segment. The scheduler resolves it under `--record-dir` and rejects anything that would escape that directory (`"invalid record file name"`). +- A `.replay` file is a single self-contained Cap'n Proto binary stream (header + snapshot + interleaved events/decisions + periodic checkpoints) — inspect one offline, no running cluster needed: + ```bash + xceadmin scheduler dump /var/log/xce/scheduler-recordings/some-session.replay + ``` + +### Simulation mode (offline debugging, no controller connection) + +`--sim` runs the scheduler as a **standalone replay tool** against a previously recorded (or hand-built) `.replay` file — zero connection to any controller, safe to run against a production recording on a laptop: + +```bash +xce_scheduler --sim /path/to/session.replay --sim-speed 0 # replay as fast as possible +xce_scheduler --sim /path/to/session.replay --sim-verify # replay + diff against the recorded decisions +xce_scheduler --sim /path/to/session.replay --sim-output /tmp/resim.replay --sim-alt-events /path/to/edited-events.replay # "what-if": substitute events, record the new decisions +xce_scheduler --sim /path/to/session.replay --interactive # step through it interactively (REPL) +``` + +- `--sim-speed`: `0` = as-fast-as-possible, `1` = real-time (default), `N>1` = `N`x real-time. +- `--sim-verify` re-derives decisions from the recorded events and asserts they match the decisions the file already recorded — the feature's core "deterministic replay" regression check, usable ad hoc against any recording. +- `--sim-alt-events FILE` swaps in a different event stream after the initial snapshot for what-if branching; `--sim-output FILE` records the resulting decisions as a new `.replay` file. +- `--interactive` opens a REPL (`step`, `continue`, breakpoints, state inspection, snapshot capture) driving the *same* simulation run a batch `--sim` would — REPL and batch produce byte-identical decisions for the same input, only the pacing differs. +- Simulation mode never opens a socket to any controller and never touches `--peers`/`--record-dir` — see `.kiro/specs/scheduler-core/design.md`'s Boundary Commitments for the full isolation contract. ## Deploying apiserver