Rongsong Shen
25e35bacaf
Elevates the normal-mode/simulation-mode driver equivalence guarantee from a one-line architectural description in CLAUDE.md to an enforced steering constraint (scheduler-driver-equivalence.md), backed by bug-1611/bug-1613 as concrete evidence of violation. Scopes the fix as its own spec (scheduler-sim-decision-parity, requirements + design) rather than reopening the closed scheduler-core spec or folding it into scheduler-sim-repl-verbosity's explicitly narrower boundary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
36 lines
6 KiB
Markdown
36 lines
6 KiB
Markdown
# Scheduler Driver Equivalence
|
|
|
|
## Philosophy
|
|
|
|
`scheduler-core` split the scheduling engine into a mode-agnostic **core** state machine (`core.c`/`core_events.c`/`core_session.c`, `xce_sched_core_t`) driven by two interchangeable **drivers**: `driver.c` (controller-connected, capnp-driven, live) and `sim.c`/`sim_loop.c` (standalone, `.replay`-file-driven). The entire value of simulation/replay mode rests on one premise: replaying the exact recording of a normal-mode run reproduces what actually happened, so an operator can debug and tune scheduling against recorded production data and trust the result. `src/scheduler/CLAUDE.md` states this as "one core, two consumers" — this file makes it an enforced constraint, not a description.
|
|
|
|
## DESIGN INVARIANT: State equivalence, not just decision equivalence
|
|
|
|
**Replaying the exact recording of a normal-mode run MUST reproduce identical scheduling decisions AND identical resource-accounting state** (`w->resInUse`, `core->sharedInUse`, `job->accounts`) **at every point in the replay** — not just a matching final decision sequence.
|
|
|
|
Decision-only equivalence is an insufficient test of this invariant. A fixture with no resource contention (every job fits regardless of what `resInUse` says) can produce identical decisions between normal and sim mode while the underlying accounting state has silently diverged — because nothing in that fixture ever depends on `resInUse` being correct. The divergence only becomes decision-visible under contention, which is exactly the scenario most replay-based debugging exists to investigate. **Treat accounting-state parity as load-bearing, not incidental to decision parity.**
|
|
|
|
This is a stricter, broader sibling of the REPL-vs-batch-sim invariant (`--interactive` and batch `--sim` both drive `sched_sim_driver_run()` gated only by `sim->interactive`, proven decision-equivalent by construction and enforced by `test_repl_equivalence.c`). That invariant is solid — both paths share the exact same driver code. Normal-mode-vs-sim-mode equivalence is weaker: they are *separate* driver implementations that must each independently uphold the same per-round contract around the shared core, and nothing mechanically forces them to stay in sync when one changes.
|
|
|
|
## Why a driver can silently violate this
|
|
|
|
The core deliberately does not own accounting. `xce_core_begin_session()`'s resval reset zeroes `resInUse` every session; re-summing it from currently-running jobs (`sched_account_jobs()`) is documented as "real, load-bearing production behavior... `xce_scheduler_t*`-coupled and therefore outside this pure-core module's reach" (`core.h`) — i.e. each driver must call it itself, in the gap between `xce_core_begin_session()` and `xce_core_signal_go()`. `driver.c`'s `xce_sched_driver_run_normal_mode_round()` does this correctly. Nothing forces a second driver implementation to remember to.
|
|
|
|
Similarly, a `ReplaySession` carries a `decisions` list — the recorded decisions from the original live run — decoded every round via `decode_ReplaySession_ptr()`. There is no structural guarantee a driver applies that data (reconstructing `job->allocs`/`job->accounts` for a job already resolved in the recording) rather than only using it for optional comparison.
|
|
|
|
## Known violations (evidence trail)
|
|
|
|
- **bug-1613** (confirmed): `sched_account_jobs()` is called by `driver.c` but never by `sim_loop.c`, which still uses the older monolithic `xce_core_go()` with no interleaving point for it. Compounding this, `sched_account_jobs()`/`sched_charge_account()`/`sched_update_account()`/`sched_update_res_accounts()` are signatured on `xce_scheduler_t*` (the normal-mode-only driver struct) even though their bodies only ever touch `sched->core` — `xce_sched_sim_driver_t` deliberately has no `xce_scheduler_t*` field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. `sched_resolve_job_picker()` went through this exact `xce_scheduler_t*` → `xce_sched_core_t*` relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent.
|
|
- **bug-1611** (suspected same root cause): `sim_loop.c` only reads `decoded_session->decisions` when `sim->verify` is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct state for a job the live picker in this replay isn't independently re-deciding. This plausibly explains both bug-1611 (no `DISPATCH` decision ever visible for such jobs) and the accounting gap in bug-1613 (their usage is never counted, not even transiently) as one underlying gap, not two.
|
|
- Both are tracked in `.wolf/buglog.json`; a dedicated fix spec is scoped separately (see `.kiro/specs/` — do not fix inline as a drive-by patch, per the testing discipline below).
|
|
|
|
## Testing discipline
|
|
|
|
Any change touching resource accounting (`resource.c`), core event application (`core_events.c`), or driver session-boundary handling (`driver.c`, `sim_loop.c`, `core_session.c`) must be validated against **a resource-contention fixture** — multiple jobs competing for capacity insufficient for all of them — not only a fixture where every job fits. A no-contention fixture cannot distinguish "accounting is correct" from "accounting is wrong but nothing currently reads it." Existing coverage in this space (a normal-mode-recording-vs-simulation-replay decision-identity test) evidently does not exercise contention, or bug-1613 would have failed it.
|
|
|
|
When adding or modifying driver-specific per-round logic in either `driver.c` or `sim_loop.c`, check the other driver for the equivalent step before assuming none is needed — the two are not required to look identical, but any core-derived state (`resInUse`, `sharedInUse`, `job->accounts`, decision counts) they produce from equivalent input MUST match.
|
|
|
|
## See Also
|
|
|
|
- `src/scheduler/CLAUDE.md` — architecture overview; "Drivers" section points here for the equivalence contract.
|
|
- `.kiro/specs/scheduler-core/` — original driver-split design and requirements.
|