Rongsong Shen
984a686974
The dimension-1 audit (previous commit) reported bug-1611 as only "partially fixed" -- but that assessment worked from bug-1611's own stale narrative text without checking scheduler-sim-eager-reconciliation (a separate spec, completed 2026-09-11, predating this session's audit). That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in the same round it is recorded, never deferred to a later confirmation event -- closing bug-1611's original motivating scenario for good. The "residual gap" my own first correction pass invented (an already-RUN-at-snapshot-load job) is also not real: such a job's allocation comes from the snapshot directly (confirmed field-complete by this session's dimension-4 audit), so it's never a reconciliation candidate in the first place. Corrected bug-1611's buglog entry to FIXED and updated scheduler-driver-equivalence.md's stale "Known violations" section (still described both bug-1611 and bug-1613 as open) to reflect current reality, with a pointer to re-verify rather than re-trust either framing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
37 lines
6.9 KiB
Markdown
37 lines
6.9 KiB
Markdown
# Scheduler Driver Equivalence
|
|
|
|
## Philosophy
|
|
|
|
`scheduler-core` split the scheduling engine into a mode-agnostic **core** state machine (`core.c`/`core_events.c`/`core_session.c`, `xce_sched_core_t`) driven by two interchangeable **drivers**: `driver.c` (controller-connected, capnp-driven, live) and `sim.c`/`sim_loop.c` (standalone, `.replay`-file-driven). The entire value of simulation/replay mode rests on one premise: replaying the exact recording of a normal-mode run reproduces what actually happened, so an operator can debug and tune scheduling against recorded production data and trust the result. `src/scheduler/CLAUDE.md` states this as "one core, two consumers" — this file makes it an enforced constraint, not a description.
|
|
|
|
## DESIGN INVARIANT: State equivalence, not just decision equivalence
|
|
|
|
**Replaying the exact recording of a normal-mode run MUST reproduce identical scheduling decisions AND identical resource-accounting state** (`w->resInUse`, `core->sharedInUse`, `job->accounts`) **at every point in the replay** — not just a matching final decision sequence.
|
|
|
|
Decision-only equivalence is an insufficient test of this invariant. A fixture with no resource contention (every job fits regardless of what `resInUse` says) can produce identical decisions between normal and sim mode while the underlying accounting state has silently diverged — because nothing in that fixture ever depends on `resInUse` being correct. The divergence only becomes decision-visible under contention, which is exactly the scenario most replay-based debugging exists to investigate. **Treat accounting-state parity as load-bearing, not incidental to decision parity.**
|
|
|
|
This is a stricter, broader sibling of the REPL-vs-batch-sim invariant (`--interactive` and batch `--sim` both drive `sched_sim_driver_run()` gated only by `sim->interactive`, proven decision-equivalent by construction and enforced by `test_repl_equivalence.c`). That invariant is solid — both paths share the exact same driver code. Normal-mode-vs-sim-mode equivalence is weaker: they are *separate* driver implementations that must each independently uphold the same per-round contract around the shared core, and nothing mechanically forces them to stay in sync when one changes.
|
|
|
|
## Why a driver can silently violate this
|
|
|
|
The core deliberately does not own accounting. `xce_core_begin_session()`'s resval reset zeroes `resInUse` every session; re-summing it from currently-running jobs (`sched_account_jobs()`) is documented as "real, load-bearing production behavior... `xce_scheduler_t*`-coupled and therefore outside this pure-core module's reach" (`core.h`) — i.e. each driver must call it itself, in the gap between `xce_core_begin_session()` and `xce_core_signal_go()`. `driver.c`'s `xce_sched_driver_run_normal_mode_round()` does this correctly. Nothing forces a second driver implementation to remember to.
|
|
|
|
Similarly, a `ReplaySession` carries a `decisions` list — the recorded decisions from the original live run — decoded every round via `decode_ReplaySession_ptr()`. There is no structural guarantee a driver applies that data (reconstructing `job->allocs`/`job->accounts` for a job already resolved in the recording) rather than only using it for optional comparison.
|
|
|
|
## Known violations (evidence trail)
|
|
|
|
**UPDATE 2026-09-19 (`/xce-audit-invariants scheduler`, re-run after `scheduler-sim-live-allocation`): both violations below are now FIXED.** The `scheduler-sim-live-allocation` feature made `driver.c`'s and `sim_loop.c`'s per-round sequence call the identical, single-implementation election/allocation/accounting chain (see `.kiro/specs/scheduler-sim-live-allocation/`); `sim_reconcile_unaccounted_jobs()` was independently narrowed to a timing-gap safety net whose own trigger no longer requires a job to have reached `JOB_IS_RUN` (`scheduler-sim-eager-reconciliation`, completed 2026-09-11, predating this update) — closing bug-1611's original real-world scenario (a dispatch and its confirmation landing many rounds apart) for good. See `.wolf/buglog.json`'s bug-1611/bug-1613 entries for full evidence. This section is kept as a historical record of the violation and its resolution — do not re-flag either as open without first re-verifying `sim_loop.c`'s current per-round sequence against `driver.c`'s, per the Testing discipline below.
|
|
|
|
- **bug-1613** (FIXED, 2026-08-24 / confirmed still fixed 2026-09-19): `sched_account_jobs()` is called by `driver.c` but never by `sim_loop.c`, which still uses the older monolithic `xce_core_go()` with no interleaving point for it. Compounding this, `sched_account_jobs()`/`sched_charge_account()`/`sched_update_account()`/`sched_update_res_accounts()` are signatured on `xce_scheduler_t*` (the normal-mode-only driver struct) even though their bodies only ever touch `sched->core` — `xce_sched_sim_driver_t` deliberately has no `xce_scheduler_t*` field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. `sched_resolve_job_picker()` went through this exact `xce_scheduler_t*` → `xce_sched_core_t*` relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent.
|
|
- **bug-1611** (FIXED, 2026-09-11 via `scheduler-sim-eager-reconciliation`, general case additionally superseded by `scheduler-sim-live-allocation`'s live election/allocation path): `sim_loop.c` only read `decoded_session->decisions` when `sim->verify` was set, purely to compare against freshly-drained live decisions — never applied to reconstruct state for a job the live picker in that replay wasn't independently re-deciding. Same underlying root cause as bug-1613 (both traced to `sim_loop.c` not mirroring a per-round obligation `driver.c` upholds).
|
|
|
|
## Testing discipline
|
|
|
|
Any change touching resource accounting (`resource.c`), core event application (`core_events.c`), or driver session-boundary handling (`driver.c`, `sim_loop.c`, `core_session.c`) must be validated against **a resource-contention fixture** — multiple jobs competing for capacity insufficient for all of them — not only a fixture where every job fits. A no-contention fixture cannot distinguish "accounting is correct" from "accounting is wrong but nothing currently reads it." Existing coverage in this space (a normal-mode-recording-vs-simulation-replay decision-identity test) evidently does not exercise contention, or bug-1613 would have failed it.
|
|
|
|
When adding or modifying driver-specific per-round logic in either `driver.c` or `sim_loop.c`, check the other driver for the equivalent step before assuming none is needed — the two are not required to look identical, but any core-derived state (`resInUse`, `sharedInUse`, `job->accounts`, decision counts) they produce from equivalent input MUST match.
|
|
|
|
## See Also
|
|
|
|
- `src/scheduler/CLAUDE.md` — architecture overview; "Drivers" section points here for the equivalence contract.
|
|
- `.kiro/specs/scheduler-core/` — original driver-split design and requirements.
|