myxce/.kiro/steering/scheduler-driver-equivalence.md

Scheduler Driver Equivalence

Philosophy

scheduler-core split the scheduling engine into a mode-agnostic core state machine (core.c/core_events.c/core_session.c, xce_sched_core_t) driven by two interchangeable drivers: driver.c (controller-connected, capnp-driven, live) and sim.c/sim_loop.c (standalone, .replay-file-driven). The entire value of simulation/replay mode rests on one premise: replaying the exact recording of a normal-mode run reproduces what actually happened, so an operator can debug and tune scheduling against recorded production data and trust the result. src/scheduler/CLAUDE.md states this as "one core, two consumers" — this file makes it an enforced constraint, not a description.

DESIGN INVARIANT: State equivalence, not just decision equivalence

Replaying the exact recording of a normal-mode run MUST reproduce identical scheduling decisions AND identical resource-accounting state (w->resInUse, core->sharedInUse, job->accounts) at every point in the replay — not just a matching final decision sequence.

Decision-only equivalence is an insufficient test of this invariant. A fixture with no resource contention (every job fits regardless of what resInUse says) can produce identical decisions between normal and sim mode while the underlying accounting state has silently diverged — because nothing in that fixture ever depends on resInUse being correct. The divergence only becomes decision-visible under contention, which is exactly the scenario most replay-based debugging exists to investigate. Treat accounting-state parity as load-bearing, not incidental to decision parity.

This is a stricter, broader sibling of the REPL-vs-batch-sim invariant (--interactive and batch --sim both drive sched_sim_driver_run() gated only by sim->interactive, proven decision-equivalent by construction and enforced by test_repl_equivalence.c). That invariant is solid — both paths share the exact same driver code. Normal-mode-vs-sim-mode equivalence is weaker: they are separate driver implementations that must each independently uphold the same per-round contract around the shared core, and nothing mechanically forces them to stay in sync when one changes.

Why a driver can silently violate this

The core deliberately does not own accounting. xce_core_begin_session()'s resval reset zeroes resInUse every session; re-summing it from currently-running jobs (sched_account_jobs()) is documented as "real, load-bearing production behavior... xce_scheduler_t*-coupled and therefore outside this pure-core module's reach" (core.h) — i.e. each driver must call it itself, in the gap between xce_core_begin_session() and xce_core_signal_go(). driver.c's xce_sched_driver_run_normal_mode_round() does this correctly. Nothing forces a second driver implementation to remember to.

Similarly, a ReplaySession carries a decisions list — the recorded decisions from the original live run — decoded every round via decode_ReplaySession_ptr(). There is no structural guarantee a driver applies that data (reconstructing job->allocs/job->accounts for a job already resolved in the recording) rather than only using it for optional comparison.

Known violations (evidence trail)

• bug-1613 (confirmed): sched_account_jobs() is called by driver.c but never by sim_loop.c, which still uses the older monolithic xce_core_go() with no interleaving point for it. Compounding this, sched_account_jobs()/sched_charge_account()/sched_update_account()/sched_update_res_accounts() are signatured on xce_scheduler_t* (the normal-mode-only driver struct) even though their bodies only ever touch sched->core — xce_sched_sim_driver_t deliberately has no xce_scheduler_t* field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. sched_resolve_job_picker() went through this exact xce_scheduler_t* → xce_sched_core_t* relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent.
• bug-1611 (suspected same root cause): sim_loop.c only reads decoded_session->decisions when sim->verify is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct state for a job the live picker in this replay isn't independently re-deciding. This plausibly explains both bug-1611 (no DISPATCH decision ever visible for such jobs) and the accounting gap in bug-1613 (their usage is never counted, not even transiently) as one underlying gap, not two.
• Both are tracked in .wolf/buglog.json; a dedicated fix spec is scoped separately (see .kiro/specs/ — do not fix inline as a drive-by patch, per the testing discipline below).

Testing discipline

Any change touching resource accounting (resource.c), core event application (core_events.c), or driver session-boundary handling (driver.c, sim_loop.c, core_session.c) must be validated against a resource-contention fixture — multiple jobs competing for capacity insufficient for all of them — not only a fixture where every job fits. A no-contention fixture cannot distinguish "accounting is correct" from "accounting is wrong but nothing currently reads it." Existing coverage in this space (a normal-mode-recording-vs-simulation-replay decision-identity test) evidently does not exercise contention, or bug-1613 would have failed it.

When adding or modifying driver-specific per-round logic in either driver.c or sim_loop.c, check the other driver for the equivalent step before assuming none is needed — the two are not required to look identical, but any core-derived state (resInUse, sharedInUse, job->accounts, decision counts) they produce from equivalent input MUST match.

See Also

• src/scheduler/CLAUDE.md — architecture overview; "Drivers" section points here for the equivalence contract.
• .kiro/specs/scheduler-core/ — original driver-split design and requirements.