docs(scheduler): correct stale bug-1611 status and driver-equivalence steering

The dimension-1 audit (previous commit) reported bug-1611 as only
"partially fixed" -- but that assessment worked from bug-1611's own
stale narrative text without checking scheduler-sim-eager-reconciliation
(a separate spec, completed 2026-09-11, predating this session's audit).
That spec already decoupled sim_reconcile_unaccounted_jobs()'s trigger
from the JOB_IS_RUN gate, so a job's DISPATCH decision is captured in
the same round it is recorded, never deferred to a later confirmation
event -- closing bug-1611's original motivating scenario for good.

The "residual gap" my own first correction pass invented (an
already-RUN-at-snapshot-load job) is also not real: such a job's
allocation comes from the snapshot directly (confirmed field-complete
by this session's dimension-4 audit), so it's never a reconciliation
candidate in the first place.

Corrected bug-1611's buglog entry to FIXED and updated
scheduler-driver-equivalence.md's stale "Known violations" section
(still described both bug-1611 and bug-1613 as open) to reflect current
reality, with a pointer to re-verify rather than re-trust either framing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC

2026-09-19 14:01:09 +00:00

6.9 KiB

Raw Blame History

Scheduler Driver Equivalence

Philosophy

scheduler-core split the scheduling engine into a mode-agnostic core state machine (core.c/core_events.c/core_session.c, xce_sched_core_t) driven by two interchangeable drivers: driver.c (controller-connected, capnp-driven, live) and sim.c/sim_loop.c (standalone, .replay-file-driven). The entire value of simulation/replay mode rests on one premise: replaying the exact recording of a normal-mode run reproduces what actually happened, so an operator can debug and tune scheduling against recorded production data and trust the result. src/scheduler/CLAUDE.md states this as "one core, two consumers" — this file makes it an enforced constraint, not a description.

DESIGN INVARIANT: State equivalence, not just decision equivalence

Replaying the exact recording of a normal-mode run MUST reproduce identical scheduling decisions AND identical resource-accounting state (w->resInUse, core->sharedInUse, job->accounts) at every point in the replay — not just a matching final decision sequence.

Decision-only equivalence is an insufficient test of this invariant. A fixture with no resource contention (every job fits regardless of what resInUse says) can produce identical decisions between normal and sim mode while the underlying accounting state has silently diverged — because nothing in that fixture ever depends on resInUse being correct. The divergence only becomes decision-visible under contention, which is exactly the scenario most replay-based debugging exists to investigate. Treat accounting-state parity as load-bearing, not incidental to decision parity.

This is a stricter, broader sibling of the REPL-vs-batch-sim invariant (--interactive and batch --sim both drive sched_sim_driver_run() gated only by sim->interactive, proven decision-equivalent by construction and enforced by test_repl_equivalence.c). That invariant is solid — both paths share the exact same driver code. Normal-mode-vs-sim-mode equivalence is weaker: they are separate driver implementations that must each independently uphold the same per-round contract around the shared core, and nothing mechanically forces them to stay in sync when one changes.

Why a driver can silently violate this

The core deliberately does not own accounting. xce_core_begin_session()'s resval reset zeroes resInUse every session; re-summing it from currently-running jobs (sched_account_jobs()) is documented as "real, load-bearing production behavior... xce_scheduler_t*-coupled and therefore outside this pure-core module's reach" (core.h) — i.e. each driver must call it itself, in the gap between xce_core_begin_session() and xce_core_signal_go(). driver.c's xce_sched_driver_run_normal_mode_round() does this correctly. Nothing forces a second driver implementation to remember to.

Similarly, a ReplaySession carries a decisions list — the recorded decisions from the original live run — decoded every round via decode_ReplaySession_ptr(). There is no structural guarantee a driver applies that data (reconstructing job->allocs/job->accounts for a job already resolved in the recording) rather than only using it for optional comparison.

Known violations (evidence trail)

UPDATE 2026-09-19 (/xce-audit-invariants scheduler, re-run after scheduler-sim-live-allocation): both violations below are now FIXED. The scheduler-sim-live-allocation feature made driver.c's and sim_loop.c's per-round sequence call the identical, single-implementation election/allocation/accounting chain (see .kiro/specs/scheduler-sim-live-allocation/); sim_reconcile_unaccounted_jobs() was independently narrowed to a timing-gap safety net whose own trigger no longer requires a job to have reached JOB_IS_RUN (scheduler-sim-eager-reconciliation, completed 2026-09-11, predating this update) — closing bug-1611's original real-world scenario (a dispatch and its confirmation landing many rounds apart) for good. See .wolf/buglog.json's bug-1611/bug-1613 entries for full evidence. This section is kept as a historical record of the violation and its resolution — do not re-flag either as open without first re-verifying sim_loop.c's current per-round sequence against driver.c's, per the Testing discipline below.

bug-1613 (FIXED, 2026-08-24 / confirmed still fixed 2026-09-19): sched_account_jobs() is called by driver.c but never by sim_loop.c, which still uses the older monolithic xce_core_go() with no interleaving point for it. Compounding this, sched_account_jobs()/sched_charge_account()/sched_update_account()/sched_update_res_accounts() are signatured on xce_scheduler_t* (the normal-mode-only driver struct) even though their bodies only ever touch sched->core — xce_sched_sim_driver_t deliberately has no xce_scheduler_t* field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. sched_resolve_job_picker() went through this exact xce_scheduler_t* → xce_sched_core_t* relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent.
bug-1611 (FIXED, 2026-09-11 via scheduler-sim-eager-reconciliation, general case additionally superseded by scheduler-sim-live-allocation's live election/allocation path): sim_loop.c only read decoded_session->decisions when sim->verify was set, purely to compare against freshly-drained live decisions — never applied to reconstruct state for a job the live picker in that replay wasn't independently re-deciding. Same underlying root cause as bug-1613 (both traced to sim_loop.c not mirroring a per-round obligation driver.c upholds).

Testing discipline

Any change touching resource accounting (resource.c), core event application (core_events.c), or driver session-boundary handling (driver.c, sim_loop.c, core_session.c) must be validated against a resource-contention fixture — multiple jobs competing for capacity insufficient for all of them — not only a fixture where every job fits. A no-contention fixture cannot distinguish "accounting is correct" from "accounting is wrong but nothing currently reads it." Existing coverage in this space (a normal-mode-recording-vs-simulation-replay decision-identity test) evidently does not exercise contention, or bug-1613 would have failed it.

When adding or modifying driver-specific per-round logic in either driver.c or sim_loop.c, check the other driver for the equivalent step before assuming none is needed — the two are not required to look identical, but any core-derived state (resInUse, sharedInUse, job->accounts, decision counts) they produce from equivalent input MUST match.

6.9 KiB Raw Blame History