Design went through 3 review rounds (fixed 6 real issues: verify-loop positional alignment, snapshot-restored-job ambiguity, a leak, a double-free/UAF from move-semantics aliasing, a const-qualified interface contradicting its own mutation requirement, and a file-location error in the accounting-function relocation plan). Tasks generated and passed an independent sanity review after one repair round (missing call-site update, missing (P) boundary tag). All phases approved, ready for implementation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
7.9 KiB
Brief: scheduler-sim-decision-parity
Problem
Operators debugging scheduling behavior via xce_scheduler -i -s <replay> (or batch --sim) cannot trust two things about jobs that were already resolved (dispatched) in the original recorded run: (1) their resource usage is never reflected in w->resInUse/core->sharedInUse (info worker always shows used=0.00 regardless of actual load — bug-1613), and (2) their DISPATCH decision is never visible in decision output, even under step (bug-1611). Both undermine the entire premise of simulation mode: that replaying a recording reproduces what actually happened, closely enough to debug and tune against.
Current State
scheduler-core(implementation-complete, closed) built the core/driver split:xce_sched_core_tis mode-agnostic;driver.c(normal mode) andsim_loop.c(simulation mode) are separate driver implementations of the same session/decision-callback protocol.- Confirmed (bug-1613):
sched_account_jobs()— re-derivesresInUse/sharedInUsefrom currently-RUN-state jobs' already-builtjob->accounts, required after every session's resval reset — is called bydriver.c'sxce_sched_driver_run_normal_mode_round()but never bysim_loop.c, which still uses the monolithicxce_core_go()(no interleaving point for it). The function is also currently signatured onxce_scheduler_t*(normal-mode-only struct), whichxce_sched_sim_driver_tdeliberately does not have (Req 5 AC 17 simulation-isolation boundary) — it cannot be called fromsim_loop.ctoday without a signature change. - Confirmed (this session, new finding):
xce_sched_driver_run_normal_mode_round()'s decision-drain loop shows every decision the normal-mode scheduler makes — includingDISPATCH, with its fullxce_job_alloc_tallocation payload — is written into the.replayfile viasched_record_session_add_decision().sim_loop.calready decodes this fulldecisionslist every round (decoded_session->decisions, viadecode_ReplaySession_ptr()), but only ever reads it whensim->verifyis set, purely to compare against freshly-drained live decisions. It is never applied to reconstructjob->allocs/job->accountsfor a job the live picker in this replay run doesn't independently re-decide — which is the normal case for any job already resolved before the replay began. This is very likely the single root cause behind both bug-1611 and bug-1613: the data needed already exists in every replay stream and is already decoded every round; it is simply never applied outside verify-mode comparison. .kiro/steering/scheduler-driver-equivalence.md(created this session) documents the constraint this spec restores: replaying a normal-mode recording must reproduce identical decisions AND identical resource-accounting state, not just matching final decisions on fixtures with no contention.
Desired Outcome
Replaying the exact recording of a normal-mode run reproduces, at every point in the replay: the same resource-accounting state (resInUse/sharedInUse/job->accounts) and the same decision visibility (a DISPATCH/PEND/PREEMPT line is observable for every job, whether the live picker re-derived it fresh this run or it was already resolved in the original recording) that the normal-mode run itself produced.
Approach
Two components, one spec:
- Accounting re-derivation parity: relocate
sched_account_jobs()and its callees (sched_charge_account/sched_uncharge_account/sched_update_account/sched_update_res_accounts) fromxce_scheduler_t*toxce_sched_core_t*— every one of their bodies already only touchessched->core, so this is a signature change, not a logic change, following the exact precedentsched_resolve_job_picker()already established (task 4.3) for the identical reason. Then call it insim_loop.cat the same relative positiondriver.cdoes: replace the monolithicxce_core_go()with the splitxce_core_begin_session()→sched_account_jobs(&(sim->core))→xce_core_signal_go(). - Recorded-decision application: for a round where the live picker/allocator does not independently produce a decision for a job that the recording shows as already resolved, apply the corresponding recorded decision from
decoded_session->decisionsdirectly — reconstructingjob->allocs/job->accounts(mirroring the field datasim_verify_job_alloc_eq()already knows how to compare) and feeding it through the same decision pipelinesim_print_decision()/xce_repl_on_decision()consume, so it is both charged and visible.
Scope
- In: Accounting-state re-derivation parity between the two drivers; applying recorded decisions for already-resolved jobs during replay (both interactive and batch
--sim); avoiding double-application when a job is both recorded-resolved and somehow also independently re-decided; resource-contention test fixtures proving the fix (perscheduler-driver-equivalence.md's testing discipline) since no-contention fixtures cannot distinguish correct from broken accounting. - Out: Any change to REPL console output/formatting (that's
scheduler-sim-repl-verbosity, a sibling spec, explicitly excluded from touching simulation state by its own Non-Goals). Any change to--sim-verify's actual comparison semantics beyond ensuring recorded-decision application doesn't produce a spurious mismatch or double-count against it. Reopening or modifyingscheduler-coreitself. New scheduling/allocation policy.
Boundary Candidates
- Accounting relocation (
resource.c/scheduler.h/core.h, signature-only change) — small, mechanical, low-risk, could land independently of the second piece. - Recorded-decision application (
sim_loop.c, andsim_verify.cfor the double-count/verify-interaction question) — larger, the real design surface: what "already resolved, not independently re-decided" means precisely, and how it composes with--sim-verify's existing comparison.
Out of Boundary
scheduler-sim-repl-verbosity's REPL output/formatting work.scheduler-sim-diagnostics's future new inspection commands (limit usage, pend-reason) — this spec restores existing data's correctness, it does not add new inspectable data.- Whether
sched_check_host_satisfy/sched_meet_host_selectionreadresInUsefor live capacity decisions — CONFIRMED during design (seeresearch.md's Research Log): they do, unconditionally. This is existingscheduler-corebehavior this spec does not change, only relies on.
Upstream / Downstream
- Upstream:
scheduler-core's core/driver split and its now-documented equivalence invariant (.kiro/steering/scheduler-driver-equivalence.md). - Downstream:
scheduler-sim-repl-verbosity's round-rollup dispatch counts (currently designed to read from the existing decision tally) become accurate once this lands, rather than permanently under-reporting dispatches for already-resolved jobs.scheduler-allocation/scheduler-preemption(both requirements-generated, not yet designed) inherit a scheduler core with a restored, tested equivalence guarantee to build on.
Existing Spec Touchpoints
- Extends: none formally —
scheduler-coreis closed; this spec restores its own documented invariant rather than reopening it. - Adjacent:
scheduler-sim-repl-verbosity(sibling, not yet implemented — benefits from this fix but does not depend on it to ship),scheduler-sim-diagnostics(sibling, brief only).
Constraints
- Must not regress REPL-vs-batch-sim decision equivalence (
test_repl_equivalence.c) — that invariant is solid today and must stay solid. - Must be validated against a resource-contention fixture (multiple jobs competing for insufficient capacity), per
scheduler-driver-equivalence.md's testing discipline — a no-contention fixture cannot prove this fix works. sched_account_jobs()'s relocation must be a pure signature change (no behavior change) to keep the accounting-parity piece low-risk and independently reviewable from the larger recorded-decision-application piece.