myxce/.kiro/specs/scheduler-sim-decision-parity/brief.md
Rongsong Shen 0b2c856d8b
docs(scheduler-sim-decision-parity): finalize design and generate tasks
Design went through 3 review rounds (fixed 6 real issues: verify-loop
positional alignment, snapshot-restored-job ambiguity, a leak, a
double-free/UAF from move-semantics aliasing, a const-qualified interface
contradicting its own mutation requirement, and a file-location error in
the accounting-function relocation plan). Tasks generated and passed an
independent sanity review after one repair round (missing call-site update,
missing (P) boundary tag). All phases approved, ready for implementation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-22 05:30:49 +00:00

7.9 KiB

Brief: scheduler-sim-decision-parity

Problem

Operators debugging scheduling behavior via xce_scheduler -i -s <replay> (or batch --sim) cannot trust two things about jobs that were already resolved (dispatched) in the original recorded run: (1) their resource usage is never reflected in w->resInUse/core->sharedInUse (info worker always shows used=0.00 regardless of actual load — bug-1613), and (2) their DISPATCH decision is never visible in decision output, even under step (bug-1611). Both undermine the entire premise of simulation mode: that replaying a recording reproduces what actually happened, closely enough to debug and tune against.

Current State

  • scheduler-core (implementation-complete, closed) built the core/driver split: xce_sched_core_t is mode-agnostic; driver.c (normal mode) and sim_loop.c (simulation mode) are separate driver implementations of the same session/decision-callback protocol.
  • Confirmed (bug-1613): sched_account_jobs() — re-derives resInUse/sharedInUse from currently-RUN-state jobs' already-built job->accounts, required after every session's resval reset — is called by driver.c's xce_sched_driver_run_normal_mode_round() but never by sim_loop.c, which still uses the monolithic xce_core_go() (no interleaving point for it). The function is also currently signatured on xce_scheduler_t* (normal-mode-only struct), which xce_sched_sim_driver_t deliberately does not have (Req 5 AC 17 simulation-isolation boundary) — it cannot be called from sim_loop.c today without a signature change.
  • Confirmed (this session, new finding): xce_sched_driver_run_normal_mode_round()'s decision-drain loop shows every decision the normal-mode scheduler makes — including DISPATCH, with its full xce_job_alloc_t allocation payload — is written into the .replay file via sched_record_session_add_decision(). sim_loop.c already decodes this full decisions list every round (decoded_session->decisions, via decode_ReplaySession_ptr()), but only ever reads it when sim->verify is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct job->allocs/job->accounts for a job the live picker in this replay run doesn't independently re-decide — which is the normal case for any job already resolved before the replay began. This is very likely the single root cause behind both bug-1611 and bug-1613: the data needed already exists in every replay stream and is already decoded every round; it is simply never applied outside verify-mode comparison.
  • .kiro/steering/scheduler-driver-equivalence.md (created this session) documents the constraint this spec restores: replaying a normal-mode recording must reproduce identical decisions AND identical resource-accounting state, not just matching final decisions on fixtures with no contention.

Desired Outcome

Replaying the exact recording of a normal-mode run reproduces, at every point in the replay: the same resource-accounting state (resInUse/sharedInUse/job->accounts) and the same decision visibility (a DISPATCH/PEND/PREEMPT line is observable for every job, whether the live picker re-derived it fresh this run or it was already resolved in the original recording) that the normal-mode run itself produced.

Approach

Two components, one spec:

  1. Accounting re-derivation parity: relocate sched_account_jobs() and its callees (sched_charge_account/sched_uncharge_account/sched_update_account/sched_update_res_accounts) from xce_scheduler_t* to xce_sched_core_t* — every one of their bodies already only touches sched->core, so this is a signature change, not a logic change, following the exact precedent sched_resolve_job_picker() already established (task 4.3) for the identical reason. Then call it in sim_loop.c at the same relative position driver.c does: replace the monolithic xce_core_go() with the split xce_core_begin_session() → sched_account_jobs(&(sim->core)) → xce_core_signal_go().
  2. Recorded-decision application: for a round where the live picker/allocator does not independently produce a decision for a job that the recording shows as already resolved, apply the corresponding recorded decision from decoded_session->decisions directly — reconstructing job->allocs/job->accounts (mirroring the field data sim_verify_job_alloc_eq() already knows how to compare) and feeding it through the same decision pipeline sim_print_decision()/xce_repl_on_decision() consume, so it is both charged and visible.

Scope

  • In: Accounting-state re-derivation parity between the two drivers; applying recorded decisions for already-resolved jobs during replay (both interactive and batch --sim); avoiding double-application when a job is both recorded-resolved and somehow also independently re-decided; resource-contention test fixtures proving the fix (per scheduler-driver-equivalence.md's testing discipline) since no-contention fixtures cannot distinguish correct from broken accounting.
  • Out: Any change to REPL console output/formatting (that's scheduler-sim-repl-verbosity, a sibling spec, explicitly excluded from touching simulation state by its own Non-Goals). Any change to --sim-verify's actual comparison semantics beyond ensuring recorded-decision application doesn't produce a spurious mismatch or double-count against it. Reopening or modifying scheduler-core itself. New scheduling/allocation policy.

Boundary Candidates

  • Accounting relocation (resource.c/scheduler.h/core.h, signature-only change) — small, mechanical, low-risk, could land independently of the second piece.
  • Recorded-decision application (sim_loop.c, and sim_verify.c for the double-count/verify-interaction question) — larger, the real design surface: what "already resolved, not independently re-decided" means precisely, and how it composes with --sim-verify's existing comparison.

Out of Boundary

  • scheduler-sim-repl-verbosity's REPL output/formatting work.
  • scheduler-sim-diagnostics's future new inspection commands (limit usage, pend-reason) — this spec restores existing data's correctness, it does not add new inspectable data.
  • Whether sched_check_host_satisfy/sched_meet_host_selection read resInUse for live capacity decisions — CONFIRMED during design (see research.md's Research Log): they do, unconditionally. This is existing scheduler-core behavior this spec does not change, only relies on.

Upstream / Downstream

  • Upstream: scheduler-core's core/driver split and its now-documented equivalence invariant (.kiro/steering/scheduler-driver-equivalence.md).
  • Downstream: scheduler-sim-repl-verbosity's round-rollup dispatch counts (currently designed to read from the existing decision tally) become accurate once this lands, rather than permanently under-reporting dispatches for already-resolved jobs. scheduler-allocation/scheduler-preemption (both requirements-generated, not yet designed) inherit a scheduler core with a restored, tested equivalence guarantee to build on.

Existing Spec Touchpoints

  • Extends: none formally — scheduler-core is closed; this spec restores its own documented invariant rather than reopening it.
  • Adjacent: scheduler-sim-repl-verbosity (sibling, not yet implemented — benefits from this fix but does not depend on it to ship), scheduler-sim-diagnostics (sibling, brief only).

Constraints

  • Must not regress REPL-vs-batch-sim decision equivalence (test_repl_equivalence.c) — that invariant is solid today and must stay solid.
  • Must be validated against a resource-contention fixture (multiple jobs competing for insufficient capacity), per scheduler-driver-equivalence.md's testing discipline — a no-contention fixture cannot prove this fix works.
  • sched_account_jobs()'s relocation must be a pure signature change (no behavior change) to keep the accounting-parity piece low-risk and independently reviewable from the larger recorded-decision-application piece.