myxce/.kiro/specs/scheduler-sim-decision-parity/tasks.md

Implementation Plan

• 1. Foundation: shared accounting surface and contention test data

• 1.1 Make resource-in-use accounting callable from either driver

  • Relocate the resource re-accounting service (currently tied to the normal-mode-only driver struct) so it operates on the mode-agnostic core state shared by both drivers, mirroring the same relocation already done for the job-picker resolution service
  • Update all three existing call sites to the new signature: the normal-mode round handler's accounting call, the scheduler's own session-begin path's accounting call, and the scheduler's decision-commit path's call into the job-allocation-commit function that is part of this same relocated group
  • Body of the accounting logic itself is unchanged — this is a pure capability-relocation step, not new behavior
  • Observable completion: the full scheduler test suite still builds and passes unchanged (zero behavior change), and the simulation driver's build now has visibility to call the same accounting service the normal-mode driver already uses
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 5.3
  • Boundary: Core-Level Resource Accounting

• 1.2 (P) Build a resource-contention replay fixture for testing

  • Create a recorded-session test fixture with multiple jobs competing for capacity insufficient to run all of them at once (existing fixtures only ever have enough capacity for every job, which cannot exercise or prove capacity-check correctness)
  • Fixture must be usable both as a normal-mode recording source and as a simulation-mode replay input, so the same scenario can be run through both drivers and compared
  • Observable completion: a scheduling round exists in the fixture where at least one job is denied capacity that a currently-running job is consuming
  • Requirements: 2.1, 2.2
  • Boundary: Scheduler test suite (contention fixture)

• 2. Core: round-boundary accounting and decision reconciliation

• 2.1 Give simulation mode the same round-boundary accounting sequence normal mode uses

  • Replace simulation mode's single combined session-start call with the same two-step sequence (reset accounting, then unblock the picker) the normal-mode driver already uses, opening an interleaving point between them
  • Call the relocated resource re-accounting service (task 1.1) in that interleaving point, in the same relative position the normal-mode driver already calls it
  • Preserve the existing error-handling behavior for a failed session start exactly as it works today
  • Interactive (step-by-step) and batch simulation runs must go through this same call site with no divergence between them
  • Observable completion: a job dispatched fresh within a simulation run (not already resolved before the replay reached it) now shows correct worker and shared resource-in-use totals immediately after the round it was dispatched in, matching what the equivalent normal-mode recording shows at the same point
  • Requirements: 1.1, 1.2, 1.3, 1.4, 5.1, 5.3
  • Depends: 1.1
  • Boundary: Sim Driver Round-Boundary Accounting

• 2.2 Detect and locate already-resolved jobs needing reconciliation

  • Identify, once per round, every currently-running job that has no known resource allocation yet (this is the signature of a job that was already resolved earlier in the original recording, before this replay run's own live decision-making ever considered it)
  • For each such job, search this round's decoded recorded-decision data for the matching originating decision by job identity
  • When no matching recorded decision is found, leave the job's allocation unknown, report its usage as unavailable rather than fabricating a value, and continue processing the rest of the replay without treating this as an error
  • A job left unresolved this round must be safe to re-check again on a later round without any adverse effect
  • Observable completion: given a fixture where a job is already running with no recorded decision available in the replay's own window (e.g. a job dispatched before the snapshot the replay started from), the job is reported as having unavailable usage and the replay continues to completion without error
  • Requirements: 4.1, 4.2
  • Boundary: Recorded-Decision Reconciliation

• 2.3 Reconstruct an already-resolved job's allocation from its recorded decision, safely

  • When a matching recorded dispatch decision is found (task 2.2), take ownership of its resource-allocation data to become the job's own allocation and accounting record — the same ownership-transfer pattern the live dispatch path already uses, not a duplicate copy
  • Once that data has been taken over, remove the now-fully-consumed recorded decision entry from this round's decoded data, releasing only the parts of it that were not taken over, and close the resulting gap so the remaining recorded entries keep their original relative order
  • A job whose allocation is reconstructed this way must never be reconsidered by this same mechanism again on a later round
  • Observable completion: reconciling a job's allocation and then later releasing that job's allocation through its normal end-of-life path is clean under the project's standard memory-safety build (no double-release, no leaked allocation) — this is the task's primary correctness bar, not a peripheral check
  • Requirements: 1.1, 1.2, 3.2
  • Depends: 2.2
  • Boundary: Recorded-Decision Reconciliation

• 2.4 Make a reconciled job's decision visible the same way a fresh one is

  • Feed a reconstructed decision (task 2.3) through the exact same decision-reporting path a freshly-computed decision already uses, preserving its original decision type and resource-allocation detail
  • If simulation mode's own live decision-making independently produces a fresh decision for a job in the same round it would otherwise be reconciled, report only the fresh one — never both
  • Observable completion: replaying a fixture where a job was already dispatched before the replay's live decision-making reaches it now shows that job's dispatch decision in step/continue output, exactly once, with the same resource-allocation detail the original recording captured
  • Requirements: 3.1, 3.3
  • Depends: 2.3
  • Boundary: Recorded-Decision Reconciliation

• 3. Integrate reconciliation into the round boundary and confirm decision-verification stays correct

  • Call the reconciliation mechanism (tasks 2.2-2.4) inside the round-boundary sequence (task 2.1), positioned after the session-start reset and before the resource re-accounting call, so the same round's re-accounting picks up every newly-reconstructed job and the still-pending live decision-making sees correct capacity
  • Confirm the round's decision-verification comparison (which walks the same recorded-decision data positionally) is unaffected by reconciliation having already consumed and removed some of those entries — no special-casing should be needed in the comparison logic itself, only in what data reaches it
  • Observable completion: running decision verification against a recording containing already-resolved jobs reports no mismatch attributable to this feature, and a job whose allocation was reconciled this round is reflected in resource-in-use totals before the live picker/allocator evaluates capacity for any other job in that same round
  • Requirements: 1.3, 1.4, 2.1, 2.2, 5.2
  • Depends: 2.1, 2.4
  • Boundary: Sim Driver Round-Boundary Accounting, Recorded-Decision Reconciliation

• 4. Validation

• 4.1 (P) Prove capacity-aware parity under contention

  • Using the resource-contention fixture (task 1.2), replay it in simulation mode and compare worker and shared resource-in-use totals against the equivalent normal-mode recording at every round
  • Assert simulation mode never makes a scheduling decision (dispatch where normal mode pended, or vice versa) that the normal-mode recording of the same state did not also make
  • Observable completion: the contention fixture replay produces byte-for-byte identical resource-in-use totals and scheduling outcomes to its normal-mode recording, at every round, not just the final one
  • Requirements: 1.4, 2.1, 2.2
  • Depends: 3
  • Boundary: Scheduler test suite (contention parity)

• 4.2 (P) Regression suite: existing guarantees still hold

  • Re-run the existing interactive-vs-batch decision-equivalence coverage unmodified and confirm it still passes
  • Extend decision-verification coverage with a recording containing already-resolved jobs and confirm no mismatch is reported as a result of this feature
  • Confirm a replay starting from a snapshot with jobs already running at snapshot time (no recorded decision available in-window) degrades gracefully rather than crashing or fabricating a value
  • Confirm that with simulation mode not enabled, resource-accounting and decision-reporting behavior is byte-for-byte unchanged from before this feature
  • Observable completion: all four checks above pass in the same test run, with the pre-existing equivalence test's pass/fail behavior unchanged from before this feature
  • Requirements: 4.1, 4.2, 5.1, 5.2, 5.3
  • Depends: 3
  • Boundary: Scheduler test suite (regression/verify coverage)

• 4.3 Live-verify against the original motivating recording

  • Replay the real recorded session that originally surfaced the resource-accounting and decision-visibility gaps this feature fixes, through to completion
  • Observable completion: inspecting a worker mid-replay now shows nonzero resource usage for jobs independently confirmed running via their ongoing usage updates, and their dispatch decisions are now visible in step/continue output where they previously never appeared
  • Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3
  • Depends: 4.1, 4.2
  • Boundary: Scheduler test suite

Implementation Notes

• (task 1.2) Whole-scenario white-box test files in src/scheduler/testsuite/ (fixture construction + real driver/allocator plumbing) routinely exceed the project's 800-line guidance (test_sim_verify.c is 934 lines, uncorrected; the new test_sim_resource_contention.c is 883) — the steering's "don't use as precedent" caveat names only non-test files, so this is an accepted pattern for this class of test, not a defect to fix per task. Tasks 4.1-4.3 will likely add similarly-sized files; don't flag file size alone as blocking for them.
• (task 2.1) Confirmed (by reading sim_picker_thread_fn()): simulation mode's live picker thread unconditionally PENDs every job today, and sched_commit_decision() (the only real setter of job->accounts) is normal-mode-only. So after 2.1 alone, resInUse is still zero end-to-end in sim mode -- 2.1 only wires the call site/interleaving slot. Tasks 2.2-2.4 (Recorded-Decision Reconciliation) are what actually populate job->accounts and make resInUse correct. Don't expect nonzero resInUse from a sim run until 2.2-2.4 land -- this is expected, not a regression.
• (task 2.1) White-box test files that #include "../src/sim_loop.c" directly (test_sim_event_loop.c, test_sim_verify.c, test_sim_alt_events.c, test_sim_record_output.c, test_repl_breakpoints_inspection.c, test_repl_snapshot_params.c, test_repl_execution_control.c, test_repl_equivalence.c) need a link-satisfying stub added whenever sim_loop.c gains a new extern reference (here: sched_account_jobs()). Tasks 2.2-2.4 will likely add more such references (e.g. into the recorded-decision/reporting machinery) -- expect to touch these same 8 files again for stub updates, and check for linker errors rather than guessing which files need it.
• (task 2.3) Do NOT check "not found" via i == decoded_session->n_decisions after the lookup loop once the loop's array can be mutated mid-iteration (e.g. once n_decisions gets decremented by a compaction) -- that check goes stale for a last-index match. Use an explicit found flag instead. This bug would only have surfaced later (task 2.3 introduced the mutation that makes the old check wrong) but was caught and fixed here.
• (task 2.3) src/lib/misc.c's free_host_account() never frees the strdup'd host field make_host_account() (src/lib/xcealloc.c:119) allocates -- a real, pre-existing leak, equally present on the live scheduler.c dispatch/release path, previously masked because this test suite is registered with ASAN_OPTIONS=detect_leaks=0. Logged as bug-1621, intentionally NOT fixed here (out of this feature's boundary -- misc.c isn't owned by scheduler-sim-decision-parity). If a later task's ASAN run surfaces this same leak, it's not a new regression -- don't waste time re-diagnosing it, it's bug-1621.
• (task 2.3) make_job_account()'s return isn't NULL-checked before use, matching the live-path precedent (scheduler.c's sched_commit_decision() has the same gap) -- informational only, not a defect to fix in this feature unless the live path itself gets hardened first.
• (task 2.4) The reconciliation function ended up needing a third parameter (a nullable recording-context handle) beyond the two design.md originally specified, because that handle only exists as a stack-local inside the round-loop and nothing on the driver struct can supply it -- design.md's Service Interface has been updated to match. Task 3 (wiring): the call site sits in the exact same lexical scope where that recording-context variable is already declared, so pass it straight through -- no new plumbing needed.
• (task 2.4) xce_decision_t objects built via make_dispatch_decision()/make_pend_decision() are never freed anywhere in this codebase on any live path (verified by grep across scheduler/src/*.c) -- this is an established, accepted pattern here, not something to "fix" if you see it under detect_leaks=1. Don't add a free for it; that would deviate from the codebase's own convention without a broader cleanup pass.
• (task 3) get_resvals() (resource.c) returns nres (the count filled in) on success, NOT 0 -- a test-authoring bug (bug-1622, not production) came from assuming the latter. If you write a test against get_resvals(), check its actual return-value contract first.
• (task 3) Feature is now end-to-end wired and independently confirmed working (reviewer removed the call site itself and watched both integration tests fail for the right reason, then restored and watched them pass). Tasks 4.1-4.3 (Validation) can now exercise the real, complete path -- no more "not fully wired yet" caveats apply.
• (task 4.1) sim_picker_thread_fn() is confirmed (by direct read, independently re-confirmed by the reviewer) to be structurally PEND-only -- it has no capacity-check/dispatch path for a job it evaluates fresh. This means task 1.2's fixture (or any single-round, freshly-arriving-jobs fixture) can NEVER be literally replayed-and-diffed for full dispatch/PEND parity -- sim's picker always PENDs a fresh job regardless of whether resource accounting is correct, so that comparison would be tautological for freshly-arriving jobs (though still meaningful for reconciled/already-resolved jobs). If a later task needs "real recording vs. sim replay" parity proof, build it the way task 4.1 did: a real driven normal-mode recording plus a hand-crafted sim-mode fixture modeling the identical facts (including at least one already-resolved-before-replay job to exercise reconciliation), not a literal single-file replay-and-diff. This is a real architectural limit, not a testing shortcut -- true dispatch-decision parity for freshly-arriving jobs is scheduler-allocation's (a separate, not-yet-designed spec) job, out of this feature's scope.
• (task 4.2) builtin_fcfs's pend-handling (builtin_fcfs_on_pend/begin_session) never clears a job from pendJobs once it's added -- a still-genuinely-PEND job recurs as a fresh PEND decision in the recorded stream EVERY round/session, not just once. This is pre-existing, unrelated-to-this-feature behavior (also visible in test_repl_equivalence.c's own existing fixture). If you build a multi-round .replay fixture with a job that stays PEND across rounds, expect and account for that job reappearing in every round's decoded decisions -- don't assume a PEND is a one-time event.
• (task 4.2) Not every reconciliation-adjacent test needs to be discriminating via the "disable the reconciliation call site" toggle to be worth keeping -- a test exercising the "not found, nothing to reconcile" branch through the REAL sched_sim_driver_run()/xce_sched_core_load_snapshot() path can be identical whether reconciliation runs or not, and still catch real bugs (fabrication, snapshot-load corruption, crashes) that a hand-built-core unit test (task 2.2's) can't reach. Judge a test's value by what regression classes it actually covers, not solely by one discriminating-power toggle.
• (task 4.3 -- IMPORTANT SCOPE FINDING, confirmed structurally by code reading, not speculation): Recorded-Decision Reconciliation only succeeds when a job's recorded DISPATCH decision and its recorded JOB_STATE_CHANGE-to-RUN event land in the SAME replay round (reconciliation searches only that round's own freshly-decoded decoded_session->decisions; each round's decode is discarded once that round finishes processing). Every test in this feature that exercises the successful-match path (test_sim_round_boundary_reconciliation.c task 3, test_sim_contention_parity.c task 4.1 "Part B", test_sim_e2e_original_motivation.c task 4.3's ANCHOR_ID) had to hand-engineer this exact coincidence -- none occurs from organic, continuously-driven scheduling, because a job cannot be "freshly PEND for this round's live picker" and "already RUN before this round's picker runs" at the same time (builtin_fcfs's dispatch/state-change hooks make these mutually exclusive by construction). In any real deployment with genuine dispatch-to-worker-confirmation latency (the normal case for an async, network-distributed scheduler), a job's DISPATCH decision and its RUN confirmation will typically land in DIFFERENT rounds -- meaning this feature, as designed and implemented, likely does NOT fix bug-1611/bug-1613's dominant real-world manifestation ("a job dispatched at round 32, confirmed running by some later round, still shows resInUse=0/no visible decision when inspected at round 100"). It only fixes the narrower case of same-round dispatch+confirmation coincidence. If bug-1611/1613's common case needs fixing, the follow-up would need reconciliation to search backward across MULTIPLE prior rounds (or maintain a persistent jobid-keyed index of not-yet-consumed recorded decisions across the whole replay, not just the current round) -- a materially different, larger design than what this feature built. Flagged for the user/future spec, not something task 4.3 could fix within its validation-only boundary.