myxce/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md

Requirements Document

Project Description (Input)

XCE scheduler operators debugging via simulation-mode replay (xce_scheduler -i -s <replay> / batch --sim) are the ones affected. Currently, Recorded-Decision Reconciliation (sim_reconcile_unaccounted_jobs(), sim_loop.c, delivered by the scheduler-sim-decision-parity spec, implementation-complete) only succeeds when a job's recorded DISPATCH decision and its JOB_STATE_CHANGE-to-RUN confirmation land in the exact same replay round -- it gates on job->state == RUN first, then searches that same round's decoded decisions for a match. In any real deployment with genuine dispatch-to-worker-confirmation latency, these two events land in different rounds, so reconciliation never fires for that job -- meaning scheduler-sim-decision-parity does not actually fix bug-1611/bug-1613's dominant real-world manifestation (a job dispatched at round N, confirmed running some rounds later, still shows resInUse=0 / no visible decision when inspected much later in the replay).

Investigation found the live/normal-mode path already gets this right: scheduler.c's sched_commit_decision() sets job->allocs/job->accounts synchronously at DISPATCH-decision time, completely independent of job->state -- and sched_account_jobs()'s own charging logic separately gates on JOB_IS_RUN(job->state) before checking job->accounts != NULL, so a job with accounts already set but still PEND-state is simply not charged yet. This is the correct, already-proven pattern.

What should change: simulation mode's recorded-decision capture should be decoupled from the RUN-state gate the same way, so it works regardless of how many rounds separate a job's DISPATCH decision from its RUN confirmation. Whenever a round's recorded decision stream contains a DISPATCH decision for a job that does not yet have job->accounts set, that decision's allocation data should be captured immediately -- regardless of the job's current state -- rather than waiting for job->state to become RUN before searching for a match. sched_account_jobs()'s existing RUN-state gate continues to defer actual resource charging until the job's state genuinely transitions to RUN (whenever that recorded event is later replayed), so behavior stays correct even though the underlying allocation data was captured earlier. This must not change job->state transition semantics themselves (used elsewhere for job status display, permissions, and preemption logic) -- only how/when the DISPATCH decision's allocation data is captured internally for accounting purposes.

This is a follow-up to the scheduler-sim-decision-parity spec (implementation-complete, all 10 tasks landed and reviewed) -- do not reopen or modify that spec's own requirements/design/tasks. This is its own new spec, building on that feature's existing sim_reconcile_unaccounted_jobs() mechanism and round-boundary wiring.

Boundary Context

• In scope: capturing a job's recorded DISPATCH allocation data as soon as simulation-mode replay encounters it in a round's recorded decision stream, regardless of the job's current state at that point; preserving correct, state-gated resource-in-use accounting once capture no longer depends on same-round state coincidence; making the dispatch decision observable at the historically accurate round; preventing double-capture or fabricated data across this wider round span.
• Out of scope: any change to how or when a job's state itself transitions (PEND/RUN/DONE/etc.) -- state transition timing and triggers are unchanged, only when allocation data is captured for accounting purposes changes; any change to --sim-verify's comparison semantics beyond not producing a new false mismatch as a side effect; any change to simulation mode's live picker (it remains PEND-only for freshly-arriving jobs, unchanged, out of this feature's scope per scheduler-allocation); any change to PEND or PREEMPT decision handling; reopening or modifying scheduler-sim-decision-parity's own requirements, design, or tasks; any change to normal-mode (non-simulation) behavior.
• Adjacent expectations: this feature extends scheduler-sim-decision-parity's existing Recorded-Decision Reconciliation mechanism and round-boundary wiring; it does not replace or duplicate that mechanism. It relies on that feature's decision-reporting channel and resource-accounting service remaining exactly as delivered.

Requirements

Requirement 1: Cross-round resource-accounting accuracy for dispatched jobs

Objective: As an operator inspecting simulation-mode resource state, I want a job's resource consumption reflected correctly regardless of how many rounds separate its dispatch from its running confirmation, so that what I inspect during a replay is trustworthy even under realistic dispatch-to-confirmation latency.

Acceptance Criteria

1. When simulation-mode replay encounters a round containing a job's originating dispatch decision, the scheduler shall capture that decision's resource-allocation data for the job, regardless of the job's current state at that point.
2. While a job's resource-allocation data has been captured but the job has not yet transitioned to a running state, the scheduler shall not include that job's resource consumption in any worker or shared resource-in-use total.
3. When a job whose resource-allocation data was already captured subsequently transitions to a running state, the scheduler shall include that job's resource consumption in its worker's and any shared resource's resource-in-use totals from that point forward, regardless of how many rounds elapsed between capture and the transition.
4. The scheduler shall report the same resource-in-use totals for a dispatched-then-confirmed-later job as normal-mode execution reported at the equivalent point in the original run, regardless of the round separation between dispatch and confirmation.

Requirement 2: Decision visibility at the historically accurate round

Objective: As an operator watching simulation-mode decision output, I want a job's dispatch decision to become visible when the replay reaches the round it was actually decided in, so I have an accurate picture of when scheduling activity happened, not only that it eventually happened.

Acceptance Criteria

1. When simulation-mode replay reaches the round containing a job's originating dispatch decision, the scheduler shall make that decision observable through the existing decision-reporting channel at that round, independent of when the job's state later transitions to running.
2. The scheduler shall preserve the original decision's type and resource-allocation detail when reporting it.
3. When simulation-mode independently computes a fresh decision for a job in a given round, the scheduler shall report only that one decision for the job, not also a duplicate from the recorded stream.

Requirement 3: No fabrication and no double-capture across the wider round span

Objective: As a maintainer relying on this feature's correctness, I want captured allocation data to never be fabricated or captured more than once, so that widening reconciliation's effective round span does not introduce a new class of correctness risk.

Acceptance Criteria

1. The scheduler shall never capture resource-allocation data for a job from anything other than that job's own originating recorded decision.
2. Once a job's resource-allocation data has been captured, the scheduler shall not attempt to capture it again from a later round's recorded decisions, regardless of how many further rounds are replayed.
3. If a job's resource-allocation data was already captured, and simulation-mode's own live decision-making subsequently produces a fresh decision for that job, the scheduler shall treat the job as already resolved and not overwrite or duplicate its already-captured data.

Requirement 4: No regression to existing guarantees

Objective: As a maintainer relying on scheduler-sim-decision-parity's already-delivered guarantees, I want this feature to preserve them, so existing verification workflows, tests, and behavior continue to hold.

Acceptance Criteria

1. The scheduler shall continue to correctly reconcile the same-round dispatch-and-confirmation case that scheduler-sim-decision-parity already handles.
2. The scheduler shall continue to support decision verification mode (comparing freshly-computed decisions against recorded ones) without reporting a mismatch caused by this feature's own behavior.
3. The scheduler shall continue to produce identical decisions between interactive (step-by-step) and batch simulation execution modes.
4. While a job has no recorded originating decision available anywhere in the replay's own data (for example, a job already running before the replay's recording window began), the scheduler shall report that job's resource usage as unavailable rather than fabricating a value, and shall continue processing the remainder of the replay without terminating.
5. While simulation mode is not enabled, the scheduler shall not change its existing resource-accounting, decision-reporting, or job-state-transition behavior.