myxce/.kiro/specs/scheduler-sim-eager-reconciliation/requirements.md

Requirements Document

Project Description (Input)

XCE scheduler operators debugging via simulation-mode replay (xce_scheduler -i -s <replay> / batch --sim) are the ones affected. Currently, Recorded-Decision Reconciliation (sim_reconcile_unaccounted_jobs(), sim_loop.c, delivered by the scheduler-sim-decision-parity spec, implementation-complete) only succeeds when a job's recorded DISPATCH decision and its JOB_STATE_CHANGE-to-RUN confirmation land in the exact same replay round -- it gates on job->state == RUN first, then searches that same round's decoded decisions for a match. In any real deployment with genuine dispatch-to-worker-confirmation latency, these two events land in different rounds, so reconciliation never fires for that job -- meaning scheduler-sim-decision-parity does not actually fix bug-1611/bug-1613's dominant real-world manifestation (a job dispatched at round N, confirmed running some rounds later, still shows resInUse=0 / no visible decision when inspected much later in the replay).

Investigation found the live/normal-mode path already gets this right: scheduler.c's sched_commit_decision() sets job->allocs/job->accounts synchronously at DISPATCH-decision time, completely independent of job->state -- and sched_account_jobs()'s own charging logic separately gates on JOB_IS_RUN(job->state) before checking job->accounts != NULL, so a job with accounts already set but still PEND-state is simply not charged yet. This is the correct, already-proven pattern.

What should change: simulation mode's recorded-decision capture should be decoupled from the RUN-state gate the same way, so it works regardless of how many rounds separate a job's DISPATCH decision from its RUN confirmation. Whenever a round's recorded decision stream contains a DISPATCH decision for a job that does not yet have job->accounts set, that decision's allocation data should be captured immediately -- regardless of the job's current state -- rather than waiting for job->state to become RUN before searching for a match. sched_account_jobs()'s existing RUN-state gate continues to defer actual resource charging until the job's state genuinely transitions to RUN (whenever that recorded event is later replayed), so behavior stays correct even though the underlying allocation data was captured earlier. This must not change job->state transition semantics themselves (used elsewhere for job status display, permissions, and preemption logic) -- only how/when the DISPATCH decision's allocation data is captured internally for accounting purposes.

This is a follow-up to the scheduler-sim-decision-parity spec (implementation-complete, all 10 tasks landed and reviewed) -- do not reopen or modify that spec's own requirements/design/tasks. This is its own new spec, building on that feature's existing sim_reconcile_unaccounted_jobs() mechanism and round-boundary wiring.

Boundary Context

• In scope: capturing a job's recorded DISPATCH allocation data as soon as simulation-mode replay encounters it in a round's recorded decision stream, regardless of the job's current state at that point; transitioning that job to a running state synchronously with capture, mirroring normal-mode's own dispatch-commit semantics (see Requirement 1's 2026-09-11 revision below); preserving correct resource-in-use accounting once capture no longer depends on same-round state coincidence; making the dispatch decision observable at the historically accurate round; preventing double-capture or fabricated data across this wider round span.
• Out of scope: any change to how or when a job transitions to a NON-running terminal or intermediate state (DONE/EXIT/preemption, etc.) -- those triggers and semantics are unchanged; any change to --sim-verify's comparison semantics beyond not producing a new false mismatch as a side effect; any change to simulation mode's live picker (it remains PEND-only for freshly-arriving jobs, unchanged, out of this feature's scope per scheduler-allocation); any change to PEND or PREEMPT decision handling; reopening or modifying scheduler-sim-decision-parity's own requirements, design, or tasks; any change to normal-mode (non-simulation) behavior.
  • Revision (2026-09-11): the original version of this bullet excluded ANY change to job->state transition timing, including the RUN transition. That exclusion is REVISED by Requirement 1's 2026-09-11 update below -- the RUN transition itself is now explicitly in scope, triggered by dispatch-decision capture rather than by the separately-recorded confirmation event. Every other state transition (DONE/EXIT/preemption) remains untouched and out of scope, as originally stated.
• Adjacent expectations: this feature extends scheduler-sim-decision-parity's existing Recorded-Decision Reconciliation mechanism and round-boundary wiring; it does not replace or duplicate that mechanism. It relies on that feature's decision-reporting channel and resource-accounting service remaining exactly as delivered.

Requirements

Requirement 1: Cross-round resource-accounting accuracy for dispatched jobs

Objective: As an operator inspecting simulation-mode resource state, I want a job's resource consumption reflected correctly regardless of how many rounds separate its dispatch from its running confirmation, so that what I inspect during a replay is trustworthy even under realistic dispatch-to-confirmation latency.

Revision (2026-09-11, user-directed, live replay session): the original version of Acceptance Criteria 2/3 below deferred a captured job's transition to a running state until its separately-recorded JOB_STATE_CHANGE confirmation event replayed, potentially many rounds after its dispatch decision was captured -- reproducing a live-observed operator confusion (info jobs/info worker showing a job as still PEND for multiple sessions after [sim] DISPATCH had already printed for it). Investigation confirmed normal-mode execution does NOT work this way: sched_commit_decision() (scheduler.c, bug-1624) sets job->state = JOB_STAT_RUN synchronously, in the SAME operation that captures the dispatch allocation -- from the live scheduler's own perspective, a job IS running the instant the scheduler decides to dispatch it, independent of whatever later, separately-recorded signal confirms the worker actually started it. The Acceptance Criteria below are revised so simulation-mode reconciliation reproduces this same scheduler's-eye-view semantics instead of the worker-confirmation-timeline semantics the original version specified. This is a deliberate, explicit decision to prioritize matching live-mode's own internal state model over preserving the separately-recorded confirmation event's timing as a distinct, independently meaningful signal -- see research.md's corresponding entry for the full rationale trail and the design point that made this ambiguous in the first place.

Correction (2026-09-21, bug-1838, superseding Acceptance Criterion 1 below): "regardless of the job's current state at that point" (originally meaning PEND was explicitly an eligible starting state) turned out to be the root cause of a live-observed over-dispatch (bug-1836: 44 single-slot jobs force-applied onto a 22-slot host, since a PEND job's recorded decision was captured with no live capacity check). Investigation found the premise that justified including PEND no longer holds: the since-completed scheduler-sim-live-allocation spec gave simulation mode its own genuine, capacity-checked live picker/allocator (sched_job_picker()/sched_allocator()), so a freshly-PEND job now correctly reaches that same live path instead of needing this feature's capture mechanism at all. Acceptance Criterion 1 is revised: eligibility is once again gated on the job already being in a running state (JOB_IS_RUN) -- reconciliation's remaining, genuinely necessary scope is a job already confirmed running (via a separately-recorded JOB_STATE_CHANGE/JOB_UPDATE event) whose allocation data was never captured, which is structurally invisible to the live picker. This does not reopen Acceptance Criterion 2 (still synchronous RUN-transition at capture) -- it is now simply a no-op re-affirmation for a job that is already running by the time this criterion's precondition can be satisfied. See design.md's corresponding 2026-09-21 Overview correction for the full rationale.

Acceptance Criteria

1. When simulation-mode replay encounters a round containing a job's originating dispatch decision for a job already in a running state with no allocation data captured yet, the scheduler shall capture that decision's resource-allocation data for the job. (Revised 2026-09-21, bug-1838 -- see correction above; originally "regardless of the job's current state.")
2. When the scheduler captures a job's originating dispatch decision, the scheduler shall also transition that job to a running state at that same round, synchronously with the capture -- mirroring normal-mode execution's own dispatch-commit semantics (Requirement 1's 2026-09-11 revision above), in which a job is considered running from the scheduler's perspective the instant the scheduler decides to dispatch it.
3. The scheduler shall include a captured job's resource consumption in its worker's and any shared resource's resource-in-use totals starting from the same round its dispatch decision is captured, not deferred to a later round.
4. When a job's dispatch decision and its separately-recorded running-state-confirmation event (JOB_STATE_CHANGE/JOB_UPDATE to a running state) land in different rounds, replaying that later confirmation event shall re-affirm an already-running job (a safe, idempotent re-application) rather than being the sole trigger for the transition to a running state.
5. The scheduler shall report the same resource-in-use totals for a dispatched job as normal-mode execution reported at the equivalent point in the original run, regardless of the round separation between the dispatch decision and its separately-recorded confirmation event.

Requirement 2: Decision visibility at the historically accurate round

Objective: As an operator watching simulation-mode decision output, I want a job's dispatch decision to become visible when the replay reaches the round it was actually decided in, so I have an accurate picture of when scheduling activity happened, not only that it eventually happened.

Acceptance Criteria

1. When simulation-mode replay reaches the round containing a job's originating dispatch decision, the scheduler shall make that decision observable through the existing decision-reporting channel at that round, independent of when the job's state later transitions to running.
2. The scheduler shall preserve the original decision's type and resource-allocation detail when reporting it.
3. When simulation-mode independently computes a fresh decision for a job in a given round, the scheduler shall report only that one decision for the job, not also a duplicate from the recorded stream.

Requirement 3: No fabrication and no double-capture across the wider round span

Objective: As a maintainer relying on this feature's correctness, I want captured allocation data to never be fabricated or captured more than once, so that widening reconciliation's effective round span does not introduce a new class of correctness risk.

Acceptance Criteria

1. The scheduler shall never capture resource-allocation data for a job from anything other than that job's own originating recorded decision.
2. Once a job's resource-allocation data has been captured, the scheduler shall not attempt to capture it again from a later round's recorded decisions, regardless of how many further rounds are replayed.
3. If a job's resource-allocation data was already captured, and simulation-mode's own live decision-making subsequently produces a fresh decision for that job, the scheduler shall treat the job as already resolved and not overwrite or duplicate its already-captured data.

Requirement 4: No regression to existing guarantees

Objective: As a maintainer relying on scheduler-sim-decision-parity's already-delivered guarantees, I want this feature to preserve them, so existing verification workflows, tests, and behavior continue to hold.

Acceptance Criteria

1. The scheduler shall continue to correctly reconcile the same-round dispatch-and-confirmation case that scheduler-sim-decision-parity already handles.
2. The scheduler shall continue to support decision verification mode (comparing freshly-computed decisions against recorded ones) without reporting a mismatch caused by this feature's own behavior.
3. The scheduler shall continue to produce identical decisions between interactive (step-by-step) and batch simulation execution modes.
4. While a job has no recorded originating decision available anywhere in the replay's own data (for example, a job already running before the replay's recording window began), the scheduler shall report that job's resource usage as unavailable rather than fabricating a value, and shall continue processing the remainder of the replay without terminating.
5. While simulation mode is not enabled, the scheduler shall not change its existing resource-accounting, decision-reporting, or job-state-transition behavior.