---
id: "bug-1611"
status: "fixed"
status_updated: "2026-09-24"
status_note: "e2bb98c8, 984a6869"
component: "scheduler"
severity: "unknown"
timestamp: "2026-08-21T15:10:00Z"
last_seen: "2026-09-19T14:00:00Z"
occurrences: 3
files: ["src/scheduler/src/sim_loop.c"]
tags: ["repl", "sim", "replay", "dispatch", "decision-visibility", "scheduler-sim-diagnostics", "scheduler-sim-eager-reconciliation"]
related: ["bug-1764", "bug-1613", "bug-1610", "bug-1609"]
fix_commits: ["e2bb98c8", "984a6869"]
legacy: true
---

# bug-1611: Discovered analyzing a real 108-session replay (xce_scheduler -i -s…

## Description
Discovered analyzing a real 108-session replay (xce_scheduler -i -s scheduler-replay-20260820-070103.replay, continue to EOF): sim_print_decision() (sim_loop.c) never printed a single DISPATCH or PREEMPT line across the entire replay -- only PEND -- even though 22 of the 30 jobs pushed at round 32 clearly transitioned to running (confirmed by ongoing JOB_UPDATE/JOB_RUSAGE ticks for jobids [0]-[21] every few rounds through round 107). An operator watching only decision output has zero confirmation those 22 jobs ever dispatched; they just silently stop appearing in PEND.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
Not yet root-caused with certainty. Plausible, benign explanation: in --sim/-s REPLAY mode, jobs already resolved (dispatched) in the ORIGINAL recorded run may be replayed purely as state deltas (JOB_UPDATE/JOB_RUSAGE) without the live picker/allocator re-deriving a fresh XCE_DECISION_DISPATCH -- only jobs still outstanding (PEND) get freshly re-decided every round, which is why PEND is the only decision op ever observed. Unconfirmed against sim_loop.c's actual replay-vs-live-decide branching; needs a source read of how sim_loop.c distinguishes replayed-decision playback from freshly-computed decisions before concluding this is 'working as designed' vs a real visibility gap.

## Fix
Not yet fixed. If confirmed benign (dispatch legitimately has no fresh decision object in replay mode), the round-summary rollup being designed for bug-1609/1610 should not label its dispatch count misleadingly (a permanent 'X dispatch' column reading 0 for an entire replay looks like a bug even when it isn't) -- consider deriving a running-job delta from info-jobs-equivalent state instead of the decision tally for replay-mode display. If NOT benign (a real decision-visibility gap), this is a separate, more serious finding than the formatting issues in bug-1609/1610.

--- STATUS UPDATE 2026-08-24 (xce-audit-invariants scheduler audit) ---

Root cause confirmed and the literal mechanism is now built: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709, landed via scheduler-sim-decision-parity tasks 2.1-3, 2026-08-22) reconstructs and reports a decision for a RUN-state job whose DISPATCH decision was only in the recorded replay stream. BUT its trigger gate (sim_loop.c:592) requires JOB_IS_RUN(job->state) before searching, and the search only scans the CURRENT round's decoded_session (decoded fresh at sim_loop.c:871, freed at line 999 -- no cross-round persistence). So reconciliation only fires when a job's DISPATCH decision and its RUN confirmation land in the exact same replay round. This bug's own original scenario (dispatch at round 32, RUN confirmation through round 107) never matches -- the round-32 DISPATCH decision is gone by the time the job reaches RUN state. The scheduler-sim-decision-parity spec's own task 4.3 implementation notes explicitly acknowledge this residual does NOT fix bug-1611's dominant real-world manifestation. A follow-up spec already exists for this exact residual: .kiro/specs/scheduler-sim-eager-reconciliation/ (created 2026-08-22, phase=requirements-generated, ready_for_implementation=false -- design/tasks not yet generated). Its diagnosis: decouple allocation CAPTURE from the JOB_IS_RUN gate (capture as soon as a DISPATCH decision is seen for a job with job->accounts == NULL, regardless of state), leaving sched_account_jobs()'s existing JOB_IS_RUN + job->accounts != NULL gate to keep deferring actual charging until state transitions to RUN. Fix location: sim_reconcile_unaccounted_jobs() (sim_loop.c:566-709), trigger gate at line 592, plus its ownership-transfer logic (609-693) needs a double-capture guard since capture would no longer imply state==RUN. Do NOT fix inline as a drive-by patch per scheduler-driver-equivalence.md's testing discipline -- promote scheduler-sim-eager-reconciliation to design/tasks instead.

--- STATUS UPDATE 2026-08-27 (scheduler-sim-eager-reconciliation task 1.1) ---

Core production fix landed: sim_reconcile_unaccounted_jobs() trigger gate (sim_loop.c:592) had !JOB_IS_RUN(job->state) removed, leaving job->accounts == NULL as the sole capture-eligibility gate -- exactly the diagnosis from the 2026-08-24 status update. Match/capture/report mechanics (609-693) were confirmed already safe for early (pre-RUN) capture without changes (see scheduler-sim-eager-reconciliation/research.md's uncharge-safety research log -- sched_uncharge_account() has zero production callers, accounting is reset-and-resum not incremental, so no double-capture guard beyond the existing job->accounts != NULL check was needed). Old test test_reconcile_skips_non_run_state_job (which asserted the bug as correct behavior) replaced with test_reconcile_captures_non_run_state_job_when_dispatch_recorded (inverted expectation). RED/GREEN verified: new test failed under the old gate (job->allocs stayed NULL, confirmed via a real build+run before the fix), then passed after the one-line fix; full sched_sim_reconcile_unaccounted_test suite green (10/10, all prior regression tests unchanged). This is task 1.1 of the spec only -- task 1.2 (capture-safety unit coverage), task 2.1 (resInUse deferral integration test), and task 3.x (contention/--sim-verify/full-regression validation) are still pending before the spec itself is implementation-complete. CORRECTED STATUS UPDATE 2026-09-19 (superseding the immediately-preceding note in this same entry, which itself worked from stale premises): scheduler-sim-eager-reconciliation (spec completed 2026-09-11, BEFORE this session's audit ran) already decoupled sim_reconcile_unaccounted_jobs()'s trigger from the JOB_IS_RUN gate -- the function now fires on job->accounts == NULL alone, regardless of PEND/RUN state (sim_loop.c:560-572), so a job's DISPATCH decision is captured in the SAME round it is recorded, never deferred until a later, separately-recorded RUN-state confirmation event. This closes the exact real-world scenario that motivated this bug (dispatch round 32, confirmation round 107) for good -- the previous note's claimed "residual gap" (a job already-RUN before the replay window starts) is not a real gap either: such a job's allocation is populated directly from the snapshot at load time (xce_sched_core_load_snapshot(), confirmed field-complete for job->allocs/accounts by this same audit's dimension-4 pass), so job->accounts is already non-NULL immediately post-snapshot-load and such a job is never a reconciliation candidate in the first place. Combined with scheduler-sim-reconcile-terminal-guard (bug-1764, excludes terminal jobs from false rematch) and scheduler-sim-live-allocation (the live picker/allocator now handles most jobs directly, reconciliation is a narrow safety net), this entry is FULLY FIXED. STATUS: FIXED, not PARTIALLY FIXED.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1764, bug-1613, bug-1610, bug-1609