--- id: "bug-1764" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "scheduler" severity: "unknown" timestamp: "2026-09-03T03:01:26Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/scheduler/src/sim_loop.c"] tags: ["scheduler", "sim", "repl", "reconciliation", "job-lifecycle", "stale-state", "array-job"] related: ["bug-1611", "bug-1734", "bug-1738", "bug-1739"] legacy: true --- # bug-1764: info worker's "assigned jobs" listing (repl_info_worker_one(), repl_info.c) can show a job that… ## Description info worker's "assigned jobs" listing (repl_info_worker_one(), repl_info.c) can show a job that has already reached a terminal state (DONE/EXIT) as still assigned to a worker, contradicting info summary's running count. Live-observed in a real interactive session (session=59): info summary reported "100 jobs (0 pending, 7 running)" and worker pc13v's slots showed used=7.00, but the same info worker output listed assigned jobs: 0-11-1788164699-0[0-49] -- the full 50-index range of one array job, all still shown assigned. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause sim_reconcile_unaccounted_jobs() (sim_loop.c:664) triggers reconciliation on job->accounts == NULL alone (sim_loop.c:690), with no check on job->state. This trigger was deliberately widened to be state-independent by the scheduler-sim-eager-reconciliation spec (task 1.1, test_sim_reconcile_unaccounted.c's own Test 5 header comment: "capture eligibility no longer depends on job->state") so a PEND-state job's early-recorded DISPATCH decision could be captured before its RUN confirmation replays. That widening only considered the forward case (PEND -> captured early -> later RUN) and missed the terminal case: a job that already ran to completion has job->accounts/job->allocs correctly freed by xce_core_push_events()'s normal end-of-life cleanup (core_events.c:1127-1135 JOB_STATE_CHANGE, :1179-1186 JOB_UPDATE) -- which also leaves job->accounts == NULL. If ANY later round's decoded_session->decisions still carries an unconsumed DISPATCH entry for that same jobid (plausible in a real recording with duplicate/retried dispatch attempts, or per-round decision-emission timing not lining up cleanly with per-job completion for an array job), sim_reconcile_unaccounted_jobs() matches it and resurrects job->allocs/job->accounts on the already-terminal job, with no state check to prevent it. Dynamically confirmed via a temporary probe test (added to test_sim_reconcile_unaccounted.c, run once, then reverted -- never committed): a job driven to JOB_STAT_DONE via xce_core_push_events() (which correctly nulled its allocs/accounts) was then fed a second decoded_session containing a fresh DISPATCH entry for the same jobid; sim_reconcile_unaccounted_jobs() unconditionally re-populated both job->allocs and job->accounts, contradicting job->state == JOB_STAT_DONE. repl_info_worker_one()'s assigned-jobs filter (repl_info.c:686, if (job->allocs == NULL) continue;) has no independent state check either, so it faithfully (and misleadingly) displays the resurrected allocation. The already-established defensive pattern for exactly this class of problem, job_state_regresses_terminal() (core_events.c:696-701, "terminal states are permanent"), is not reused here. ## Fix FIXED (2026-09-03, scheduler-sim-reconcile-terminal-guard spec, task 1.2). sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c) now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state), alongside the existing job->accounts != NULL skip -- a terminal job is never searched for a matching recorded DISPATCH entry, mirroring job_state_regresses_terminal()'s existing 'terminal is permanent' invariant (core_events.c). TDD: task 1.1's RED-phase tests (test_reconcile_never_reallocates_a_done_job_despite_stray_dispatch_entry, test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry, test_sim_reconcile_unaccounted.c) pinned the exact reproduction before the fix landed; both pass after. Confirmed no regression to the pre-existing PEND eager-capture case or to core_events.c's JOBS_PUSH resync-merge terminal-state tests. The analogous resync-merge branch was investigated and confirmed already correct (job_state_regresses_terminal() call), no change needed there. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1611, bug-1734, bug-1738, bug-1739