---
id: "bug-1778"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "scheduler"
severity: "unknown"
timestamp: "2026-09-07T06:50:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/scheduler/src/repl_info.c"]
tags: ["job-lifecycle", "stale-state", "repl", "snapshot", "scheduler", "related-to-bug-1776", "assigned-jobs", "persistence", "needs-live-repro", "related-to-bug-1764", "interim-fix", "sim"]
related: ["bug-1764", "bug-1776"]
legacy: true
---

# bug-1778: Live-observed (interactive sim session=29, replaying a real captured trace): three plain…

## Description
Live-observed (interactive sim session=29, replaying a real captured trace): three plain (non-array) jobs already at JOB_STAT_DONE (0-14-1788751409-0, 0-14-1788751435-0, 0-14-1788751504-0) still appear in `info worker pc13v`'s 'assigned jobs' listing (each annotated '*2' via bug-1776's slot-count fix), alongside the genuinely-running array job elements. User: 'The finished job(exit/done) should be de-assign from worker.'

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
PARTIALLY CONFIRMED, NOT FULLY ROOT-CAUSED this session (user asked to record, not fix). Confirmed facts: (1) repl_info_worker_one()'s assigned-jobs loop (repl_info.c, ~line 686) gates solely on `job->allocs != NULL`, with no independent job->state check -- if allocs is non-NULL for a terminal job, it will always display, by design (mirrors the exact structural gap bug-1764 already documented for this same function). (2) The worker's ACTUAL resource accounting is NOT confused by this: the same session's 'resource slots: value=22.00 used=20.00' correctly reflects only the 5 genuinely-running 4-slot array elements (5*4=20), excluding the 3 DONE jobs entirely -- proving core.resmgr/resInUse (the real admission-control state) is fine; only the assigned-jobs DISPLAY loop (which walks job->allocs directly, a separate field) is stale for these 3 jobs. (3) This is NOT a regression of bug-1764's fix: verified sim_loop.c:713-714 still correctly excludes JOB_IS_DONE(job->state)||JOB_IS_EXIT(job->state) from sim_reconcile_unaccounted_jobs()'s resurrection path, so that specific documented mechanism is not what's reintroducing these allocs. (4) core_events.c's XCE_CORE_EVENT_JOB_STATE_CHANGE (line ~1155-1163) and XCE_CORE_EVENT_JOB_UPDATE (line ~1207-1215) handlers both correctly free_JobAlloc_ptr()+null job->allocs atomically alongside the state write whenever `!JOB_IS_RUN(new_state)` -- so if either of THESE handlers was what drove these 3 jobs to DONE, allocs should already be NULL; something else must be true. Leading unconfirmed hypothesis: snapshot_load.c:490-493 deep-copies wj->allocs from the .replay snapshot's wire payload UNCONDITIONALLY whenever wj->allocs != NULL, with no check against the snapshot's own wj->state -- if the live production recording's snapshot-capture moment (snapshot_wire.c's save side) captured these 3 jobs' allocs and state in a mutually inconsistent way (e.g. a save-time field-read ordering gap, or the snapshot genuinely predates their JOB_STATE_CHANGE/JOB_UPDATE terminal cleanup while a separately-recorded later event updated state to DONE without a matching allocs-clearing event reaching this replay stream), the replay would faithfully reproduce that inconsistency verbatim. NOT verified with a live trace/gdb this session -- flagged per this project's own persistence.md field-completeness concern (a state that 'recovers correctly' while an unrelated field goes stale is the exact class of bug that steering calls out), not asserted as confirmed.

## Fix
INTERIM DISPLAY-LAYER MITIGATION APPLIED (2026-09-07, per user request). repl_info_worker_one()'s assigned-jobs loop (repl_info.c) now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state), in addition to the existing job->allocs != NULL check -- mirrors the same 'terminal is permanent' guard job_state_regresses_terminal()/sim_reconcile_unaccounted_jobs() already apply elsewhere. TDD: new test test_info_worker_assigned_job_listing_excludes_terminal_job_despite_stale_allocs (test_repl_resource_table.c) directly assigns the SAME allocs struct to a DONE job and a RUN job on the same worker, forcing the DONE job's state after assigning allocs (reproducing the exact state/allocs inconsistency without needing to root-cause how it arises live) -- confirms the DONE job is excluded while the RUN job still shows. Full scheduler suite: 76/77 pass (same pre-existing unrelated test_core_events.c failure as before, bug-1762). NOTE: this masks the symptom at the display layer only -- the underlying question of WHY job->allocs stays non-NULL for a terminal job (suspected snapshot save/load inconsistency) remains unresolved; a real fix should also prevent/clear the stale allocs at its source.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1764, bug-1776