fix(scheduler-sim-reconcile-terminal-guard): task 1.2 - bug-1764 terminal-state guard

sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c)
now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state),
alongside the existing job->accounts != NULL skip. A terminal job is
never searched for a matching recorded DISPATCH entry, so a leftover
or duplicate DISPATCH decision from a later replay round can no
longer resurrect allocs/accounts on a job that has already completed
- the exact mechanism behind info worker's assigned-jobs listing
showing a finished job as still consuming a worker's resources.

The exclusion mirrors job_state_regresses_terminal()'s existing
"terminal is permanent" invariant (core_events.c) rather than
inventing a new rule. The function's doc comment gains a paragraph
clarifying the terminal exclusion is complementary to, not a reversal
of, the existing PEND/RUN state-independence rationale that motivated
this trigger's original widening.

Both RED-phase tests (task 1.1) pass; the pre-existing PEND
eager-capture regression test and core_events.c's JOBS_PUSH
resync-merge terminal-state tests are unaffected. The analogous
resync-merge branch was investigated and confirmed already correct.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd

.kiro/specs/scheduler-sim-reconcile-terminal-guard/tasks.md

@@ -7,7 +7,7 @@
   - _Requirements: 1.1, 1.2, 1.3_
   - _Boundary: Terminal-State Guard_
 
-- [ ] 1.2 Implement the terminal-state exclusion in sim_reconcile_unaccounted_jobs()
+- [x] 1.2 Implement the terminal-state exclusion in sim_reconcile_unaccounted_jobs()
   - Add JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state) as an additional exclusion in the function's existing per-job trigger check, alongside the existing job->accounts != NULL skip; update the function's own doc comment to state the terminal exclusion alongside its existing state-independence rationale so the two don't read as contradictory
   - Observable: all tests from 1.1 pass; a terminal job's allocs/accounts are provably unchanged before and after a call that presents a matching stray DISPATCH entry for it
   - _Requirements: 1.1, 1.2, 1.3_

.wolf/buglog.json

@@ -20801,6 +20801,31 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-09-02T13:00:00Z"
+    },
+    {
+      "id": "bug-1764",
+      "timestamp": "2026-09-03T03:01:26Z",
+      "error_message": "info worker's \"assigned jobs\" listing (repl_info_worker_one(), repl_info.c) can show a job that has already reached a terminal state (DONE/EXIT) as still assigned to a worker, contradicting info summary's running count. Live-observed in a real interactive session (session=59): info summary reported \"100 jobs (0 pending, 7 running)\" and worker pc13v's slots showed used=7.00, but the same info worker output listed assigned jobs: 0-11-1788164699-0[0-49] -- the full 50-index range of one array job, all still shown assigned.",
+      "file": "src/scheduler/src/sim_loop.c",
+      "root_cause": "sim_reconcile_unaccounted_jobs() (sim_loop.c:664) triggers reconciliation on job->accounts == NULL alone (sim_loop.c:690), with no check on job->state. This trigger was deliberately widened to be state-independent by the scheduler-sim-eager-reconciliation spec (task 1.1, test_sim_reconcile_unaccounted.c's own Test 5 header comment: \"capture eligibility no longer depends on job->state\") so a PEND-state job's early-recorded DISPATCH decision could be captured before its RUN confirmation replays. That widening only considered the forward case (PEND -> captured early -> later RUN) and missed the terminal case: a job that already ran to completion has job->accounts/job->allocs correctly freed by xce_core_push_events()'s normal end-of-life cleanup (core_events.c:1127-1135 JOB_STATE_CHANGE, :1179-1186 JOB_UPDATE) -- which also leaves job->accounts == NULL. If ANY later round's decoded_session->decisions still carries an unconsumed DISPATCH entry for that same jobid (plausible in a real recording with duplicate/retried dispatch attempts, or per-round decision-emission timing not lining up cleanly with per-job completion for an array job), sim_reconcile_unaccounted_jobs() matches it and resurrects job->allocs/job->accounts on the already-terminal job, with no state check to prevent it. Dynamically confirmed via a temporary probe test (added to test_sim_reconcile_unaccounted.c, run once, then reverted -- never committed): a job driven to JOB_STAT_DONE via xce_core_push_events() (which correctly nulled its allocs/accounts) was then fed a second decoded_session containing a fresh DISPATCH entry for the same jobid; sim_reconcile_unaccounted_jobs() unconditionally re-populated both job->allocs and job->accounts, contradicting job->state == JOB_STAT_DONE. repl_info_worker_one()'s assigned-jobs filter (repl_info.c:686, if (job->allocs == NULL) continue;) has no independent state check either, so it faithfully (and misleadingly) displays the resurrected allocation. The already-established defensive pattern for exactly this class of problem, job_state_regresses_terminal() (core_events.c:696-701, \"terminal states are permanent\"), is not reused here.",
+      "fix": "FIXED (2026-09-03, scheduler-sim-reconcile-terminal-guard spec, task 1.2). sim_reconcile_unaccounted_jobs()'s per-job trigger check (sim_loop.c) now also excludes JOB_IS_DONE(job->state) || JOB_IS_EXIT(job->state), alongside the existing job->accounts != NULL skip -- a terminal job is never searched for a matching recorded DISPATCH entry, mirroring job_state_regresses_terminal()'s existing 'terminal is permanent' invariant (core_events.c). TDD: task 1.1's RED-phase tests (test_reconcile_never_reallocates_a_done_job_despite_stray_dispatch_entry, test_reconcile_never_reallocates_an_exit_job_despite_stray_dispatch_entry, test_sim_reconcile_unaccounted.c) pinned the exact reproduction before the fix landed; both pass after. Confirmed no regression to the pre-existing PEND eager-capture case or to core_events.c's JOBS_PUSH resync-merge terminal-state tests. The analogous resync-merge branch was investigated and confirmed already correct (job_state_regresses_terminal() call), no change needed there.",
+      "tags": [
+        "scheduler",
+        "sim",
+        "repl",
+        "reconciliation",
+        "job-lifecycle",
+        "stale-state",
+        "array-job"
+      ],
+      "related_bugs": [
+        "bug-1611",
+        "bug-1734",
+        "bug-1738",
+        "bug-1739"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-03T03:01:26Z"
     }
   ]
 }

src/scheduler/src/sim_loop.c

@@ -586,26 +586,49 @@ static void* sim_picker_thread_fn(void* arg)
  * site's own comment below. Still directly unit-testable standalone too
  * (test_sim_reconcile_unaccounted.c), per design.md's own Testing Strategy.
  *
- * Trigger condition (scheduler-sim-eager-reconciliation spec, task 1.1):
- * `job->accounts == NULL` is now the SOLE capture-eligibility gate --
- * state-independent, reused here as both the trigger AND (now that the
- * FOUND branch below populates job->accounts) the double-application guard.
- * Any job in `sim->core.jobs` with `job->accounts == NULL` is eligible to be
- * searched for a matching recorded DISPATCH decision this round, regardless
- * of its current `job->state` -- a job's recorded DISPATCH decision and its
- * RUN-state confirmation can legitimately land many rounds apart in the
- * original recording (the dominant real-world case), and this function is
- * called unconditionally every round, so a not-yet-running job is simply
- * re-checked round after round until its originating decision is found (or
- * the replay ends). Actual resource charging remains fully deferred to
- * sched_account_jobs()'s own unchanged `JOB_IS_RUN` gate (scheduler.c) --
- * capturing a job's allocation here while it is not yet RUN is inert until
- * that gate later includes it; this function performs no charging itself.
- * Requirement 3.3's "report only the fresh one" guarantee needs no separate
- * mechanism here: a job that independently receives a fresh live decision in
- * this same round already has job->accounts set by sched_commit_decision()
- * (scheduler.c, unchanged) by the time this trigger check runs, so this
- * function simply finds nothing to do for it.
+ * Trigger condition (scheduler-sim-eager-reconciliation spec, task 1.1;
+ * narrowed by scheduler-sim-reconcile-terminal-guard task 1.2, bug-1764):
+ * `job->accounts == NULL` is the primary capture-eligibility gate --
+ * state-independent WITH RESPECT TO PEND vs. RUN, reused here as both the
+ * trigger AND (now that the FOUND branch below populates job->accounts) the
+ * double-application guard. Any job in `sim->core.jobs` with `job->accounts
+ * == NULL` is eligible to be searched for a matching recorded DISPATCH
+ * decision this round, regardless of whether it is PEND or RUN -- a job's
+ * recorded DISPATCH decision and its RUN-state confirmation can legitimately
+ * land many rounds apart in the original recording (the dominant real-world
+ * case), and this function is called unconditionally every round, so a
+ * not-yet-running job is simply re-checked round after round until its
+ * originating decision is found (or the replay ends). Actual resource
+ * charging remains fully deferred to sched_account_jobs()'s own unchanged
+ * `JOB_IS_RUN` gate (scheduler.c) -- capturing a job's allocation here while
+ * it is not yet RUN is inert until that gate later includes it; this
+ * function performs no charging itself. Requirement 3.3's "report only the
+ * fresh one" guarantee needs no separate mechanism here: a job that
+ * independently receives a fresh live decision in this same round already
+ * has job->accounts set by sched_commit_decision() (scheduler.c, unchanged)
+ * by the time this trigger check runs, so this function simply finds
+ * nothing to do for it.
+ *
+ * This PEND/RUN state-independence does NOT extend to a job that has
+ * already reached a terminal state: `JOB_IS_DONE(job->state) ||
+ * JOB_IS_EXIT(job->state)` is an additional, unconditional exclusion,
+ * checked alongside `job->accounts != NULL` in the same trigger `if`. A
+ * terminal job's `job->accounts`/`job->allocs` are freed by
+ * xce_core_push_events()'s own end-of-life cleanup on the terminal
+ * transition (core_events.c's JOB_STATE_CHANGE/JOB_UPDATE cases), which
+ * leaves `job->accounts == NULL` -- the exact same shape as a legitimately
+ * not-yet-captured PEND/RUN job. Without this exclusion, a later round's
+ * decoded_session carrying a stale/duplicate DISPATCH entry for that same
+ * jobid (e.g. a retried dispatch attempt for one element of an array job)
+ * would be indistinguishable from a genuine early-dispatch case and would
+ * resurrect the allocation on an already-dead job (bug-1764). The two rules
+ * compose without contradiction: state-independence is about WHEN a
+ * (still-live) job's own DISPATCH decision happens to replay relative to
+ * its RUN confirmation -- it says nothing about a job that can never
+ * legitimately be reconsidered again because it has already finished. This
+ * mirrors job_state_regresses_terminal()'s (core_events.c) existing
+ * "terminal is permanent" invariant, applied here at the allocation layer
+ * instead of the state-transition layer.
  *
  * For each such job, searches `decoded_session->decisions` for a matching
  * DISPATCH entry by jobid -- a job can only ever be captured via a recorded
@@ -687,7 +710,8 @@ static void sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t* sim,
             break;
         }
 
-        if ((job == NULL) || (job->accounts != NULL)) {
+        if ((job == NULL) || (job->accounts != NULL) || JOB_IS_DONE(job->state)
+            || JOB_IS_EXIT(job->state)) {
             continue;
         }